Gangs, Fabric, and Googput

This is the post where "improve on Borg today" stops being a slogan. Borg was designed for a world of fungible CPU and memory, and that world is gone for an enormous and growing slice of what data centers actually do, which is train large models. Almost every assumption in Borg's resource model quietly breaks for ML, and the breaks aren't small.

Borg packs independent, fungible resources, spreads work for headroom, and recovers from failure by killing and requeueing. Now picture a 64-GPU training job. The accelerators are not fungible, the job needs all 64 or it makes no progress, so "spread for headroom" is exactly wrong. Placement isn't about free cores, it's about the network fabric between the GPUs, because that's what the all-reduce step rides on and that's what decides whether the thing converges. And kill-and-requeue is catastrophic, because the thing you killed was forty hours into a run. You can't staple this onto a CPU bin-packer. It needs its own scheduler.

The shape of the problem

Four things have to be true at once for ML scheduling to work, and none of them is true in a classic scheduler.

The placement has to be all-or-nothing. A training job half-placed is worse than not placed: it's holding expensive accelerators that other jobs could use, while making zero progress, waiting for members that may never come.

The placement has to be topology-aware. GPUs talking over NVLink inside one host move data orders of magnitude faster than GPUs talking across a network rail, and a collective operation runs at the speed of its slowest link. Put the gang on the wrong fabric and training slows to a crawl even though every GPU shows 100% utilization.

Which is the third thing: the objective is goodput, not utilization. A job pinning every accelerator at full tilt while making no training progress is pure waste that looks like perfect efficiency on a dashboard.

And reclaiming a gang requires checkpoint, not kill, or preemptibility is a lie you tell to improve utilization numbers while destroying days of work.

Gangs and the fabric they live on

The object model already carries the intent. A module declares itself a gang:

pub enum GangPolicy {
    Independent,
    Gang { min_members: u32, topology: AcceleratorTopology },
}

pub enum AcceleratorTopology {
    Any,
    SameNvlinkDomain,
    SameHost,
    SameRail,
    SameZone
}

SameNvlinkDomain is the tightest and fastest; SameZone the loosest. The scheduler that places these is the GangNavigator, and it reasons over a structure the other schedulers don't have: a FabricGraph describing the physical network, not just a list of nodes with free capacity.

pub struct NodeTopology {
    pub node_id: NodeId,
    pub accelerators: Vec<AcceleratorSlot>,
    pub rail: RailId,
    pub zone: String,
}

pub struct AcceleratorSlot {
    pub node_id: NodeId,
    pub kind: AcceleratorKind,   // Gpu("h100"), Tpu("v6e"), ...
    pub count: u32,
    pub nvlink_island: NvlinkIslandId,
    pub rail: RailId,
    pub zone: String,
}

This is the locality information Borg never modeled. Borg's only notion of locality was "this machine already cached the package." The FabricGraph knows which nodes share an NVLink island, which share a rail, which share a zone, the actual hierarchy that decides collective-communication bandwidth.

When the GangNavigator looks for places to put a gang, it asks the fabric for groups that satisfy the requested topology, and each tier comes back with a bandwidth score baked in:

AcceleratorTopology::SameNvlinkDomain => /* bandwidth_score 1000 */,
AcceleratorTopology::SameHost         => /* bandwidth_score  900 */,
AcceleratorTopology::SameRail         => /* bandwidth_score  500 */,
AcceleratorTopology::SameZone         => /* bandwidth_score  200 */,
AcceleratorTopology::Any              => /* bandwidth_score  100 */,

Those numbers are a proxy, but the ordering is the physics: NVLink beats same-host beats same-rail beats same-zone beats wherever-there's-room. The scheduler prefers tighter fabric because tighter fabric is faster training.

Finding a set that actually fits

Here's where gang scheduling is genuinely harder than the greedy services case. Placing one capsule is "find the best node." Placing a gang is "find N nodes that simultaneously have room for a member each, all on compatible fabric", a constraint-satisfaction problem, not a sort. The GangNavigator does a backtracking search over a candidate group, reserving resources as it descends and unwinding when a branch dead-ends:

fn search(&self, index, selected, scalar_remaining, accel_requested) -> bool {
    if selected.len() == self.member_count {
        return true;
    } // found a full set
    let remaining = self.candidates.len().saturating_sub(index);
    if selected.len() + remaining < self.member_count {
        return false;
    } // can't finish: prune

    for next in index..self.candidates.len() {
        let node = &self.candidates[next];
        // does this node still have the scalar resources a member needs?
        if !available.covers(self.needs) { continue; }
        // does it still have enough un-reserved accelerators?
        if !self.navigator.node_has_accelerators(node, self.kind, new_accel, self.reserved) { continue; }

        selected.push(node.clone());
        // reserve this node's slice, then recurse
        if self.search(next + 1, selected, &next_scalar_remaining, &next_accel_requested) { return true; }
        selected.pop(); // dead end: give the resources back and try another node
    }
    false
}

It tracks two kinds of reservation as it goes, scalar resources (the CPU and memory each member needs for data loading and the like) and accelerators per node, because a node might host more than one member, and you can't double-count its GPUs. The prune at the top (selected.len() + remaining < member_count) cuts off branches that can't possibly complete, which is what keeps this from being hopeless on large candidate sets. It's not free, and I'll come back to that, but it's the honest shape of the problem: you're solving for a feasible set, not picking a winner.

Among the topology groups that yield a feasible set, it scores by goodput proxy and penalizes larger groups, because more members means more chances for a straggler to stall the collective:

fn goodput_score(&self, group: &GangGroup) -> f64 {
    let straggler_penalty = (group.nodes.len() as f64 - 1.0) * 10.0;
    group.bandwidth_score - straggler_penalty
}

Tightest fabric, fewest stragglers. Not most free cores.

All-or-nothing, enforced by the ledger

Now the property that makes the whole thing safe. The GangNavigator emits a single proposal for the entire gang, which becomes one atomic commitment guarded by one precondition:

Self::Gang { gang_id, .. } => Commitment::GangPlacement {
    gang,
    members,
    epoch 
},
// guarded by:
Precondition::GangUnplaced { gang: gang_id.clone() }

There is no path where half the gang lands. The commitment places every member or it isn't committed at all, and the ledger's own invariant checks, from the commitment post, reject a GangPlacement if the gang is already placed or if any member capsule is already spoken for. Partial placement isn't prevented by careful coding in the scheduler; it's impossible by construction, because the unit of commitment is the whole gang. A training job can never get stuck holding 40 of its 64 GPUs hostage while waiting for the other 24, because the ledger would never have accepted 40 in the first place.

This is the cleanest example in the whole system of pushing a correctness property down into the type that represents it. The scheduler does a hard search and proposes; the ledger guarantees atomicity. Neither has to trust the other to get gangs right.

Checkpoint instead of kill

The last requirement was reclaiming a gang without throwing away the run, and that's where the Satellite's checkpoint support and the gang's checkpoint policy meet. A gang carries a checkpoint policy, and preemption is defined as checkpoint-then-stop, the Preempt command from the Satellite post does exactly that:

pub struct GangSpec {
    // ...
    pub checkpoint: Option<CheckpointPolicy>,
}

So preempting a training gang to reclaim its accelerators for something higher priority costs one checkpoint interval, minutes, instead of the whole run. That turns the reclamation story from the resources post into something that actually works for ML: a gang can run as preemptible work in capacity that's available now, and when the owner of that capacity wants it back, the gang checkpoints, yields, and resumes later from where it left off. Without checkpointing, "preemptible training" is a phrase that means "we delete your work to improve our utilization metric," and nobody would opt into it.

The GPU health telemetry from the firehose post closes the loop here too. A gang should not be placed on a card throwing double-bit ECC errors and thermal-throttling, because that card is the straggler that stalls the whole collective. Surfacing GPU health as telemetry is what lets the fabric-aware placement also be a health-aware placement.

The honest part

I flagged two risks about this in the very first post and I'm not going to walk them back now that we're here.

The backtracking search is correct, and its worst case is exponential. The prune helps, real candidate sets are far from worst case, and the topology grouping shrinks the search before it starts, but "we find a feasible 64-member set quickly across fifty thousand nodes" is a performance claim I haven't earned at scale, and it needs the same equivalence-class and relaxed-randomization pruning that the services scheduler needs, applied to sets instead of singletons. It's the hardest open scaling problem in the system and I'd rather name it than bury it.

And checkpoint cost is the load-bearing assumption under the whole preemptible-gang story. Checkpointing CPU state with CRIU is one thing; checkpointing tens of gigabytes of GPU memory is heavier and slower, and if it's too slow, then the minutes-not-days math falls apart and gangs can't be reclaimed aggressively after all. That directly constrains how much of the fleet you can safely run as preemptible training, and it's a number I want from a real machine, not a design doc.

What I'm confident in is the structure: gangs as a first-class atomic unit, a fabric graph that models the network instead of pretending it's flat, goodput as the objective, and checkpoint-before-preempt as the reclamation primitive. The 2015 resource model couldn't express any of that. This one is built around it.

Next we come up for air and deal with the thing users actually touch: configuration, and why I think the answer to YAML is a typed language that compiles.