The Advisory Layer

We've built one Constellation: a few tens of thousands of machines, a ledger, a flock of schedulers, an autoscaler, identity, the works. Now you have more than one, because you have more than one region, or you've grown past what a single failure domain should hold. The question is what sits above them, and it's a question Kubernetes and Borg answered in opposite, instructive ways.

Kubernetes made the cluster the unit of scaling, so the moment you outgrow one you're running several and reaching for a federation ecosystem to make them act like one. You got cluster independence for free, because they were just separate clusters, and then paid for coordination with a whole bolted-on apparatus. Borg went the other way and just made the cell bigger, capping its size only for blast radius. Orbit wants both things that are in tension here: the containment of independent cells, and fleet-wide optimization across them. The way to get both is to be very disciplined about what the layer above a Constellation is allowed to do.

It's allowed to advise. It is not allowed to command. That single rule is the whole design.

A Constellation needs nothing from above it

Start from the property that can't be compromised: a Constellation is a hard failure domain, and it keeps running no matter what happens to anything outside it. That was true of the control plane being down (the Satellite post), and it has to stay true of the Fleet plane being down. If a Constellation can't admit work, schedule, or operate without the Fleet plane, then the Fleet plane is a single point of failure spanning your entire fleet, and you've recreated, at the top, exactly the correlated-failure risk that capping cell size was meant to prevent.

So the Fleet plane is advisory only, and the interface is built to make that obvious:

#[async_trait]
pub trait FleetPlane: Send + Sync {
    async fn register(&self, summary: ConstellationSummary) -> Result<(), FleetError>;
    async fn list_constellations(&self) -> Result<Vec<ConstellationSummary>, FleetError>;
    async fn placement_hint(&self, needs: &ResourceVec) -> Result<Option<PlacementHint>, FleetError>;
    async fn set_global_quota(&self, principal: &Principal, quota: GlobalQuota) -> Result<(), FleetError>;
    async fn check_global_quota(&self, principal: &Principal, needs: &ResourceVec) -> Result<bool, FleetError>;
    async fn record_consumption(&self, constellation: &str, amount: &ResourceVec) -> Result<(), FleetError>;
    // deregister, get_constellation ...
}

Constellations register a summary of themselves and record what they've consumed. The Fleet plane aggregates and suggests. Nowhere in there is a method that places a capsule, commits anything to a Constellation's ledger, or blocks a Constellation from operating. The data flows up, summaries and consumption, and what flows down is only ever a hint.

What a Constellation looks like from orbit

The Fleet plane sees each Constellation as a summary, not as the millions of objects inside it:

pub struct ConstellationSummary {
    pub name: String,
    pub capacity: ResourceVec,
    pub used: ResourceVec,
    pub node_count: u32,
    pub mission_count: u32,
    pub health: ConstellationHealth, // Healthy | Degraded { reason } | Unreachable
}

This is the same instinct as the two-plane split from the very first technical post, applied one level up. Down inside a Constellation, the churny per-node telemetry is kept out of the consistent ledger. Up here, the churny per-capsule detail of every Constellation is kept out of the Fleet plane, it gets a rolled-up summary, refreshed as Constellations report, deliberately coarse. A Fleet plane that tried to hold every Constellation's full state would be the etcd mistake wearing a bigger coat.

The health field is the load-balancing primitive. A Constellation that's degraded or unreachable is simply not eligible for new work:

pub fn is_eligible_for_placement(&self) -> bool {
    matches!(self.health, ConstellationHealth::Healthy)
}

The hint is a suggestion, and only a suggestion

The one piece of "intelligence" the Fleet plane offers is a placement hint: given a workload's needs, which Constellation looks like the best home right now?

async fn placement_hint(&self, needs: &ResourceVec) -> Result<Option<PlacementHint>, FleetError> {
    // among healthy constellations with enough available capacity...
    // ...prefer the emptiest one return the best, or None
    let score = 1.0 - summary.utilization();
}

pub struct PlacementHint {
    pub constellation: String,
    pub reason: String, // "lowest utilization with sufficient capacity"
    pub score: f64,
}

It picks the healthy Constellation with room and the lowest utilization, and hands back a name, a score, and a human-readable reason. Notice what it returns: a suggestion with a justification, not an order. A client deciding where to launch a new mission can ask the Fleet plane "where's got room?" and get a sensible answer that spreads load across the fleet. But the client doesn't have to ask, and if the Fleet plane is down, the client picks a Constellation some other way and launches anyway. The hint improves placement; it's never required for placement. That's the difference between advisory and load-bearing, and it's the whole ballgame.

Global quota, deliberately kept off the critical path

The airlock post left a thread hanging: quota there is per-Constellation, but a team has a budget that spans the whole fleet. That global budget lives here:

pub struct GlobalQuota { pub total: ResourceVec, pub used: ResourceVec }

async fn check_global_quota(&self, principal: &Principal, needs: &ResourceVec) -> Result<bool, FleetError>;
async fn record_consumption(&self, constellation: &str, amount: &ResourceVec) -> Result<(), FleetError>;

And here's the subtle, important part. Global quota is accounting, not enforcement in the consistent sense. The authoritative, must-be-agreed quota grant is still the per-Constellation ledger commitment from the airlock; that's linearizable, that's what actually gates admission, and it keeps working when the Fleet plane is gone. The Fleet plane's global quota is the eventually-consistent aggregate that Constellations report into and consult for planning.

I made that choice on purpose, and it's the two-plane philosophy again: the must-keep-running path (admitting work into a Constellation) does not get to depend on a slow, wide, cross-region service that's allowed to be unavailable. The cost is real and I won't hide it, because global quota is eventual, a team can briefly overshoot its fleet-wide budget across two Constellations before the aggregate catches up and a later admission gets steered away. That's a deliberate trade of precise global enforcement for availability, and whether it's acceptable depends entirely on what the quota is protecting. For "don't let one team starve the fleet," eventual is fine. For "this is a hard billing ceiling," you'd want a tighter mechanism, and building that without making it load-bearing is open design work.

Why this beats federation-by-accretion

The Kubernetes path gives you independence as an accident, separate clusters are separate because they're separate, and then makes you assemble coordination yourself out of a federation control plane, cross-cluster service meshes, and quota tooling, each its own thing to run and reason about. Orbit inverts it: containment is the designed-in default (a Constellation is a hard failure domain whether or not a Fleet plane exists), and coordination is a single thin advisory layer that you can run, or not, without ever putting the fleet's availability in its hands. You get the correlated-failure isolation Borg prized and the fleet-wide optimization, without the optimization layer becoming a thing that can take everything down.

The honest part

I'll be straight that this is the least-built-out layer in the system, and that's by design, not by accident. The build order I sketched in the first post puts the Fleet plane dead last, after a single Constellation is genuinely solid, precisely because it's advisory; you don't need it to have a working orchestrator, and building it early would tempt you into making it load-bearing to justify the effort. What exists is the reference InMemFleetPlane, the gRPC service around it, and the quota and summary types. What's thin is the integration: a scheduler actually consulting placement hints as part of where-to-launch, and global quota reconciling cleanly against the authoritative per-Constellation ledgers under churn. That's frontier, and I'd rather say so.

And the deepest risk here isn't code, it's discipline. "Advisory only" is a property you have to actively defend, because there's always a tempting next feature, just one critical check, just one little cross-Constellation lock, that quietly makes the global layer authoritative. The first time a Constellation can't operate without the Fleet plane, you've thrown away the blast-radius containment that justified the whole architecture, and you probably won't notice until the Fleet plane has its first bad day and takes the fleet with it. Keeping this layer advisory is a thing I'll have to keep choosing, not a thing the types can fully enforce for me.

Where this leaves us

That's the system, end to end. Two planes split by what each part needs from consistency. A tiny ledger of promises with a flock of optimistic schedulers racing to commit against it. An agent that outlives its own control plane. Reclamation and right-sizing that claw back the capacity everyone over-reserves. Identity instead of addresses. Gangs and fabric for the workloads that didn't exist when Borg was designed. A typed config that compiles, a simulator that is the real code, and this advisory layer holding it all in formation without ever being allowed to give an order.

One idea runs under all of it. The avoidable problems in cluster management are nearly all over-uniformity, one store, one cluster, one scheduler, one config language, one address space, and the genuinely hard problems are tradeoffs, which have no right answer, only a placed one. Orbit refuses the false uniformity, places each real tradeoff on purpose, and spends most of its effort hiding the consequences so an ordinary user never has to look. Where it can, it makes the compiler do the enforcing, illegal states that don't compile, data races that can't be written, and where the compiler can't reach, property tests and a mutation audit stand in for the proof. That's about as close to knowing a system is correct as this kind of software gets, rather than hoping.

I'll end where the first post did, on honesty, because a series that only listed wins would be a sales deck and I'm not selling anything. The core is real and tested. The frontier is real too, and I named it at every step: exact batch assignment at fifty thousand nodes, GPU-checkpoint cost deciding whether preemptible training is a feature or a slide, config error messages, global quota reconciling under churn, the coverage that thins out at the kernel boundary. Those are the years ahead, not the weeks. But the shape is right, and the shape is the part you can't bolt on later, you can make a scheduler faster, but you can't retrofit the two-plane split or identity-based networking onto a system that didn't start with them. Orbit started with them, on purpose, because we finally have the decade of written-down regret that lets you build a cluster manager from the tradeoffs instead of from the defaults.

That's the whole thing. Thanks for reading all of it, genuinely. See you in the next one.

Oh, by the way. I promised code, and here it is: GitHub.