Rehearsal

Way back in the object-model post I made a point of saying orbit-core has no I/O, no network, no disk, just types and the functions over them, and in the ledger post I kept calling it an append-only, replayable log. Those weren't aesthetic preferences. They were setup for this post, which is about the highest-leverage tool in the whole system: a simulator that is the real code.

Fauxmaster, and why it was a big deal

Borg had a thing called Fauxmaster. It ran the actual Borgmaster scheduling code against stubbed Borglets, fake node agents that accepted placements and reported state without running any containers. You could feed it a checkpoint of a real cell's state and ask it questions. It was used for debugging, for capacity planning, and for the question that keeps SREs up at night: if I make this config change, will it evict anything important?

The reason Fauxmaster was so valuable is the reason it's hard to build: it only works if the simulator runs the *real* scheduler. A simulator that's a separate model of how you think your scheduler behaves is worse than useless, because it drifts from the real one and gives you confident, wrong answers. The whole value is fidelity, and fidelity comes from there being no second implementation.

Orbit gets this almost for free, because of those early decisions. The core is pure, so it runs anywhere with no setup. The ledger is a replayable log, so you can feed it a checkpoint. The only thing you have to fake is the machine.

Stub the node, keep everything else real

The fake is the StubSatellite. It has a capacity and it accepts placements, and it does the resource arithmetic, but there's no container runtime, no cgroups, no youki:

pub fn start_capsule(&mut self, cap: CapsuleId, env: ResourceEnvelope) -> Result<(), String> {
    if !self.config.healthy {
        return Err("node unhealthy".to_string());
    }
    if self.capsules.contains_key(&cap) {
        return Err("capsule already running".to_string());
    }
    if !self.remaining().covers(&env.limit) {
        return Err("insufficient capacity".to_string());
    }
    self.capsules.insert(cap, env.clone());
    self.usage = self.usage.add(&env.limit);
    Ok(())
}

Crucially, "would this capsule fit?" is answered by the same covers and add operations the real node uses, the resource algebra from the reclamation post, property-tested and shared. The stub fakes the machine, not the math.

Everything above the machine is the production code, unmodified. The simulator builds a real scheduling view from the stub state, runs a real Navigator's plan, commits through a real Ephemeris, and replays the committed snapshot back into the stubs:

pub async fn schedule_commit_and_replay<N, E>(&self, navigator: &N, ephemeris: &E, pending: Vec<PendingCapsule>)
    -> Result<SchedulingSimulationResult, SimulationError>
where N: Navigator + ?Sized, E: Ephemeris + ?Sized,
{
    self.replay_ephemeris(ephemeris).await?; // rebuild stub state from the ledger
    for capsule in pending {
        let view = self.scheduling_view(vec![capsule.clone()]); // real SchedView
        let proposals = navigator.plan(&view, &[capsule]).await; // real scheduler
        // real ephemeris.propose(...), counting committed / conflicts / infeasible
    }
}

That generic signature is the tell, it takes &dyn Navigator and &dyn Ephemeris, so you hand it the same ServicesNavigator or GangNavigator and the same ledger you'd run in production. The simulator can't disagree with production about how scheduling works, because it's calling production's scheduler. There's no model to keep in sync. There's just the system, with the machines drawn as cardboard.

What a rehearsal tells you

A run comes back as a structured result, and it reports more than "did it fit":

pub struct SchedulingSimulationResult {
    pub replay: SimulationResult, // placed / failed / evicted / ignored / utilization
    pub proposed: u32,
    pub committed: u32,
    pub conflicts: u32,
    pub unclaimed: u32,
    pub infeasible: u32,
}

You see how many capsules got placed, how the utilization landed across nodes, and, because it's the real optimistic-commit path, how many proposals hit conflicts and how many were infeasible. That's the scheduler's actual behavior, observable, on a laptop, in milliseconds. The two questions Fauxmaster was built to answer fall right out of this.

Do I have room for this launch? Add a few thousand stub nodes with the capacity of your real fleet, throw the workload at them, read the utilization and the failed count. The orbit-demo binary does exactly this from a .orbit file: load a config, simulate placing it, print a capacity report. No production cluster touched.

Will this change evict anything? Replay the current ledger checkpoint into the stubs so they reflect what's actually running, then apply the proposed change and watch the evicted count. If a config tweak would knock over something important, you find out in a rehearsal instead of from a pager.

It rehearses more than scheduling

The scheduler isn't the only thing worth rehearsing, and the same trick, real code, faked edges, extends to the identity and networking story from a couple posts back. There's a second simulator that wires up the *real* airlock, transponder, beacon, and dataplane:

pub struct IdentityNetworkSimulator {
    dataplane: Arc<UserspaceDataplane>,
    beacon: DataplaneBeacon<InMemBeacon>,
    airlock: InMemAirlock,
    // ...
}

With it you can rehearse the whole admission-to-routing flow: issue a client SVID, admit a mission, publish endpoints for its capsules, and then try to route from one identity to another and confirm the deny-by-default policy does what you expect. The network policy for a workload is testable before it ships, the same way the placement is. Given how the airlock post built admission as a saga that installs and tears down dataplane policy, being able to dry-run "can the frontend actually reach the backend after this admission" is worth a lot.

Determinism is the feature

One property makes all of this trustworthy: a rehearsal is reproducible. The scheduler sorts its candidates, the ledger orders its commits, and the stubs apply the snapshot deterministically, so the same input produces the same trace every time. That's what turns the simulator from a curiosity into a debugging tool. When something behaves strangely in a rehearsal, you can replay the exact scenario as many times as you need, change one thing, and run it again. A flaky simulator would be a liar; a deterministic one is a witness.

It's also where the open scaling risks I keep flagging get their first real stress test. The exact-batch-assignment problem, the gang search across many candidate groups, those are exactly the things you want to push against thousands of stub nodes before you trust them on real ones. The simulator won't tell you about kernel quirks, but it'll tell you whether the scheduler's logic falls over at scale, and it'll tell you reproducibly.

The honest part

The fidelity is uneven, and knowing where it's high and where it's low is the whole skill of using it. It's high exactly where it matters most and is hardest to test otherwise: the scheduler, the ledger, the optimistic-commit dance, the admission saga, the resource math. All of that is the real code, so a rehearsal of it is as trustworthy as the code itself.

It's low at the physical layer, by construction. The stub models capacity and placement; it does not model a slow GPU-memory checkpoint, a network partition that strands a node, a kernel that throttles differently than the math predicts, or a card that throws ECC errors forty minutes into a training run. A rehearsal that says "this fits and evicts nothing" is necessary but not sufficient, it clears the logic, not the physics. The physics is what the live smoke tests on real hardware are for, and a healthy respect for the gap between "the simulator is happy" and "production is happy" is the difference between a useful tool and a false sense of security.

But that gap is narrow and it's in the right place. The expensive, scary, hard-to-test part, does the scheduler do the right thing, will this change hurt, is the part the simulator runs for real. That's the whole reason the core was built with no I/O and the ledger as a replayable log. Those constraints felt like discipline at the time. This is what they were for.

Next we widen the lens past a single Constellation to the thin layer that ties many of them together, and why that layer is allowed to advise but never to command.