The Agent That Outlives Its Brain

Everything I've written about so far is the control plane: the ledger, the schedulers, the airlock. Now go stand next to an actual machine in an actual rack. On that machine runs the Satellite, the per-node agent, and it has one property that matters more than all the others combined. When the entire control plane goes dark, ledger down, schedulers gone, airlock unreachable, the Satellite keeps your containers running. The brain can have a stroke and the work doesn't even hiccup.

This is the largest crate in Orbit, and it's the one I'd point to if you asked which part is closest to load-bearing. A scheduler that's down means no new work gets placed, which is annoying. A node agent that panics when it can't phone home means running work dies, which is an outage. Borg understood this and so does every system that's survived contact with production: the data plane has to be independent of the control plane.

The pull model

The first decision is who calls whom. Orbit uses Borg's answer: the control plane polls the Satellite, not the other way around. The agent never pushes unprompted, never floods, never participates in a thundering herd when something recovers.

#[async_trait]
pub trait SatelliteControlPlane: Send + Sync {
    async fn poll(&self, node_id: &NodeId, previous_delta: &NodeStateDelta)
        -> Result<Vec<Command>, SatelliteError>;
}

A poll is a round trip: the control plane hands down a batch of commands, the Satellite applies them and reports back its full current state. The commands are a small, closed set:

pub enum Command {
    Start    { capsule: CapsuleId, image: String, envelope: ResourceEnvelope, network: Option<CapsuleNetwork> },
    Stop     { capsule: CapsuleId, deadline: DateTime<Utc> },
    Envelope { capsule: CapsuleId, envelope: ResourceEnvelope }, // resize limits in place
    Checkpoint { capsule: CapsuleId },
    Preempt  { capsule: CapsuleId, deadline: DateTime<Utc> },    // checkpoint, then stop
}

Start and Stop are obvious. Envelope resizes a running capsule's resource limits without restarting it; that's the autoscaler's adjustment landing on the machine, and it's a couple of posts away. Checkpoint and Preempt are the ML story, also later. The point for now is that the whole control surface of a node is five verbs, and the agent reports full state after each batch rather than a diff, exactly the resilience trick from the telemetry post, applied at the source. Putting flow control in the hands of the poller is what stops a recovering control plane from getting knocked flat by fifty thousand agents all reconnecting at once.

The loop, and the part that can't fail

The agent's loop is small on purpose. Poll, apply, report, sleep, repeat. The interesting bit is what happens when the poll *fails*:

pub async fn poll_once(&self, previous_delta: &NodeStateDelta) -> Result<PollOutcome, SatelliteError> {
    // reconcile local container state first; this does NOT depend on the control plane
    if let Err(error) = self.satellite.reconcile_networks().await {
        return Ok(PollOutcome::LocalApplyFailed { /* still return a delta */ });
    }
    match self.control_plane.poll(&self.node_id, previous_delta).await {
        Ok(commands) => match self.satellite.on_poll(commands).await {
            Ok(delta)  => Ok(PollOutcome::Applied { commands: commands.len(), delta }),
            Err(error) => Ok(PollOutcome::LocalApplyFailed { /* ... */ }),
        },
        Err(error) => Ok(PollOutcome::ControlPlaneUnavailable {
            error: error.to_string(),
            delta: self.satellite.collect_delta().await, // collect state anyway
        }),
    }
}

Look at the signature. poll_once returns Result<PollOutcome, _>, and a control plane outage is not the Err arm; it's an Ok(ControlPlaneUnavailable). The distinction is the whole philosophy. A failed poll is a normal outcome, not an error condition. The agent shrugs, collects its state anyway, and carries on. It never reaches a branch where "I couldn't reach the control plane" leads to "so I'll stop the containers." That branch doesn't exist.

And notice the reconcile happens before the poll and doesn't depend on it at all. The Satellite keeps its own containers converged with its own local record of what should be running, control plane or no control plane. There's a test that pins this exact behavior, the runtime gets reconciled even when the poll errors out:

async fn poll_once_reconciles_runtime_even_when_control_plane_is_unavailable() {
    // control plane returns an error...
    let outcome = agent.poll_once(&empty_delta).await.unwrap();
    assert!(matches!(outcome, PollOutcome::ControlPlaneUnavailable { .. }));
    assert_eq!(calls.load(Ordering::Relaxed), 1); // ...but the runtime still reconciled once
}

When the loop sees an unavailable control plane, it backs off, exponentially, capped, and it logs sparingly, only on the first failure and then on powers of two, so a control plane that's down for an hour produces a handful of log lines instead of a flood. Then it keeps polling, because the moment the plane comes back, the next poll just works and the agent catches up.

Proving the agent doesn't lose work

The "running work survives" claim is the kind of thing that's easy to assert and hard to keep true as code changes, so it's nailed down with a property test rather than a comment. Generate a random sequence of polls, some succeeding with a start command, some failing outright, and assert that the number of running capsules at the end equals the number of successful starts, control plane errors never remove anything:

fn prop_control_plane_errors_never_remove_previously_started_capsules(
    outcomes in proptest::collection::vec(any::<bool>(), 1..24)
) {
    // ... drive the agent through the random success/error script ...
    prop_assert_eq!(state.list().await.unwrap().len(), expected_running);
    prop_assert_eq!(delta.capsules.len(), expected_running);
}

No matter how the control plane flickers, up, down, up, down, every capsule that was successfully started stays started. That's the availability property as an executable check, run on every build, and it's worth more to me than any prose I could write about resilience.

Assembled from swappable parts

A Satellite has to do genuinely hard, OS-specific things: run OCI containers, program cgroup v2, read kernel pressure, checkpoint process state, persist to disk. None of that is testable in a unit test on a laptop, and all of it is exactly the code you most want under test. The way out is the same trait-pluggability the rest of Orbit leans on. A Satellite is built from four traits:

Satellite::new(
    node_id,
    runtime,  // OciRuntime:        start / stop / checkpoint / list / reconcile
    cgroups,  // CgroupEnforcement: program limits, read PSI pressure
    state,    // LocalState:        durable record of what should be running here
    telemetry // reporter
)

Each trait has a real implementation and an in-memory one. The OCI runtime is youki in production and an InMemOciRuntime in tests. Cgroup enforcement is a real cgroup v2 filesystem implementation or an in-memory stand-in. Local state is JSON files on disk or a map in memory. So the agent loop, the survive-the-plane logic, the backoff, the full property test suite above, all of it runs in milliseconds against fakes, while the same code drives youki and real cgroups on an actual node. The hard parts are isolated behind traits so the logic can be tested exhaustively and the I/O can be swapped for the environment. There are live smoke tests for the real paths, but they're a separate thing you run on a real Linux box with privileges, not a tax on every build.

Enforcement and pressure

Two of those traits deserve a closer look because they're where the resource model meets the kernel. Cgroup enforcement programs the capsule's ResourceEnvelope limit into cgroup v2, so the limit isn't a number in a database that something politely respects; it's enforced by the OS, and a process that tries to exceed it is stopped by the kernel, not by Orbit asking nicely.

The same subsystem reads pressure back out. Linux exposes PSI (Pressure Stall Information), and the Satellite turns it into the signals the telemetry post already showed flowing up the firehose:

pub fn memory_critical(&self) -> bool { self.memory >= 0.9 }
pub fn any_critical(&self) -> bool {
    self.memory_critical() || self.cpu >= 0.95 || self.io >= 0.95
}

This is where compressible and non-compressible resources start to matter, and it's the hinge for the reclamation post. Memory pressure is dangerous in a way CPU pressure isn't: you can throttle a CPU-starved task and it just runs slower, but a memory-starved node has to kill something, because you can't throttle your way out of being out of RAM. So memory_critical is the signal that forces an eviction decision, while CPU pressure only ever throttles. The Satellite is where that distinction has teeth, because it's the thing holding the kernel's hand.

Why local durable state is the linchpin

Pull all of this together and the survive-the-plane property comes down to one idea: on its own machine, the Satellite is the source of truth for what's running, not the control plane. The LocalState trait persists that truth to disk. A Satellite that reboots reloads its capsules from local state and reconciles them back into existence. A Satellite cut off from the control plane keeps enforcing the limits and running the containers it already knows about. When the plane returns, it polls, reports its full state, and the control plane reconciles its own view to match reality, not the other way around.

That inversion is the whole game. The control plane expresses desire; the Satellite holds reality and defends it. As long as those are separate, the brain can go down without the body dying.

The honest edges are mostly about the real I/O paths. youki, cgroup v2, and CRIU checkpointing all need a real kernel and real privileges, so the production paths get exercised by live smoke tests rather than the fast in-memory suite, and that's a weaker net than I'd like, fakes can agree with each other and both be wrong about the kernel. Checkpointing in particular gets heavier and slower once GPUs are involved, which is a thread I keep promising to pull and will, in the ML post.

Next, though, the thing the Satellite has been quietly enforcing this whole time: the resource model, and Borg's quietest superpower, reclaiming the gap between what a workload asks for and what it actually uses.