Stationkeeping

The reclamation post ended on a debt: the whole utilization story depends on the reservation being a good estimate of real usage, and nobody said where that estimate comes from. The answer is a control loop, and the deeper answer is a position I'll defend for the rest of this post, the right number of resource knobs for a user to set is zero.

The tension you can't config your way out of

There's a fight built into every orchestrator between the power user and the casual user. The power user wants a knob for everything and gets cranky when the system hides one. The casual user wants to run a web server and is horrified to learn there are knobs at all. Borg served the power user and ended up with BCL and its 230-odd parameters. Kubernetes served, arguably, no one, and ended up with YAML and a templating ecosystem on top of it.

You can't resolve this with a nicer config language, because the tension isn't about syntax. It's about whether a human has to know the answer to "how much memory will this use under load", and most humans don't, and shouldn't have to, and when forced to guess they guess high, which is exactly the stranded capacity the last post was trying to reclaim.

Borg's own remedy was to build automation on top, which Google productized as Autopilot, and the results are the entire argument. Autopiloted jobs ran at 23% resource slack versus 46% for hand-managed ones, and got hit by OOMs roughly ten times less often. Half the waste, a tenth of the pages, by taking the numbers away from people. Orbit makes that the default path rather than an opt-in feature, and the thing that does it is the Stationkeeper.

The name is literal. Stationkeeping is the small continuous thrust a satellite fires to hold its assigned orbital slot against the constant tug of drift. This does the same to a workload: continuous small corrections to keep it the right size for its declared intent.

Three loops, each minding one thing

The Stationkeeper is three independent control loops, CPU, memory, and replica count, and keeping them independent is deliberate:

pub struct Stationkeeper {
    cpu: ControlLoop<PeakWithMargin>,        // vertical
    mem: ControlLoop<PeakWithMargin>,        // vertical
    horizontal: ControlLoop<PeakWithMargin>, // replica count
}

pub fn tick(&self, module: &Module, history: &UsageWindow, current_reservation: &ResourceVec, current_replicas: u32) -> Vec<Adjustment> {
    let budget = OomBudget::from(module.appclass);
    let mut out = Vec::new();
    if let Some(a) = self.cpu.propose_cpu(&module.name, history, current_reservation.cpu_millis, budget) { out.push(a); }
    if let Some(a) = self.mem.propose_memory(&module.name, history, current_reservation.memory_bytes, budget) { out.push(a); }
    if matches!(module.replicas, ReplicaPolicy::Managed) {
        if let Some(a) = self.horizontal.propose_replicas(&module.name, history, current_replicas, current_reservation.cpu_millis) { out.push(a); }
    }
    out
}

Each loop proposes at most one adjustment per tick, or none. They don't coordinate, and they don't need to, because each one is idempotent, a missed tick self-corrects on the next pass, since each loop looks at current state and observed usage every time rather than tracking a delta it might lose. A control loop you can drop ticks from without consequence is a control loop you can run cheaply and restart freely.

Notice the horizontal loop only runs when replicas are Managed. That's the user's one decision: hand the system the replica count, or pin it. Declare intent, or declare a number. Most modules should declare intent.

The estimator is Borg's line, drawn explicitly

What turns a window of usage samples into a reservation is a ResourceEstimator, and the default one is about as simple as it can be while still being honest:

impl ResourceEstimator for PeakWithMargin {
    fn estimate(&self, history: &UsageWindow, budget: OomBudget) -> ResourceVec {
        let margin = budget.margin();
        ResourceVec {
            cpu_millis:   (history.cpu_p99() as f64 * margin) as u64,
            memory_bytes: (history.mem_p99() as f64 * margin) as u64,
            // disk dimensions take the observed max
        }
    }
}

Reservation equals the p99 of observed usage times a safety margin. That's the whole estimator, and it's Borg's tightrope written as one line: shrink the slack as far as you can without shrinking it so far the task gets killed. The p99 captures "how much does it really use, ignoring the rare spike," and the margin buys back headroom for the spike.

The margin is the interesting part, because it's where the OOM-risk decision lives, and it's set by the workload's class rather than by a human:

fn margin(&self) -> f64 {
    match self {
        OomBudget::Tight    => 1.30, // latency-sensitive
        OomBudget::Moderate => 1.15, // batch
        OomBudget::Relaxed  => 1.05, // best-effort
    }
}

impl From<AppClass> for OomBudget {
    fn from(class: AppClass) -> Self {
        match class {
            AppClass::LatencySensitive { .. } => OomBudget::Tight,
            AppClass::Batch                   => OomBudget::Moderate,
            AppClass::BestEffort              => OomBudget::Relaxed,
        }
    }
}

A latency-sensitive service gets a fat 30% margin because killing it is expensive and you'd rather waste a little memory than OOM a user-facing process. A best-effort job gets 5%, because it's preemptible anyway and the whole point is to claw back the gap. The user picked their app class, which is a meaningful, intent-level choice, and the margin falls out of it. Nobody typed "1.30."

Not thrashing is half the job

A naive control loop oscillates: it nudges the reservation, the nudge changes the usage ratio, it nudges back, and now it's flapping forever, churning the ledger and the schedulers for no benefit. Two mechanisms keep that from happening.

The vertical loops ignore changes below a minimum step, 10% by default, so a 3% drift doesn't trigger a rewrite:

let ratio = new_val as f64 / current_reservation as f64;
if (ratio - 1.0).abs() >= self.min_step { Some(adjustment) } else { None }

The horizontal loop uses a dead band, with different thresholds for scaling up and down so it can't ping-pong across a single line:

let new_count = if load > 0.80 {
    /* scale up */
} else if load < 0.40 {
    /* scale down, floor at 1 */
} else {
    current_count // stable zone: do nothing between 40% and 80%
};

Between 40% and 80% load, it does nothing. That gap is what stops a workload hovering near a threshold from adding and removing a replica every minute. Hysteresis is unglamorous and it's most of what makes an autoscaler usable instead of a liability.

It goes through the front door

The Stationkeeper doesn't get a special channel to mutate state. When the controller decides on an adjustment, it submits it through the airlock, the same door from the admission post, with an epoch that increments per adjustment:

for adjustment in adjustments {
    airlock.adjust_module(&mission.id, &module, adjustment_to_commitment(adjustment), expected_epoch).await?;
    expected_epoch = Epoch(expected_epoch.0.saturating_add(1));
}

An adjustment becomes a ModuleAdjustment commitment in the ledger, guarded by an expected epoch, which is the ModuleAdjustmentVersion precondition from the ledger post. So two adjustments racing, or a stale Stationkeeper running on an old view, get caught by the same optimistic-concurrency check as everything else. The autoscaler is just another client proposing conflict-checked commitments. It earns no privileges, and it can't corrupt anything the schedulers and the airlock can't.

And the loop it moves is the reservation, not the limit. That's the direct payoff of the reclamation post: the Stationkeeper continuously decays each module's reservation toward its real usage, and the gap between reservation and limit is the reclaimable capacity that best-effort and batch work runs in. Stationkeeping is what makes the gap real. Without it the reservation is a stale human guess and there is no gap; with it the reservation tracks reality and the utilization win actually shows up.

The honest part

I want to be straight about how simple the current estimator is. Real Autopilot used decaying histograms and a model trained across workloads; PeakWithMargin is p99 times a constant. It's a defensible starting point and it's honest about its failure mode, too tight and you OOM, too loose and you waste, but it is not the sophisticated thing, and the quality of the estimator is most of what separates "the autoscaler I trust with my reservations" from "the autoscaler I turned off after it paged me."

There's also a gap visible right in the controller: each pass builds a one-sample window, UsageWindow::new(1), so today the "history" the estimator reasons over is a single fresh observation rather than a real multi-tick window. The windowing machinery, percentiles, means, load ratios, is all there and tested, but the persistence that would feed it a deep history across ticks isn't wired up yet. That's the next real piece of work on this loop, and it's the kind of thing I'd rather show you in the code than paper over, because the gap between "the math is implemented" and "the math is fed good data" is exactly where autoscalers quietly underdeliver.

Next we leave resources behind for a while and deal with a different inherited wound: networking. Borg gave every task an IP and regretted it, Kubernetes gave every pod an IP and built an industry around managing the consequences, and Orbit addresses workloads by who they are instead of where they landed.