Reclaiming the Gap

Here's a number that should bother you: most clusters run at something like a third of their capacity, and the operators know it, and they pay for the other two thirds anyway. The reason is that everyone sizes their workloads for the worst case. You ask for the memory your service uses during its annual traffic spike, and then it sits at a fraction of that for the other 364 days, and the difference is stranded, reserved, paid for, unused.

Borg's quietest superpower was refusing to leave that on the table. Roughly a fifth of Google's workload ran in resources that were technically already spoken for, in the gap between what jobs reserved and what they actually touched. Kubernetes treated this as an afterthought, in-place resize and the vertical autoscaler showed up years late and still feel bolted on. In Orbit the machinery for reclaiming the gap is in the core types, because you can't add it convincingly later. It has to be the shape of the resource model from the start.

Limit and reservation are different numbers

The whole thing rests on splitting one idea most systems conflate. How much can a workload use, and how much do we expect it to use, are not the same number, and Orbit keeps them apart:

pub struct ResourceEnvelope {
    pub limit: ResourceVec,       // the ceiling: what it may consume
    pub reservation: ResourceVec, // the live estimate: what we expect it to
}

The limit is the hard ceiling enforced by the kernel, the number from the Satellite post that gets programmed into cgroup v2. The reservation is a running estimate of actual need. The space between them is capacity that's nominally allocated but not really being used, and that space has a name and a method:

/// The reclaimable gap: capacity reserved but not guaranteed.
pub fn reclaimable(&self) -> ResourceVec {
    self.limit.saturating_subtract(&self.reservation)
}

That gap is where reclaimed work runs. A latency-sensitive service holds a limit big enough for its spike but a reservation that tracks its real, lower usage, and the difference gets handed to preemptible batch work that's happy to be kicked off the moment the service actually needs the room. Two workloads, one pile of hardware, and the expensive one never notices.

App classes decide who runs where

What makes the gap safe to fill is that different workloads get different deals, and the deal is a type:

pub enum AppClass {
    LatencySensitive { reserve_cores: bool }, // guaranteed; may pin physical cores
    Batch,                                    // guaranteed reservation, preemptible
    BestEffort,                               // zero reservation; lives in the gap
}

The envelope a workload gets is a function of its class:

pub fn for_workload(limit: ResourceVec, appclass: AppClass, _preemptible: bool) -> Self {
    match appclass {
        AppClass::BestEffort => Self::best_effort(limit),        // reservation = 0
        AppClass::LatencySensitive { .. } | AppClass::Batch => Self::guaranteed(limit), // reservation = limit
    }
}

A best-effort workload reserves nothing. Its whole existence is the reclaimable gap, it runs in capacity other workloads aren't currently using, and it's the first thing evicted when they want it back. This is the cleanest version of the idea: best-effort work is pure reclamation, costing zero reserved capacity by construction.

The scheduler honors the split without any special-casing, which I want to show because it's a place the model quietly pays off. Recall from the scheduling post that a placement proposal's precondition guards on envelope.reservation. A best-effort capsule has a zero reservation, so it consumes no reserved capacity when it commits, but the greedy engine still checks the capsule's limit against the node's physical capacity, so you can't drop a 9-core best-effort job onto a 4-core box just because it reserves nothing. There's a test for exactly that pair of behaviors: zero reservation consumed, physical limit still enforced. The reservation governs contention; the limit governs reality; and because they're separate fields, the scheduler gets both right for free.

Feasibility follows Borg's rule, falling straight out of which number you check. Latency-sensitive work is sized against its limit, so it never depends on reclaimed capacity being available, it has to fit for real. Batch work can be packed against its lower reservation, so it slots into the gap. Same scheduler, same envelope type, different field, opposite behavior.

No fixed buckets

There's a second, less obvious decision baked into the resource type. Resources are fine-grained and independent, every dimension is its own number, and there are no t-shirt-size buckets:

pub struct ResourceVec {
    pub cpu_millis: u64,
    pub memory_bytes: u64,
    pub disk_bytes: u64,
    pub disk_bw_bps: u64,
    pub accelerators: Vec<AcceleratorReq>,
}

This sounds like a detail and it's worth real money. Borg measured it: forcing workloads into fixed buckets, small, medium, large, instead of letting them ask for exactly what they need cost 30 to 50 percent more machines, because every workload rounds up to the next bucket and the rounding is pure waste at fleet scale. So a capsule asks for 1,400 millicores and 3.2 GB, not "medium," and packs into whatever space genuinely fits.

Independence is what makes the arithmetic clean, and the arithmetic is the kind of thing that has to be exactly right because errors compound across fifty thousand nodes into either overcommitted machines or stranded capacity. covers is a partial order, does this vector have at least as much of every dimension as that one, and the operations are checked the way you'd check a small algebra:

// these hold for all generated vectors, on every build
prop_covers_reflexive();      // a covers a
prop_covers_transitive();     // a covers b, b covers c => a covers c
prop_add_subtract_identity(); // (a + b) - a == b
prop_reclaimable_safe();      // limit always covers the reclaimable gap

And the invariant that ties the envelope together: a valid envelope's reservation plus its reclaimable gap exactly equals its limit, with nothing lost or invented in between:

assert_eq!(env.reservation.add(&env.reclaimable()), env.limit);

Property tests on resource arithmetic feel like overkill until you remember that an off-by-a-little here doesn't crash, it quietly oversubscribes a machine, or quietly wastes a percent of the fleet, and you find out from a billing dashboard six months later. I'd rather find out from the compiler.

Compressible versus not: throttle or kill

The last piece is the distinction that decides what happens when a machine runs out. Some resources you can take back gently and some you can't:

pub enum Compressibility {
    Compressible,    // CPU: throttle it, the workload just runs slower
    NonCompressible, // memory, disk: can't throttle; exhaustion forces a kill
}

CPU is compressible. A task that wants more CPU than is available just runs slower, and that's recoverable, nobody dies. Memory is not. You cannot throttle your way out of being out of RAM; something has to be killed to free it. This is why the Satellite's pressure signals from the last post treat memory specially, with memory_critical as the trigger that forces an eviction while CPU pressure only ever throttles.

When the machine does have to give capacity back, the order is the one you'd hope for: lowest priority first, preemptible before guaranteed, best-effort before batch. The reclaimed work was borrowing the gap, so taking it back is the system working as designed, not a failure. The best-effort job that got evicted to make room for a latency spike did exactly its job, it kept the hardware busy until the moment the owner wanted it.

The honest part

All of this hinges on the reservation being right. The gap is only safe to fill if the reservation actually tracks real usage, set it too high and there's no gap to reclaim, so you're back to a third utilization; set it too low and the latency-sensitive workload gets squeezed by reclaimed work it should have displaced. The reservation isn't a number a human picks. It's owned by a control loop that watches real usage and decays the reservation toward it, continuously, which is the entire subject of the next post.

That's the deeper bet of this whole series, actually. Reclamation is where the utilization win lives, and the utilization win is most of the argument for building any of this. But it's only real if the automation that sizes reservations is good enough to trust with the gap. So next: the Stationkeeper, and why I think the right answer to "what number should this be" is to stop asking the user entirely.