Identity Over Location

Jeremy Tregunna

Jeremy Tregunna

— 6 min read
Share

Networking is the one place Orbit refuses to take Borg's advice. Borg's own retrospective says to give every unit its own IP, and Kubernetes did exactly that, an IP per pod, and then spent a decade building CNI plugins, overlay networks, IPAM to hand out the addresses, and iptables and IPVS rules that don't scale, all to manage the consequences. Borg's earlier sin was the opposite shape and the same root: tasks shared a machine IP and carved up a port range, which dragged port allocation into the scheduler and made it everyone's problem.

Both mistakes come from the same assumption: that you address a workload by where it is. Orbit addresses a workload by who it is, and nearly all of the networking tax just stops applying.

Every workload gets a Transponder

When a mission is admitted, each capsule gets a cryptographic identity, a SPIFFE-style SVID, which is a short-lived signed document saying "I am this workload." It's called a Transponder, and the identity is bound in the ledger as an IdentityBind commitment, so identity is a promise the consistent store keeps, not a label anyone can spoof.

pub struct Svid {
    /// e.g. "spiffe://orbit/iad1/mission/checkout/module/api/capsule/7"
    pub uri: String,
    pub subject: TransponderId,
    pub issued_at: SystemTime,
    pub ttl: Duration,
    pub cert: Option<Vec<u8>>, // X.509 in production
    pub key: Option<Vec<u8>>,  // ephemeral private key
}

Look at the URI, because it's doing something deliberate. The identity *is* the object-model path: constellation, mission, module, capsule. The hierarchy from the object-model post isn't just how humans group things; it's literally the workload's name on the network:

pub fn spiffe_uri(constellation: &str, mission: &MissionId, module: &ModuleName, capsule: &CapsuleId) -> String {
    format!("spiffe://orbit/{constellation}/mission/{}/module/{}/capsule/{}", mission, module, capsule)
}

And it parses back the other way, so a peer presenting an SVID tells you exactly which mission and module it belongs to, which is the basis for any policy you'd want to write. The Transponder trait is the issue-and-verify surface, and the SVIDs are short-lived on purpose, a leaked identity expires on its own:

#[async_trait]
pub trait Transponder: Send + Sync {
    async fn issue(&self, capsule: &CapsuleId, module: &ModuleName, mission: &MissionId, ttl: Duration)
        -> Result<Svid, TransponderError>;
    async fn verify(&self, svid: &Svid) -> Result<TransponderId, TransponderError>;
    async fn verify_certificate(&self, cert: &[u8]) -> Result<Svid, TransponderError>;
    async fn revoke(&self, id: &TransponderId) -> Result<(), TransponderError>;
}

This is the machinery underneath the airlock from the admission post. When a client calls admit, the airlock verifies the client's SVID to get the principal. That verify is this trait. Identity is checked at the front door and then it's the currency everything downstream trades in.

Beacon resolves identity to wherever it landed

If you don't route by IP, you need a way to find the current address of a workload, and that's Beacon. It keeps Borg's BNS idea, a stable name independent of physical location, but the name is an identity, not a DNS entry, and it resolves to wherever the capsule actually is right now:

#[async_trait]
pub trait Beacon: Send + Sync {
    async fn publish(&self, ep: Endpoint) -> Result<(), BeaconError>;
    async fn resolve(&self, id: &TransponderId) -> Vec<Endpoint>;
    async fn resolve_port(&self, id: &TransponderId, port_name: &str) -> Vec<Endpoint>;
    async fn withdraw(&self, id: &TransponderId, addr: &str) -> Result<(), BeaconError>;
    async fn watch(&self, id: &TransponderId) -> Vec<Endpoint>;
}

An Endpoint ties a named port to a concrete address and the identity behind it:

pub struct Endpoint {
    pub port_name: String,           // "http": resolved, not hardcoded
    pub addr: String,                // "10.0.1.5:8080"
    pub transponder_id: TransponderId,
    pub registered_at: SystemTime,
}

Now walk the scenario that makes location-based addressing miserable: a capsule gets rescheduled to a different machine. Its IP changes. In an IP-addressed world, every client holding the old address is now talking to nothing, and you need some convergence dance to fix it. Here, the capsule keeps its TransponderId across the move, identity is bound in the ledger, not tied to the host, and on the new machine it publishes a new endpoint under the same identity. Clients resolve the identity, get the new address, and never know anything happened. The named port matters too: clients ask for "http" on an identity, not for port 8080 on a host, so the physical port is free to be whatever the machine had available. That's Borg's port problem dissolved, ports are an implementation detail Beacon hides, not a scarce resource the scheduler fights over.

The dataplane keys on identity, denies by default

Resolution tells you where to send traffic. The dataplane decides whether you're allowed to, and it makes that decision on identity, never on address:

#[async_trait]
pub trait Dataplane: Send + Sync {
    async fn bind(&self, id: TransponderId, endpoint: DataplaneEndpoint) -> Result<(), DataplaneError>;
    async fn set_policy(&self, source: TransponderId, destination: TransponderId, decision: PolicyDecision)
        -> Result<(), DataplaneError>;
    async fn route(&self, source: &TransponderId, destination: &TransponderId) -> Result<Route, DataplaneError>;
    fn capabilities(&self) -> DataplaneCapabilities;
    // unbind, remove_policy ...
}

A route call takes two identities, who's calling, who they want, and either returns a path or refuses. The refusals are the interesting part, because they encode a closed-by-default posture:

pub enum DataplaneError {
    NoRoute(TransponderId),
    PolicyDenied { source_id: TransponderId, destination: TransponderId },
    // ...
}

No endpoint bound for the destination means no route. No policy permitting the pair means denied. You have to explicitly allow a source identity to reach a destination identity, and nothing is reachable until someone does. This is where the airlock's admission saga from the earlier post pays off: when it admits a mission, it installs the allow-policies that let the right clients reach the new capsules, and it tears them down on rejection. The network for a workload is opened by admission and closed by default, and the unit of policy is "identity may talk to identity," which is a sentence a human can actually reason about, unlike an iptables chain.

One contract, two backends

Identity-keyed in-kernel routing is the Cilium direction, and in production the dataplane is EBPF: programs hooked into the socket `connect` path that consult maps of endpoints and policies and enforce the decision in the kernel:

pub const EBPF_ENDPOINTS_MAP_NAME: &str = "orbit_endpoints";
pub const EBPF_POLICIES_MAP_NAME:  &str = "orbit_policies";
pub const EBPF_CONNECT4_PROGRAM_NAME: &str = "orbit_policy_connect4";
pub const EBPF_CONNECT6_PROGRAM_NAME: &str = "orbit_policy_connect6";

EBPF is powerful and it ties you to kernel and feature support, which is a real portability cost. So the design doesn't start from EBPF. It starts from the *trait*, and the first implementation is a deterministic userspace dataplane:

pub enum DataplanePreference { UserspaceOnly, PreferEBPF, RequireEBPF }

The invariants, deny by default, no route without an endpoint, identity-keyed policy, are defined by the contract and implemented in userspace, where every one of them is testable on a laptop with no special kernel. Then the EBPF backend has to satisfy the same contract, verified by a live smoke test on a BPF-capable machine. An operator who needs kernel enforcement sets RequireEBPF and fails closed if it's unavailable; one who wants portability sets PreferEBPF and falls back to userspace with the policy semantics preserved. Same guarantees, different teeth.

There's a nice consequence I'll point at without dwelling on it: the control plane's own traffic runs on this. The telemetry client and the Satellite's control-plane client both connect through an `EBPFOutboundConnector` that routes a gRPC channel by source and destination identity. Orbit isn't identity-routed for workloads and IP-routed for itself. It's identity all the way down.

What this buys

Add it up and the entire location-based apparatus is gone. There's no overlay network, because you're not tunneling addresses around. There's no IPAM handing out per-pod IPs, so there's no IPAM exhaustion. There's no growing pile of iptables or IPVS rules that turns service routing into an O(services) scaling problem, because a route is an identity lookup in a map. A capsule's address is an implementation detail that can change whenever the scheduler wants, and nothing breaks, because nothing was ever addressing it by location.

And policy gets better, not just cheaper. "The checkout API may call the payments ledger" is a statement about two identities, and it stays true when either side is rescheduled, scaled, or moved to a different rack. Compare that to a network policy written against pod CIDRs that has to be rewritten every time the topology shifts.

The honest part

The same caveat from the Satellite post applies here, doubled, because security code is exactly where fakes lying to each other is most dangerous. The in-memory Transponder and userspace dataplane are deterministic and exhaustively testable, but "the userspace fallback enforces the invariant" and "the EBPF program enforces the same invariant in the kernel" are two different claims, and only the live smoke test bridges them. I trust the contract; I respect that the kernel is a different country.

Short-lived SVIDs also need a renewal path that rotates identities before they expire without flapping every long-lived connection mid-request, which is the same genre of problem as the airlock's certificate renewal, fiddly, security-critical, and unforgiving of off-by-one-window bugs. It's real work, and it's the price of identities being short-lived, which is a price worth paying because a credential that expires on its own is a credential a breach can't keep.

Next we get to the part that genuinely didn't exist when Borg was designed, and the reason "improve on Borg today" earns its keep: scheduling machine learning, where the network fabric decides whether your job converges and a half-placed gang is worse than no gang at all.

Read more