Fleet-scale backpressure

This is a simulation of a distributed system showing how traffic is admitted from clients into the system. Every API host asks one central concurrency tracker before admitting a request, and the tracker can only answer so fast. When a tenant floods the fleet, which per-tenant limiter on the hosts keeps the tracker alive: a static token bucket, or AIMD?

The system

A fleet of API hosts serves many tenants. Before a host lets a request start, it asks one central concurrency tracker whether that tenant has a free slot. The tracker keeps an exact count of in-flight requests per tenant, which is what makes it the right place to enforce concurrency limits. It is also a single service with a hard ceiling: it can answer only so many checks per second.

Every simulated request goes through the same steps:

  1. It arrives at one host. If that host is over its total limit, it is shed without regard to tenant.
  2. The host asks its per-tenant limiter (token bucket or AIMD). If the limiter says no, the request is rejected locally and the tracker never hears about it.
  3. Otherwise the host sends a check to the tracker, which answers admitted or at limit.
  4. The answer goes back to the limiter that sent it: onSuccess() or onBackoff().

Past its capacity the tracker fails a growing share of checks (1 − capacity ÷ load) for every tenant, and hosts reject those requests (fail closed). So one tenant’s flood takes everyone down unless the hosts shed it before it reaches the tracker. A failed check also counts as a negative signal, so AIMD protects the tracker’s total capacity as well as each tenant’s limit.

Why a static token bucket is so hard to tune

A token bucket needs a rate. The usual approach turns each tenant’s concurrency limit into one:

rate per host = factor × tenant max concurrency ÷ fleet size

By Little’s law, concurrency = request rate × request duration. So the rate a tenant can legitimately sustain depends on how long its requests take, and that varies by tenant. Ember sends 8,000 fast 50 ms calls a second while using about 400 of its 1,000 slots. Fjord’s 5 s calls fill 2,000 slots at 400 a second. Traffic also lands unevenly on hosts, and a fixed per-host split can’t follow it.

  • Factor too low: tenants are shed while they still have free slots (over-throttling). On a normal day here you need a factor of about 12 before every tenant is served.
  • Factor too high: during an attack each surging tenant may send factor × its limit checks per second, no matter what the tracker answers (under-throttling). For the major DDoS, anything above about 4.4 overloads the tracker.

For the mild and major attacks here those ranges don’t overlap. How wide the gap is depends on how traffic happens to land on hosts, which no static factor can follow. The band under the factor slider computes both bounds live for the current settings.

Why AIMD converges by itself

AIMD uses the same bucket but lets the tracker set its rate. Each host starts every tenant at a maximum rate that is effectively unlimited. Each at limit or error answer multiplies that host’s rate for that tenant by the decrease factor (0.9). Each admitted request adds the increase (+1 TPS).

The key is amplification. One rejection lowers the rate for every later request from that tenant on that host, so a few thousand rejections a second become hundreds of thousands of local rejections across the fleet. The tracker only has to say no enough times to shape the fleet; it never has to absorb the flood. No one configures a rate: each host settles near its share of what the tracker actually admits.

At steady state each host balances additive increases against multiplicative decreases:

increase × admitted/s ≈ (1 − decrease) × rate × rejected/s

That puts an upper bound on the rejected probes each host keeps sending per surging tenant: increase ÷ (1 − decrease), which is 10 TPS with the defaults.

What AIMD costs

  • Overhead grows with the fleet. A surging tenant’s fleet-wide limit never drops below hosts × floor, and the hosts keep sending up to hosts × increase ÷ (1 − decrease) of rejected probes per surging tenant. So one setting works at every scale only if it is sized for the largest fleet. Our first-guess values (+2, ×0.98, floor 50) work at 40 hosts and overload the tracker at 300 (chapter 8). The sidebar shows both numbers live.
  • Too steep a decrease starves surging tenants below the concurrency they are entitled to. Too gentle converges slowly and leaves the tracker overloaded.
  • A short transient. Every host starts wide open, so the first fraction of a second of an attack reaches the tracker before the rates come down.
  • It can’t fix a saturated fleet. If hosts are over their own total limit they shed every tenant before any per-tenant limiter runs (chapter 7).

Choosing the defaults

The defaults are +1, ×0.9, floor 10 (max rate 10k, bucket 100), chosen so one setting is safe at every fleet size up to 1,000 hosts. Each host keeps up to 1 ÷ (1 − 0.9) = 10 TPS of rejected probes per surging tenant, so the tracker stays at or under 85% of capacity in every attack from 20 to 1,000 hosts, and bystanders lose nothing.

The cost is how quickly surging tenants get their slots back. The first burst of rejections in an attack drives each host’s rate to the floor, and an increase of 1 climbs it back slowly: about 3.5 s to reach 95% of their slots at 40 hosts or more, with occasional dips below it at 20-30 hosts. A larger increase (+2, ×0.95) recovers in under 3 s but overloads the tracker past about 700 hosts. Our first-guess values (+2, ×0.98, floor 50) overload it from about 300 hosts (chapter 8).

The tension is structural: this limiter decreases on every rejection, so the first burst crashes the rate. Decreasing at most once per short window, as TCP does once per round trip, should relieve it. Run npm run sweep to repeat the search.

The model and its simplifications

  • Fixed 50 ms ticks. Poisson arrivals per host and tenant. Normal traffic is split unevenly across hosts (log-normal, σ = 0.5); attack traffic is spread evenly.
  • Request durations are exponential with each tenant’s mean. Releasing a slot costs the tracker nothing.
  • The tracker answers within the tick. There are no client retries, so a real attack with retries would look worse.
  • The limiter classes in the Source code tab are the exact modules the simulation runs, called for every request that gets past its host’s total limit.
  • Runs are seeded and deterministic. Each dot on the stage stands for a fixed number of real requests, the same for every tenant (the scale is shown on the stage). Dots are drawn from a separate random stream, so watching never changes the outcome.

How the chapters are checked

Each chapter states what it shows, and the same engine runs headlessly in the test suite to check those claims. For example, the factor-12 bucket must push the tracker past 2× capacity in the major DDoS, and AIMD must recover within a second while bystanders keep at least 98% of their traffic. Run npm test or npm run verify to see them.