The system
A fleet of API hosts serves many tenants. Before a host lets a request start, it asks one central concurrency tracker whether that tenant has a free slot. The tracker keeps an exact count of in-flight requests per tenant, which is what makes it the right place to enforce concurrency limits. It is also a single service with a hard ceiling: it can answer only so many checks per second.
Every simulated request goes through the same steps:
- It arrives at one host. If that host is over its total limit, it is shed without regard to tenant.
- The host asks its per-tenant limiter (token bucket or AIMD). If the limiter says no, the request is rejected locally and the tracker never hears about it.
- Otherwise the host sends a check to the tracker, which answers admitted or at limit.
- The answer goes back to the limiter that sent it:
onSuccess()oronBackoff().
Past its capacity the tracker fails a growing share of checks (1 − capacity ÷ load) for
every tenant, and hosts reject those requests (fail closed). So one tenant’s flood takes everyone down
unless the hosts shed it before it reaches the tracker. A failed check also counts as a negative signal,
so AIMD protects the tracker’s total capacity as well as each tenant’s limit.
Why a static token bucket is so hard to tune
A token bucket needs a rate. The usual approach turns each tenant’s concurrency limit into one:
rate per host = factor × tenant max concurrency ÷ fleet size
By Little’s law, concurrency = request rate × request duration. So the rate a tenant can legitimately sustain depends on how long its requests take, and that varies by tenant. Ember sends 8,000 fast 50 ms calls a second while using about 400 of its 1,000 slots. Fjord’s 5 s calls fill 2,000 slots at 400 a second. Traffic also lands unevenly on hosts, and a fixed per-host split can’t follow it.
- Factor too low: tenants are shed while they still have free slots (over-throttling). On a normal day here you need a factor of about 12 before every tenant is served.
- Factor too high: during an attack each surging tenant may send
factor × its limitchecks per second, no matter what the tracker answers. For the major DDoS, anything above about 4.4 overloads the tracker.
For the mild and major attacks here those ranges don’t overlap. How wide the gap is depends on how traffic happens to land on hosts, which no static factor can follow. The band under the factor slider computes both bounds live for the current settings.
Why AIMD converges by itself
AIMD uses the same bucket but lets the tracker set its rate. Each host starts every tenant at a maximum rate that is effectively unlimited. Each at limit or error answer multiplies that host’s rate for that tenant by the decrease factor (0.98). Each admitted request adds the increase (+2 TPS).
The key is amplification. One rejection lowers the rate for every later request from that tenant on that host, so a few thousand rejections a second become hundreds of thousands of local rejections across the fleet. The tracker only has to say no enough times to shape the fleet; it never has to absorb the flood. No one configures a rate: each host settles near its share of what the tracker actually admits.
At steady state each host balances additive increases against multiplicative decreases:
increase × admitted/s ≈ (1 − decrease) × rate × rejected/s
That puts an upper bound on the rejected probes each host keeps sending per surging tenant:
increase ÷ (1 − decrease), which is 100 TPS with the defaults.
What AIMD costs
- Overhead grows with the fleet. A surging tenant’s fleet-wide limit never drops below
hosts × floor, and the hosts keep sending up tohosts × increase ÷ (1 − decrease)of rejected probes per surging tenant. At 300 hosts the defaults overload the tracker again (chapter 8), so the floor and increase must shrink as the fleet grows. The sidebar shows both numbers live. - Too steep a decrease starves surging tenants below the concurrency they are entitled to. Too gentle converges slowly and leaves the tracker overloaded.
- A short transient. Every host starts wide open, so the first fraction of a second of an attack reaches the tracker before the rates come down.
- It can’t fix a saturated fleet. If hosts are over their own total limit they shed every tenant before any per-tenant limiter runs (chapter 7).
The model and its simplifications
- Fixed 50 ms ticks. Poisson arrivals per host and tenant. Normal traffic is split unevenly across hosts (log-normal, σ = 0.5); attack traffic is spread evenly.
- Request durations are exponential with each tenant’s mean. Releasing a slot costs the tracker nothing.
- The tracker answers within the tick. There are no client retries, so a real attack with retries would look worse.
- The limiter classes in the Source code tab are the exact modules the simulation runs, called for every request that gets past its host’s total limit.
- Runs are seeded and deterministic. The dots on the stage are sampled with a separate random stream, so watching never changes the outcome.
How the chapters are checked
Each chapter states what it shows, and the same engine runs headlessly in the test suite to check those
claims. For example, the factor-12 bucket must push the tracker past 2× capacity in the major DDoS, and
AIMD must recover within a second while bystanders keep at least 98% of their traffic. Run
npm test or npm run verify to see them.