An API rate limit incident checklist should stop a retry storm before it turns a temporary limit into a wider outage. Confirm what is failing, respect provider signals, reduce demand, and restore traffic gradually while preserving evidence.
Confirm the limit and its scope
Capture the HTTP status, provider request ID, response headers, endpoint, account or token, deployment version, and first failure time. A 429 response often indicates a limit, but provider-specific headers can distinguish global, endpoint, concurrency, or resource limits. Stripe, for example, documents a response header that identifies several limit reasons and notes that some 429 responses can instead be object lock timeouts (Stripe Docs).
Check whether failures affect one endpoint, one credential, one customer, or all traffic. Compare request rate and concurrency with the normal baseline. Look for a batch job, new release, loop, queue replay, missing cache, or duplicate webhook processing that changed the traffic shape.
Stop amplification
Pause nonessential batch work and cap queue consumers. Disable unbounded immediate retries. Preserve critical user actions by prioritizing them over reports, migrations, polling, and background refreshes. Do not rotate credentials merely to escape a limit; that can spread traffic and obscure the root cause.
GitHub advises waiting for the reset indicated by rate-limit information and using an exponentially increasing delay for repeated secondary-limit failures, with a maximum retry count (GitHub Docs). Stripe likewise recommends exponential backoff with randomness to reduce synchronized retries (Stripe Docs).
Apply controlled recovery
Honor Retry-After or provider reset information when present. Otherwise use bounded exponential backoff with jitter. Keep idempotency protections in place for write requests so a retry does not duplicate a charge, order, or other side effect. If the provider distinguishes concurrency from request rate, reduce simultaneous in-flight work as well as requests per second.
Recover in stages:
- allow a small percentage of critical traffic;
- verify the 429 rate and latency are falling;
- restore normal interactive traffic;
- resume background queues with a strict cap;
- pause again if the error rate rises.
Measure the real workload
Count attempts, successful responses, retries, unique operations, and queue depth separately. A falling request count with a rising queue may mean the limiter is working while customer delay is still increasing. Track the age of the oldest job and the number of actions abandoned after the retry limit.
Review caching, pagination, polling intervals, webhook use, and duplicate calls. For provider APIs with multiple limits, create alerts for both sustained rate and concurrent requests. Keep capacity assumptions in configuration rather than scattered hard-coded delays.
Document and prevent recurrence
Record the trigger, affected endpoints, provider signals, mitigation time, customer impact, and safe recovery rate. Add a load-shedding rule, bounded retry policy, idempotency test, and dashboard if any were missing. If a planned event may raise traffic, coordinate with the provider instead of discovering the boundary during production load.
Quick completion test
The incident can close when critical operations succeed, retries are bounded, queues are draining, request rate remains below the observed threshold, no duplicate side effects appeared, and a prevention owner has a dated action list.
Sources
- Rate limits for the REST API — GitHub
- Rate limits — Stripe