Rate limits, retries and switching models in LLM apps
Classify every provider error before you react to it. A per-minute limit means wait and retry, a spent quota or a bad key means switch or pause, and a rejected request should never be sent again as it is.
When an LLM call fails, the right reaction depends on why it failed, and providers don't make that easy to tell. Here is a way to sort errors into a few classes, act on each one, and keep a long job alive on a free tier.
Sort errors by what they mean
A retry loop around every model call is the usual first version. It works until the day the provider says something a retry can't fix. Some errors clear in seconds, some in hours, some never, and treating them alike either burns a day's quota on retries or gives up on a job that only needed to wait.
So classify first and react to the class, never to the raw exception. Read the evidence in a fixed order: your SDK's typed exceptions, then the provider's structured fields (an error code, a quota id, a Retry-After header), and only then the words of the message.
| Class | Typical evidence | Reaction |
|---|---|---|
| Rate limited | a per-minute quota, rate_limit_exceeded, a throttling code |
wait for the provider's hint, retry the same model |
| Quota exhausted | a per-day quota, insufficient_quota |
stop using that connection: switch, or pause |
| Overloaded | 503 "high demand", Anthropic's 529, timeouts | a few retries and a circuit breaker, then switch |
| Bad key | 401, "API key not valid" | stop every model that uses that key |
| No access or no model | 403, 404 | stop that model only |
| Context too long | a context-overflow error | split the work and retry |
| Bad request | 400 with a specific message | never resend it as it is |
| Transient | network errors | a few retries with backoff |
Record the class, the provider's code and the quota id with every call. When a job stalls, that record tells you which limit it actually hit.
A worked example: two 429s with the same words
Gemini sends the same message for its per-minute and its per-day limit: "You exceeded your current quota, please check your plan and billing details." We captured both payloads live on 15 October 2026 and keep them as test fixtures. Only a structured field tells them apart:
gemini_429_per_minute.json quotaId GenerateRequestsPerMinutePerProjectPerModel-FreeTier retryDelay 0s
gemini_429_daily.json quotaId GenerateRequestsPerDayPerProjectPerModel-FreeTier retryDelay 51227s
The first means "try again in a moment". The second means "come back in about 14 hours".
Early in Nehemvix's development, the daily one was read as an overload. The client tripped its breaker and kept retrying a quota that had run out, and on some providers every one of those retries is a counted request. Classified by its quota id, the same error becomes a pause with a reset time the user can see.
Waiting is not failing
A per-minute limit is normal on a free tier. When the provider says how long to wait, wait exactly that, plus a little jitter, and retry the same connection. Count these waits apart from failures, so a slow job on a free tier doesn't look like a broken one.
Without a hint, use exponential backoff with full jitter and a ceiling. Nehemvix waits a random delay of up to 1.5 × 2^attempt seconds, never more than 30, and retries at most four times.
One guard helps. A "per-minute" answer that asks for more than 120 seconds is really a quota problem, so switch rather than sleep.
One limiter per credential and model
Retries alone don't stop 429s if parallel branches pace themselves separately. Each branch thinks it has room, and together they burst. Free tiers make this sharp: when we reviewed the Gemini AI Studio free tier, it allowed 5 requests a minute and 20 a day on Gemini 3.8 Flash.
The fix is to pace per connection: one provider, credential, model and region. Keep one limiter per connection in the process, shared by every job and every branch that uses it, and acquire it before every attempt, retries included. It should:
- skip a connection the provider already called exhausted, until its reset time;
- count requests per day against the provider's own day boundary;
- wait for room in sliding 60-second windows for requests and tokens;
- cap how many calls run at once.
Treat your own counters as advisory. The same key may be used by other tools, so the provider's 429 is always the final word. Say "used via this app today", never "remaining".
Overloads, bad keys and bad requests
Overloaded. Retry a 503 or 529 a few times, but open a circuit breaker when they cluster. Nehemvix skips the primary model for 90 seconds after three overloads within 60 seconds. On 26 October 2026 Gemini 3.8 Flash answered one of our live tests with a 503 "high demand", then a 504 deadline, then its free daily quota: three classes in a row, each needing its own path.
Bad key. A rejected key is bad for every model that uses it. Disable all of them, and if nothing is left, pause the job so the user can fix the key without losing finished work.
Bad request. A 400 with a specific message gets the same answer when you ask again, and you may be billed again. Don't retry it, and after a few identical rejections mark that connection incompatible. Handle known exceptions in code instead. Some models reject a temperature setting such as temperature=0.2, so when a rejection names the temperature, Nehemvix drops it once and retries.
Switch, or pause
When a connection becomes unusable, walk a fallback chain of other connections, on any provider, each with its own spending cap if you want one. When nothing usable is left, pause rather than fail: keep the reason, the reset time and the progress so far, and let the user wait, switch model or publish what is finished. Pause and resume without paying twice shows how finished work survives the pause.
Free tiers make the difference concrete. On 27 October 2026 an OpenRouter free model answered 429 free-models-per-day on the very first call, because the key had used its 50 free requests that day. The job paused at once with the reset time, having spent nothing, and after the reset the full guide ran at $0.
Waiting, switching and pausing are decisions any agent loop has to make in code, and we wrote separately about what starts, finishes and stops an agent loop.
How Nehemvix puts it together
Nehemvix runs on the user's own key, often on a free tier, so these rules live in its model client. The classifier maps every exception to one of ten classes. It reads LangChain's standard exception types first, then Gemini's QuotaFailure.quotaId and RetryInfo.retryDelay, OpenAI's error code, Anthropic's error.type, botocore's Error.Code and the rate-limit headers, and the message text last.
A connection's id is an HMAC-SHA256 fingerprint of the credential and model under a server secret, so it can't be reversed and never sits next to a key. When the user sets no limit, pacing defaults to 60 requests a minute. The fallback chain holds up to three more connections. Every call's row in the cost ledger keeps the class, the provider's code, the quota id and the time spent waiting.
The tests fake only the provider's answers, never the client. A per-minute 429 waits and continues on the same connection, an overload falls back, a systematic bad request pauses instead of degrading the guide, and resuming on a connection that is still exhausted pauses again at no cost. What users see is in limits and pauses.
Checklist
- Capture real error payloads from every provider you support and keep them as fixtures.
- Classify on structured fields first, message text last.
- Count waits apart from failures.
- Share one limiter per credential and model across every parallel branch, and acquire it before every retry.
- Never resend an identical 400.
- Treat a bad key as bad for every model that uses it.
- Make "out of quota" a pause the user can act on, not a failure.
Related reading: the cost side of retries in estimate and cap LLM costs, and where people stay in control in human in the loop for AI agents.
Learning this to build your own agent? AELSHA™ can help you design it.