Rate Limits¶
Rate limits keep the models fast for everyone. They cap how much each project can send to each model:
- Requests per minute (RPM): how many requests you can send in a minute.
- Tokens per minute (TPM): how many tokens you can use in a minute, input and output together.
Limits are set for each model you enable in a project, so each model is counted separately and projects don't share limits. They reset at the start of every minute.
When a request starts, it reserves its input tokens plus its max_tokens against your TPM. The tokens it doesn't use are returned when it finishes.
Rate limit tiers¶
Your limits depend on your account's tier. New accounts start on the first tier, and higher tiers have higher limits.
When you enable a model, it gets the limits of your tier, in every project of your account. When your tier changes, the models you've enabled move to the new limits.
Check your current limits¶
Every response includes headers with your limits for the model, what's left in the current minute, and when the limits reset:
| Header | What it tells you |
|---|---|
x-ratelimit-limit-requests |
Your RPM limit |
x-ratelimit-remaining-requests |
Requests left in the current minute |
x-ratelimit-reset-requests |
Time until the request limit resets, for example 12s |
x-ratelimit-limit-tokens |
Your TPM limit |
x-ratelimit-remaining-tokens |
Tokens left in the current minute |
x-ratelimit-reset-tokens |
Time until the token limit resets |
Handling 429 errors¶
When you go over a limit, the API returns 429 with the code rate_limit_exceeded and a Retry-After header with the number of seconds to wait. A busy model can also return 429 while you're within your limits.
To handle 429 well:
- Back off and retry. Wait for
Retry-After. Without it, wait 1 second before the first retry and double the wait each time, with some random jitter. - Control concurrency. Send requests through a queue or a semaphore, so you never send more at once than your limits allow.
- Combine small requests. Where you can, send fewer, larger requests instead of many small ones. This uses less of your RPM.
- Keep
max_tokensrealistic. It reserves TPM until the request finishes, so a value close to what you need leaves room for more requests.
Request higher rate limits¶
Contact support with your project, the model and the limits you need. We can move your account to a higher tier or set custom limits for your workload.