Proxy errors
429 Too Many Requests
You are sending requests faster than the site allows. More proxies will spread the load, but reading the Retry-After header and pacing yourself is what actually fixes it.
A 429 is rate limiting. The site understood the request and is telling you to slow down. Unlike a 403, it is usually about timing rather than identity.
- Error
- 429 Too Many Requests
- Comes from
- The target website
What makes 429 different
A 429 is the most cooperative error in this family. The site is not refusing you outright and is not questioning who you are. It is saying that the rate is too high, and it very often tells you exactly how long to wait.
That makes it the easiest to fix properly, and also the easiest to make worse. The instinctive response is to add more proxies and keep the same total rate. Sometimes that works. Often it converts a soft rate limit into a hard block, because the pattern of many addresses all requesting in lockstep is more suspicious than one address going too fast.
Read the response before changing anything
Most services that return 429 also return headers describing their limit. These are worth reading, because they turn guesswork into arithmetic.
Retry-After is the important one. It is either a number of seconds or an HTTP date, and honouring it is usually enough on its own. X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset, where present, let you pace requests so you never hit the limit in the first place.
curl -sS -D - -o /dev/null \
--proxy proxy.example.com:8000 --proxy-user 'USER:PASS' \
https://example.com/api/resource | grep -i -E 'http/|retry-after|x-ratelimit'Fixes, cheapest first
Work down this list in order. The early items cost nothing and solve most cases; the later ones cost money and should not be the first move.
- Honour Retry-After. If the response says wait 30 seconds, wait 30 seconds. Many scrapers ignore this and then wonder why the block escalates.
- Add exponential backoff with jitter. Double the wait after each 429 and add a small random offset so that parallel workers do not retry in unison.
- Lower the concurrency. Fewer simultaneous requests is frequently more effective than more addresses, and it costs nothing.
- Cache aggressively. The fastest request is the one you do not make. Many scrapers re-fetch pages that have not changed.
- Check for an official API. Rate limits on documented APIs are usually far more generous than on the public site, and using one removes the problem rather than working around it.
- Spread across more exit addresses, with randomised pacing per address. This is the point at which buying more proxies genuinely helps.
Pacing that works
A fixed delay between requests is better than none, but it produces a perfectly regular pattern that is trivially identifiable. Real traffic is irregular. Adding jitter, so each wait is the base delay plus a random fraction, costs nothing and makes the pattern far less distinctive.
If you are running many workers, make sure they do not share a clock. Workers that start together and retry together create bursts even when each one individually is well behaved.
import random, time
delay = 1.0
for url in urls:
while True:
r = session.get(url, proxies=proxies)
if r.status_code != 429:
delay = max(1.0, delay / 2)
break
wait = float(r.headers.get("Retry-After", delay))
time.sleep(wait + random.uniform(0, wait * 0.3))
delay = min(delay * 2, 120)
time.sleep(random.uniform(0.5, 2.0))When 429 turns into something worse
Ignoring rate limits has a cost beyond the failed requests. Sites escalate: a 429 becomes a 403, then a block on the address, then sometimes a block on the whole range. An address that has been escalated against does not recover quickly.
This is the argument for pacing rather than brute force. A job that runs slightly slower and finishes is better than one that runs fast for ten minutes and then cannot run at all.
Per address, per account, or per fingerprint
Rate limits are not all counted the same way, and this decides whether more proxies help at all. A limit counted per IP address is spread by rotating addresses. A limit counted per API key, per logged-in account or per browser fingerprint follows you across every address you own, and buying more proxies changes nothing.
You can tell them apart quickly. Make the same request from two different exit addresses, without any shared cookie or token. If the second address starts with a fresh allowance, the limit is per IP. If it arrives already exhausted, something other than the address is identifying you.
When the limit is per account, the honest options are to slow down, to request a higher quota, or to use more than one account where the terms of service genuinely allow it. Rotating addresses against an account-level limit tends to look like evasion, which is how soft limits turn into closed accounts.
The option people skip
Before building rotation infrastructure, check whether the site publishes an API. Documented APIs usually carry limits that are an order of magnitude more generous than the public website, because the operator would rather serve structured requests than parse scraper traffic out of their logs.
An API also removes the entire category of problem. There is no fingerprinting, no bot protection and no fragile HTML to parse, and the rate limit is stated in writing rather than discovered by hitting it. Where a paid tier exists, it is frequently cheaper than the proxy bandwidth it replaces.
Frequently asked questions
Will more proxies fix a 429?
Sometimes, but pacing is the real fix. Many addresses requesting in lockstep often escalates the block instead of avoiding it.
What is Retry-After?
A header giving the seconds, or a date, to wait before retrying. Honouring it is usually enough on its own.
Is 429 the same as being banned?
No. It is a temporary limit, and it is the stage before a ban. Backing off now usually prevents one.
Do rotating proxies avoid rate limits?
They spread requests across addresses, which helps per-address limits. Limits applied per account or per session are unaffected.
Ready to try it?
Create an account, add funds from US$1 and pick your country and proxy type.