Skip to content
Get Started
G2G2
4.7/5
Excellent
Trustpilot

What Is Rate Limiting?

Rate limiting is a server's cap on how many requests it accepts from one client in a period of time. A client that goes over the limit is slowed down or refused, usually with an HTTP 429 Too Many Requests response.

Try Now for $4.99Continue with Google
  • Session Control
  • High Speed
  • Privacy First
Best fit for Rate limitingRotating Residential Proxiesfrom$0.63/GB
how it works

How rate limiting works

The server counts requests per key: an IP address, an API key, an account, a session cookie or a mix. Common algorithms are the fixed window (100 requests per minute, reset on the minute), the sliding window (the last 60 seconds at any moment) and the token bucket, which refills at a steady rate and allows short bursts up to the bucket's size.

When a client goes over, the standard response is status 429 Too Many Requests, defined in RFC 6585. It may include a Retry-After header holding either a number of seconds (Retry-After: 120) or an HTTP date to wait until. Some APIs also send headers with the remaining quota. Sites that prefer not to reveal their limits answer with a CAPTCHA, a 403 or a slower response instead.

why it matters

Why rate limiting matters for proxy users

Limits are applied per IP more often than per anything else, so the IPs you route through set your real throughput. High concurrency from one IP reaches the limit fast, and ignored 429s add negative history to the IP's reputation.

Rate limiting: what it means for you
  • Honour Retry-After exactly. When it is missing, back off exponentially (1, 2, 4, 8 seconds) with some random jitter.
  • Keep the request rate per IP and per site modest, and spread volume across a pool instead of pushing one address harder.
  • Cache what you have already fetched and request only what changed.
  • Treat a rise in 429s as a signal to slow the whole job down, not to retry faster. The guide to fixing proxy errors covers 429 alongside other status codes.
try it

Handle 429 and Retry-After in Python

When Retry-After is an HTTP date, the sample falls back to exponential backoff. CleanProxies rotating residential proxies can rotate the exit IP on every request across a 50M+ pool, which keeps the load on any single IP low for large public data jobs.

Retry with Retry-After and backoffPython
import random
import time

import requests

proxies = {"https": "http://USER:PASS@PROXY_HOST:PORT"}


def get(url, attempts=5):
    for attempt in range(attempts):
        r = requests.get(url, proxies=proxies, timeout=30)
        if r.status_code != 429:
            return r
        retry_after = r.headers.get("Retry-After", "")
        wait = int(retry_after) if retry_after.isdigit() else 2 ** attempt + random.random()
        time.sleep(wait)
    return r

Rotating residential proxies from $0.63/GB

The CleanProxies proxy that fits best here, with a $4.99 trial to test it first.

Go deeper

Guides from the blog

Setup guides and best practices from the CleanProxies blog.

Get the latest insightsAll Blogs

Related terms

Terms that come up alongside Rate limiting.

All glossary terms

Rate limiting FAQs

Straight answers about Rate limiting.

The server received more requests from your client than it allows in a time window and refused this one. Wait before retrying, for as long as the Retry-After header says if there is one.

Got more questions?All FAQs