How to Rotate Proxies in Python (requests, httpx, Scrapy)

Team CleanProxies

Oct 29, 202621 Min Read

By the CleanProxies team. Every code block below was run on Python 3.12 with requests 2.34, httpx 0.28 and Scrapy 2.19 (see "How we tested").

To rotate proxies in Python, either send every request to a rotating gateway that picks the exit IP for you, or keep a list of static proxies and choose a different one per request, resting any proxy that gets a 403/429 and retiring any that keeps failing to connect. This guide gives you tested code for both methods in requests, httpx (async) and Scrapy, plus the failure handling that decides whether rotation helps you or just spreads errors across more IPs.

TL;DR

Rotating residential or mobile proxies: you don't need rotation code. Point your client at one gateway (resipro.cleanproxies.com:5555 or mobile.cleanproxies.com:5555) with a rotating line and the gateway assigns the exit IP per request. Use sticky lines when a task needs one IP for a while.
Static ISP proxies: rotate over your list in your own code: round-robin, cooldown on 403/429/503, retirement after repeated connection or 407 failures, and a wait (not a crash) when every IP is resting.
Retries make the difference. On our test bench, a naive random picker with no retries completed 55–71 of 200 requests. The rotators below completed 200/200 on every run with the same broken proxies in the pool.
CleanProxies gives you both: US ISP proxies as low as $0.75/IP per month (10,000-IP tier) with unlimited bandwidth, and rotating residential as low as $0.63/GB (10,000 GB tier) across 190+ countries. Test either with the $4.99 trial.

Gateway rotation vs list rotation at a glance

	Gateway rotation	List rotation (your code)
Best product	Rotating residential or mobile	Static ISP proxies
Who changes the IP	The provider's gateway	Your rotator
Code you need	One proxy URL	Pool, picker, cooldowns, retries
Same IP for a login flow	Sticky line (5, 10, 15, 30, 60 min or 24 h)	Pin one proxy to that account or job
Billing	Per GB, data never expires	Per IP per month, unlimited bandwidth
Protocol	HTTP(S)	HTTP(S) or SOCKS5
Authentication	Username:password	Username:password or IP whitelist
Typical use	Wide crawls, many targets, geo checks	Steady jobs on known sites, accounts, monitoring

Many teams use both: a gateway for wide discovery crawls, and a fixed set of clean ISP IPs for the jobs where a stable, trusted address matters. If you're unsure which mode a task needs, read rotating vs sticky sessions first.

Before you start: get your proxy list

Every example reads the format the CleanProxies dashboard exports: one proxy per line as host:port:username:password.

ISP proxies: download the IP list for your order. Each line is one dedicated IP.
Rotating residential / mobile: open the generator in the proxy manager, pick a country (residential can also target state, city or ASN; mobile targets country and ASN), choose Rotating or a sticky duration, and export the lines. The username carries the targeting and session you picked, so copy it exactly and don't edit it by hand.

If your list is in a different shape, convert it with the free proxy list formatter. For first-time setup in Python, the Python requests integration guide and the httpx guide cover the basics; this post is about rotating across many requests.

pip install requests httpx scrapy

Method 1: gateway rotation (no rotation code)

With a rotating residential or mobile line, the gateway hands out the exit IP. Your code stays short:

from rotator import proxy_url    # from Method 2 below; it URL-encodes the username and password
import requests

proxy = proxy_url("resipro.cleanproxies.com:5555:YOUR_USERNAME:YOUR_PASSWORD")   # one generator line

for _ in range(5):
    r = requests.get("https://api.ipify.org?format=json",
                     proxies={"http": proxy, "https": proxy}, timeout=(5, 20))
    print(r.json()["ip"])

If you keep seeing the same IP. HTTP clients reuse connections: a requests.Session() or an httpx.Client keeps the connection (and, for HTTPS through a proxy, the CONNECT tunnel) open between requests. If repeated requests through a rotating line come back from the same address, check whether your client is reusing a connection; a plain requests.get() per call, or a Connection: close header, gives each request its own connection. For a sticky task, do the opposite: use a sticky line and one session.

Sticky sessions for multi-step tasks. Logins, carts and paginated searches break when the IP changes mid-flow. Generate sticky lines (5, 10, 15, 30 or 60 minutes, or 24 hours) and give each task its own line. A sketch, where tasks and run_task are your own code:

import itertools
from rotator import proxy_url

sticky = [proxy_url(l) for l in open("sticky.txt") if l.strip()]
for task, proxy in zip(tasks, itertools.cycle(sticky)):
    run_task(task, proxy=proxy)        # the whole task runs on one exit IP

Residential exits are real household connections, so a device can go offline before the window ends. Write tasks so they can resume from the last completed step.

Method 2: rotate a static ISP list with requests

With static ISP proxies you own the rotation. Save this as rotator.py; the httpx example imports from it too. It does what free-list tutorials skip:

Round-robin, so load spreads evenly across your IPs (random choice clusters).
Cooldown on 403/429/503. The IP is fine; the site wants it to slow down. It honours Retry-After in both forms (seconds or an HTTP date), keeps 0 as zero, and never shortens a cooldown that's already running.
Retirement on connect errors and 407. These mean an unreachable proxy or wrong credentials, not a block. After max_failures consecutive failures (any HTTP answer resets the count), the proxy leaves the pool.
Wait, don't crash. If every IP is cooling down, get() sleeps until the first one is ready, up to max_wait. It only raises when nothing is left.
Timeouts and errors are classified. A read timeout is the target being slow, so the request is retried without blaming the proxy; 500/502/504 are retried; other 4xx responses are returned to you because retrying won't fix them. is_valid lets you reject a 200 that's actually a challenge or empty page.
Safe credentials. Usernames and passwords are URL-encoded, so a #, ? or % in a password doesn't break the proxy URL.

import email.utils, itertools, random, threading, time
from urllib.parse import quote

import requests


def proxy_url(line):
    """host:port:user:pass (the dashboard export format) -> http://user:pass@host:port, safely encoded."""
    host, port, user, password = line.strip().split(":", 3)
    return f"http://{quote(user, safe='')}:{quote(password, safe='')}@{host}:{port}"


def load_proxies(path):
    with open(path) as f:
        return [proxy_url(line) for line in f if line.strip()]


def retry_after_seconds(value, default):
    """Retry-After is either delay-seconds or an HTTP date (RFC 9110)."""
    if value is None:
        return default
    value = value.strip()
    if value.isdigit():
        return int(value)                        # "0" stays 0
    try:
        return max(0.0, email.utils.parsedate_to_datetime(value).timestamp() - time.time())
    except (TypeError, ValueError):
        return default


class NoProxiesLeft(RuntimeError):
    pass


class ProxyRotator:
    """Round-robin over static proxies: cooldowns for blocked IPs, retirement for broken ones."""

    def __init__(self, proxies, cooldown=60, max_failures=3, max_wait=120):
        self.proxies = list(proxies)
        self.cooldown = cooldown
        self.max_failures = max_failures        # consecutive connection/auth failures before retiring
        self.max_wait = max_wait                 # longest we wait for a cooled-down proxy
        self.failures = {p: 0 for p in self.proxies}
        self.ready_at = {p: 0.0 for p in self.proxies}
        self._cycle = itertools.cycle(self.proxies)
        self._lock = threading.Lock()

    def acquire(self):
        """Return (proxy, 0) if one is ready now, or (None, seconds until the next one is)."""
        with self._lock:
            if not self.proxies:
                raise NoProxiesLeft("every proxy was retired")
            now = time.monotonic()
            for _ in range(len(self.proxies)):
                p = next(self._cycle)
                if self.ready_at[p] <= now:
                    return p, 0.0
            return None, min(self.ready_at.values()) - now

    def get(self):
        deadline = time.monotonic() + self.max_wait
        while True:
            proxy, wait = self.acquire()
            if proxy:
                return proxy
            if time.monotonic() + wait > deadline:
                raise NoProxiesLeft(f"all proxies cooling down for another {wait:.0f}s")
            time.sleep(wait)

    def blocked(self, proxy, seconds=None):
        """The site refused this IP (403/429/503): rest it. Never shortens an existing cooldown."""
        with self._lock:
            if proxy in self.ready_at:
                until = time.monotonic() + (self.cooldown if seconds is None else seconds)
                self.ready_at[proxy] = max(self.ready_at[proxy], until)
            if proxy in self.failures:
                self.failures[proxy] = 0         # it answered, so the connection works

    def failed(self, proxy):
        """Could not connect, or the proxy rejected our credentials (407)."""
        with self._lock:
            if proxy not in self.failures:
                return
            self.failures[proxy] += 1
            if self.failures[proxy] >= self.max_failures:
                for d in (self.failures, self.ready_at):
                    del d[proxy]
                self.proxies.remove(proxy)
                self._cycle = itertools.cycle(self.proxies)

    def ok(self, proxy):
        with self._lock:
            if proxy in self.failures:
                self.failures[proxy] = 0


BLOCKED = {403, 429, 503}
RETRY_LATER = {500, 502, 504}


def fetch(url, rotator, attempts=5, timeout=(5, 20), is_valid=lambda r: True):
    """GET url through the rotator. is_valid(response) lets you reject CAPTCHA/empty pages."""
    for attempt in range(attempts):
        proxy = rotator.get()
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=timeout)
        except (requests.exceptions.ProxyError, requests.exceptions.ConnectTimeout):
            rotator.failed(proxy)                # the proxy itself is unreachable
        except (requests.exceptions.ReadTimeout, requests.exceptions.ConnectionError):
            pass                                 # slow or dropped target: retry, don't blame the proxy
        else:
            if r.status_code == 407:
                rotator.failed(proxy)            # wrong credentials, or the order is in IP-auth mode
            elif r.status_code in BLOCKED:
                rotator.blocked(proxy, retry_after_seconds(r.headers.get("Retry-After"), None))
            elif r.status_code in RETRY_LATER:
                rotator.ok(proxy)
            elif r.status_code < 400 and not is_valid(r):
                rotator.blocked(proxy)           # soft block: 200 with a challenge page
            else:
                rotator.ok(proxy)
                return r                         # success, or a 4xx that retrying won't fix
        time.sleep(min(2 ** attempt, 30) * random.uniform(0.5, 1.0))   # backoff with jitter
    raise RuntimeError(f"gave up on {url} after {attempts} attempts")

Usage:

from rotator import ProxyRotator, fetch, load_proxies

urls = [f"https://example.com/products?page={n}" for n in range(1, 51)]   # pages you're allowed to fetch
rotator = ProxyRotator(load_proxies("proxies.txt"), cooldown=60, max_failures=3)
for url in urls:
    r = fetch(url, rotator, is_valid=lambda r: b"captcha" not in r.content.lower())
    print(url, r.status_code, len(r.content))

The lock makes the rotator safe to share across a ThreadPoolExecutor. Backoff waits of 1, 2, 4, 8… seconds (capped at 30), each scaled by a random 0.5–1.0, keep a burst of failures from turning into a burst of retries. If every proxy returns 407, check the credentials, or whether the order is in IP-whitelist mode; how to fix proxy errors covers 407, 403, 429 and SSL errors in depth.

Method 3: async rotation with httpx

httpx binds the proxy to the client, not the request, so this keeps one AsyncClient per proxy and reuses the rotator from rotator.py. httpx 0.28 removed the old proxies= argument; use proxy=.

import asyncio, random
import httpx

from rotator import BLOCKED, RETRY_LATER, NoProxiesLeft, ProxyRotator, load_proxies, retry_after_seconds


async def get_proxy(rotator):
    """Async version of rotator.get(): wait for a cooled-down proxy without blocking the event loop."""
    loop = asyncio.get_running_loop()
    deadline = loop.time() + rotator.max_wait
    while True:
        proxy, wait = rotator.acquire()
        if proxy:
            return proxy
        if loop.time() + wait > deadline:
            raise NoProxiesLeft(f"all proxies cooling down for another {wait:.0f}s")
        await asyncio.sleep(wait)


async def crawl(urls, proxy_urls, concurrency=20, attempts=5):
    rotator = ProxyRotator(proxy_urls, cooldown=60, max_failures=3)
    # httpx binds a proxy to a client, so keep one client (and connection pool) per proxy
    clients = {p: httpx.AsyncClient(proxy=p, timeout=httpx.Timeout(20, connect=5)) for p in proxy_urls}
    sem = asyncio.Semaphore(concurrency)

    async def fetch(url):
        for attempt in range(attempts):
            async with sem:
                proxy = await get_proxy(rotator)          # pick only when a slot is free
                try:
                    r = await clients[proxy].get(url)
                except (httpx.ProxyError, httpx.ConnectError, httpx.ConnectTimeout):
                    rotator.failed(proxy)
                except (httpx.ReadTimeout, httpx.ReadError, httpx.RemoteProtocolError):
                    pass                                   # target side problem: retry elsewhere
                else:
                    if r.status_code == 407:
                        rotator.failed(proxy)
                    elif r.status_code in BLOCKED:
                        rotator.blocked(proxy, retry_after_seconds(r.headers.get("Retry-After"), None))
                    elif r.status_code in RETRY_LATER:
                        rotator.ok(proxy)
                    else:
                        rotator.ok(proxy)
                        return r
            await asyncio.sleep(min(2 ** attempt, 20) * random.uniform(0.5, 1.0))
        raise RuntimeError(f"gave up on {url}")

    try:
        # return_exceptions: one failed URL doesn't cancel the rest or race the client cleanup below
        return await asyncio.gather(*(fetch(u) for u in urls), return_exceptions=True)
    finally:
        await asyncio.gather(*(c.aclose() for c in clients.values()))

Three details are easy to get wrong in async code. The proxy is picked inside the semaphore, so a task that waited in the queue never uses an IP that was rested or retired while it waited. Waiting for a cooled-down IP uses asyncio.sleep, not time.sleep, so the event loop keeps running. And return_exceptions=True means one URL that gives up doesn't abort the batch while other requests are still using the clients.

Method 4: a Scrapy downloader middleware

Scrapy's built-in HttpProxyMiddleware reads request.meta["proxy"] and sends the Proxy-Authorization header. This middleware fills that key, classifies failures the same way as rotator.py, and waits without blocking (Scrapy's default asyncio reactor allows async middleware methods) when every IP is resting:

import asyncio, random, time
from urllib.parse import quote

from scrapy.exceptions import IgnoreRequest


class RotatingProxyMiddleware:
    """Rotate PROXY_LIST per request: rest IPs on 403/429/503, retire them after repeated
    connection or 407 failures, and wait (without blocking) when every IP is cooling down."""

    def __init__(self, proxies, max_retries, cooldown, max_failures):
        self.proxies = proxies
        self.max_retries = max_retries
        self.cooldown = cooldown
        self.max_failures = max_failures
        self.ready_at = {p: 0.0 for p in proxies}
        self.failures = {p: 0 for p in proxies}

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        proxies = []
        with open(s.get("PROXY_LIST")) as f:
            for line in f:
                if line.strip():
                    host, port, user, password = line.strip().split(":", 3)
                    proxies.append(f"http://{quote(user, safe='')}:{quote(password, safe='')}@{host}:{port}")
        return cls(proxies, s.getint("PROXY_MAX_RETRIES", 5), s.getint("PROXY_COOLDOWN", 60),
                   s.getint("PROXY_MAX_FAILURES", 3))

    async def _pick(self, exclude=None):
        while True:
            if not self.proxies:
                raise IgnoreRequest("every proxy was retired")
            now = time.monotonic()
            ready = [p for p in self.proxies if p != exclude and self.ready_at[p] <= now]
            if ready:
                return random.choice(ready)
            if len(self.proxies) == 1 and self.proxies[0] == exclude and self.ready_at[exclude] <= now:
                return exclude
            await asyncio.sleep(max(0.05, min(self.ready_at[p] for p in self.proxies) - now))

    async def process_request(self, request, spider=None):
        if "rotating_proxy" not in request.meta:
            proxy = await self._pick()
            request.meta["proxy"] = proxy
            # Scrapy's HttpProxyMiddleware strips user:pass out of meta["proxy"], so keep our own copy
            request.meta["rotating_proxy"] = proxy

    async def process_response(self, request, response, spider=None):
        proxy = request.meta.get("rotating_proxy")
        if response.status == 407:
            self._failed(proxy)
        elif response.status in (403, 429, 503):
            wait = response.headers.get(b"Retry-After", b"").decode()
            seconds = int(wait) if wait.isdigit() else self.cooldown
            if proxy in self.ready_at:
                self.ready_at[proxy] = max(self.ready_at[proxy], time.monotonic() + seconds)
        else:
            if proxy in self.failures:
                self.failures[proxy] = 0
            return response
        return await self._retry(request, response)

    async def process_exception(self, request, exception, spider=None):
        self._failed(request.meta.get("rotating_proxy"))
        return await self._retry(request, None)

    def _failed(self, proxy):
        if proxy not in self.failures:
            return
        self.failures[proxy] += 1
        if self.failures[proxy] >= self.max_failures:
            self.proxies.remove(proxy)
            del self.failures[proxy], self.ready_at[proxy]

    async def _retry(self, request, response):
        tries = request.meta.get("proxy_retries", 0)
        if tries >= self.max_retries:
            return response                     # None lets Scrapy log the original error
        proxy = await self._pick(exclude=request.meta.get("rotating_proxy"))
        retry = request.replace(dont_filter=True)
        retry.meta.update(proxy=proxy, rotating_proxy=proxy, proxy_retries=tries + 1)
        return retry

Enable it in settings.py with a priority below 750, so it runs before Scrapy's proxy middleware, and switch off Scrapy's own retries so one component owns the retry budget:

DOWNLOADER_MIDDLEWARES = {"myproject.middlewares.RotatingProxyMiddleware": 610}
RETRY_ENABLED = False          # this middleware does the retries
PROXY_LIST = "proxies.txt"
PROXY_MAX_RETRIES = 5
PROXY_COOLDOWN = 60
PROXY_MAX_FAILURES = 3

The bug we hit while testing. Our first version tracked cooldowns by request.meta["proxy"]. Scrapy's HttpProxyMiddleware rewrites that value without the user:pass part once it moves the credentials into the header. So when a request failed, the rotator looked up a proxy URL it had never seen, never rested the bad proxy, and kept retrying through it. That version completed only 38 and 46 of 50 requests in two runs. Keeping our own copy in meta["rotating_proxy"] fixed it. If you write your own middleware, track proxies under your own meta key.

One more version note: Scrapy deprecated start_requests() in 2.13 and removed it in 2.16 (release notes). On current Scrapy, a spider that only defines start_requests() crawls nothing, which is what happened to our first test spider. Define async def start(self) (or both, if you also run Scrapy older than 2.13). The middleware methods take spider=None so they work before and after Scrapy stopped passing the spider argument.

What our test bench showed

Rotator (200 logical requests per run, 3 runs)	Completed	Time per run
Naive `random.choice`, no retries	55, 70, 71	–
requests + `ProxyRotator`, sequential	200, 200, 200	10.2–10.7 s
httpx async, concurrency 20	200, 200, 200	1.2–2.5 s
Scrapy middleware (final version)	200, 200, 200	1.5–1.6 s
Scrapy middleware, first version with the meta bug (50 requests, 2 runs)	38, 46	–

"Completed" counts logical requests that ended in a 200, including the ones that needed retries. The bench has a healthy proxy, a proxy the site always rate-limits and a proxy that's down, so two of three proxies are unusable at any moment.

What the numbers mean:

Rotation without retries just spreads errors around. The naive picker failed about two-thirds of the time, roughly the share of bad proxies in the pool.
Classifying failures is what makes it work. Resting the rate-limited proxy and retiring the dead one sent every remaining request through the healthy IP.
Concurrency made the async and Scrapy versions several times faster than the sequential requests loop, mostly because requests and backoff waits overlap; the clients and connection handling also differ.
These are local numbers for comparing code, not a measurement of any proxy network.

Retries, bans and when rotation won't help

Response	What it usually means	What the code above does
200 with the expected content	Success	Returns it, resets the failure count
200 with a CAPTCHA or empty page	Soft block	`is_valid` fails, so the IP rests
403 / 429 / 503	Refused or rate-limited for this IP	Rests the IP (`Retry-After` if sent), retries elsewhere
407	Proxy rejected your credentials	Counts a failure; retires after 3 in a row
500 / 502 / 504	Server or upstream error	Retries with backoff
Other 4xx (404, 410…)	The request itself is wrong	Returns it; retrying won't help
Connect error / connect timeout	Proxy unreachable	Counts a failure; retires after 3 in a row
Read timeout	Target is slow	Retries without blaming the proxy

Rotation is not a fix for every 403. If every IP gets blocked within a few requests, the problem is usually the request: missing headers, a broken cookie flow or simply too high a rate. A new IP also doesn't change your client's TLS or HTTP fingerprint, which is the same for every proxy you rotate through. Slow down, fix the request, then rotate. Respect each site's terms and robots rules, and collect public data only.

IP quality matters too. Rotating through flagged, shared addresses means you'll spend your retries on IPs that were burned before you got them. That's why CleanProxies ISP IPs are 100% dedicated, never shared, with a 0 fraud score on Tier-1 US carrier ASNs, 100Gbps network ports, <20ms latency and unlimited bandwidth and threads. You can check any IP yourself with the proxy checker before it goes into your pool.

A quick health check before you rotate

Run this once over a new list. It sends one request per proxy and prints the exit IP, so you can spot dead lines and duplicates before they reach your rotator:

import requests
from rotator import load_proxies

for p in load_proxies("proxies.txt"):
    host = p.rsplit("@", 1)[1]
    try:
        r = requests.get("https://api.ipify.org?format=json", proxies={"http": p, "https": p}, timeout=(5, 15))
        print(host, r.status_code, r.json().get("ip") if r.ok else "")
    except requests.RequestException as e:
        print(host, "FAILED", type(e).__name__)

Sizing your pool

A rough rule for list rotation: pool size ≈ requests per minute ÷ the rate one IP can safely sustain on that site. If a site tolerates about 6 requests per minute per IP and you need 600, plan on at least 100 IPs, plus headroom for cooldowns. Our guide how many proxies do I need walks through this with a sizing script and real page weights for GB-based plans.

Testing checklist before you scale

Export a small list and convert it with the proxy list formatter if needed.
Run the health check above and confirm each exit IP.
Check reputation with the proxy checker.
Run your rotator at low concurrency against the real target and log status codes per proxy.
Watch the share of 403/429 responses. If it climbs, lower the rate before you add IPs.
Only then scale concurrency and the pool size.

The cheapest way to do steps 1–5 on real infrastructure is the $4.99 CleanProxies trial: 5 ISP IPs for 1 hour, or 100 MB of residential or mobile data. Start the trial.

How we tested

We ran every example against a local bench on one machine: a target site on port 9000 and authenticated HTTP proxies on ports 8001 (healthy) and 8002 (the target always answers 429 with Retry-After: 1), with nothing listening on 8003 (connection refused). proxies.txt lists all three as 127.0.0.1:<port>:user:pass, and each test sends 200 requests to http://127.0.0.1:9000/ip, which echoes which proxy the request came through. Here is the bench, so you can reproduce the numbers:

"""Local test bench: a target site + 3 authenticated forward proxies.
proxy A (8001) healthy, proxy B (8002) gets 429 from the target, proxy C (8003) is down (nothing listening)."""
import base64, http.server, socketserver, threading, urllib.request, json, sys

AUTH = "Basic " + base64.b64encode(b"user:pass").decode()

class Target(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        exit_ip = self.headers.get("X-Exit", "direct")
        if exit_ip == "10.0.0.2":                      # this "IP" is rate limited by the site
            self.send_response(429); self.send_header("Retry-After", "1"); self.end_headers(); return
        body = json.dumps({"ip": exit_ip, "path": self.path}).encode()
        self.send_response(200); self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)
    def log_message(self, *a): pass

def make_proxy(exit_ip):
    class Proxy(http.server.BaseHTTPRequestHandler):
        def do_GET(self):
            if self.headers.get("Proxy-Authorization") != AUTH:
                self.send_response(407); self.send_header("Proxy-Authenticate", 'Basic realm="p"'); self.end_headers(); return
            req = urllib.request.Request(self.path, headers={"X-Exit": exit_ip})
            try:
                r = urllib.request.urlopen(req, timeout=5); code, data, hdrs = r.status, r.read(), r.headers
            except urllib.error.HTTPError as e:
                code, data, hdrs = e.code, e.read(), e.headers
            self.send_response(code)
            for k in ("Content-Type", "Retry-After"):
                if hdrs.get(k): self.send_header(k, hdrs[k])
            self.send_header("Content-Length", str(len(data))); self.end_headers(); self.wfile.write(data)
        def log_message(self, *a): pass
    return Proxy

class TS(socketserver.ThreadingMixIn, http.server.HTTPServer):
    daemon_threads = True; request_queue_size = 256

def start():
    servers = [TS(("127.0.0.1", 9000), Target), TS(("127.0.0.1", 8001), make_proxy("10.0.0.1")),
               TS(("127.0.0.1", 8002), make_proxy("10.0.0.2"))]
    for s in servers: threading.Thread(target=s.serve_forever, daemon=True).start()

if __name__ == "__main__":
    start(); print("bench up", flush=True); threading.Event().wait()

The test drivers are three short loops: call fetch() 200 times; asyncio.run(crawl([url] * 200, ...)); and a Scrapy spider whose async def start() yields 200 requests with RotatingProxyMiddleware enabled and RETRY_ENABLED = False.

Key stats

429 Too Many Requests is defined in RFC 6585 §4; servers may send Retry-After to say how long to wait.
407 Proxy Authentication Required and Retry-After (delay-seconds or an HTTP date) are defined in RFC 9110.
httpx 0.28.0 removed the deprecated proxies argument; use proxy or mounts (httpx changelog).
Scrapy 2.16 removed start_requests(), deprecated since 2.13 (Scrapy release notes); its proxy support reads request.meta["proxy"] (HttpProxyMiddleware docs).
Our bench: naive rotation completed 55–71 of 200 requests; the classified-retry rotators completed 200/200 in all 9 runs across requests, httpx and Scrapy.

CleanProxies pricing for rotation

Product	How you rotate	Price
ISP proxies	Your code, over dedicated static IPs	As low as $0.75/IP per month (10,000 IPs), unlimited bandwidth
Rotating residential	Gateway: per request or sticky up to 24 h, 190+ countries	As low as $0.63/GB (10,000 GB), data never expires
Mobile proxies	Gateway: per request or sticky, 150+ countries	As low as $0.79/GB (10,000 GB)

CleanProxies offers the cheapest pricing in the market, instant delivery, 99.9% uptime and 24/7 support. See every tier on the pricing page, compare providers on /compare, or read more about proxies for web scraping and the Scrapy integration.

Ready to rotate? View pricing · Get ISP proxies · Check an IP with the proxy checker

Frequently Asked Questions

No. A rotating residential or mobile gateway gives each new connection a new exit IP, so you send every request to one endpoint. You only write rotation code when you manage a list of static ISP proxies yourself.

Your client is reusing its connection. requests.Session and httpx.Client keep connections open, so several requests travel through the same tunnel and exit IP. For a new IP per request, call requests.get without a session or send a Connection: close header.

Rotate per request for independent page fetches. Keep one IP for anything with state, such as logins, carts or paginated searches: use a sticky line (5 to 60 minutes or 24 hours) on residential or mobile, or pin one static ISP proxy to that task.

Rest that IP for the number of seconds in the Retry-After header (or a fixed cooldown), retry the request through another proxy with exponential backoff, and lower your request rate if 429s keep rising.

Pass proxy='http://user:pass@host:port' to httpx.Client or httpx.AsyncClient. The old proxies= argument was removed in httpx 0.28. Because the proxy belongs to the client, create one client per proxy when you rotate.

Yes. The $4.99 CleanProxies trial gives you 5 ISP IPs for 1 hour, or 100 MB of residential or mobile data, which is enough to run the rotators in this guide against your own target.

Team CleanProxies

Proxy Infrastructure Team