How to use proxies with Scrapy

• An active CleanProxies plan: static ISP proxies for fixed US IPs, or rotating residential / mobile proxies for pools. The trial works too.
• Your credential line from the dashboard's proxy manager: host (or IP), port, username and password.
• A Scrapy 2.x project (pip install scrapy, scrapy startproject myproject).

New to proxies? For the overview, read Proxies with Python Requests, Playwright and Puppeteer and Best residential proxies for web scraping. This guide goes deeper on Scrapy only.

Most tools want the same details as a URL: http://USERNAME:PASSWORD@HOST:PORT for HTTP(S) or socks5://USERNAME:PASSWORD@HOST:PORT for SOCKS5. The examples on this page use those placeholders; copy the host, port, username and password from your CleanProxies dashboard. To convert a whole list at once, paste it into the proxy list formatter.

1. 1

   Copy your credentials

   Take a rotating residential line, or export your ISP list, from the CleanProxies dashboard.
2. 2

   Add the proxy to requests

   Set meta["proxy"] on each Request, or add a downloader middleware that does it for every request.
3. 3

   Tune the crawl

   Enable AutoThrottle, set sensible concurrency per domain and keep retries on.
4. 4

   Check the exit IP

   Run a one-off spider against https://api.ipify.org and log the response body.

import scrapy

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

class IpCheckSpider(scrapy.Spider):
    name = "ip_check"

    async def start(self):  # Scrapy 2.13+; use start_requests() on older versions
        yield scrapy.Request("https://api.ipify.org", meta={"proxy": PROXY})

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.text)

Use the HTTP(S) form of your CleanProxies credentials: Scrapy's default download handlers don't speak SOCKS5. The middleware also honours http_proxy and https_proxy environment variables when no meta proxy is set.

A downloader middleware can assign a proxy to every request. Put it before the built-in HttpProxyMiddleware (priority 750) so the credentials are processed.

import random

class CleanProxiesMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        path = crawler.settings.get("PROXY_LIST", "proxies.txt")
        with open(path) as f:
            lines = [line.strip() for line in f if line.strip()]
        urls = []
        for line in lines:
            host, port, user, password = line.split(":", 3)
            urls.append(f"http://{user}:{password}@{host}:{port}")
        return cls(urls)

    def process_request(self, request, spider):
        request.meta.setdefault("proxy", random.choice(self.proxies))

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CleanProxiesMiddleware": 740,
}
PROXY_LIST = "proxies.txt"

AUTOTHROTTLE_ENABLED = True
CONCURRENT_REQUESTS_PER_DOMAIN = 8
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]

To keep one IP per logged-in account, store the chosen proxy with the account in meta and pass it along to follow-up requests instead of picking a new one.

Recent Scrapy releases (2.17 and later) ship an experimental HTTPX-based download handler that accepts socks5:// proxy URLs in meta["proxy"]. Scrapy's docs flag it as not yet recommended for production, so HTTP(S) stays the default choice.

DOWNLOAD_HANDLERS = {
    "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
    "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
}

# then, per request:
# meta={"proxy": "socks5://USERNAME:PASSWORD@HOST:PORT"}

Run scrapy crawl ip_check. The log line Exit IP: … should show the proxy's address, not your server's.

Open What is my IP? through the proxy to see the address and headers a website receives, or paste up to ten lines into the free proxy checker to test connectivity, latency and exit IP before you start.

Still stuck? Our 24/7 support team answers fast: contact us with the error and the line you used (leave the password out).