How to Use Proxies with Python Requests, Playwright and Puppeteer

Getting a proxy into your code takes one line. Getting it to work reliably at scale takes more: your real IP leaking through an environment variable, a rotating gateway that keeps handing you the same exit, Chrome silently refusing a SOCKS5 password, or a headless browser burning gigabytes on images you never parse. And even perfect code fails if the IP behind it is a flagged datacenter address. This guide covers both halves: the code patterns that work in Python Requests, Playwright and Puppeteer, and the proxy choices that keep them running.

TL;DR

• Python Requests: pass a proxy URL such as http://USERNAME:PASSWORD@HOST:PORT in the proxies dictionary, ideally on a Session with retries.

• Playwright: pass a proxy object with server, username and password to launch(), or to new_context() for one IP per context.

• Puppeteer: start Chrome with --proxy-server=http://HOST:PORT and call page.authenticate() with your credentials.

• The catch: Chrome can't send a username and password to a SOCKS5 proxy, so use the HTTP endpoint, or SOCKS5 with IP whitelisting.

• The IP matters as much as the code. CleanProxies specializes in US ISP proxies on Tier-1 carriers with 0 fraud score IPs, 100Gbps ports and under 20 ms average latency, at the cheapest pricing in the market from $0.75/IP, so your scripts hit fewer CAPTCHAs.

See pricing · Buy US ISP proxies · Test our free proxy checker

CleanProxies for automation at a glance

Why the IP behind your code matters

Anti-bot systems look up the network (ASN) behind every request before they look at your headers or your browser fingerprint. Requests from AWS, Google Cloud and budget hosts start with a low trust score; Cloudflare found those two clouds alone originated a quarter of global bot traffic in 2025. Requests from consumer carriers like Comcast, Spectrum or Verizon start high. A US ISP proxy gives your script a static IP on one of those carriers, hosted on fast server hardware, so you get household-level trust at data-center speed.

That's why CleanProxies leads with US ISP proxies. Every IP is real, 100% dedicated, screened for a 0 fraud score, and served from Ashburn, Virginia on 100Gbps ports with 99.9% uptime, with free replacements and 24/7 support from the same team that runs BartProxies. Your code runs the same either way; it just stops getting challenged.

This guide goes past the basics. If you only need a one-line request through a proxy, start with How to Use Proxies with cURL, Python and Node.js. Here we cover sessions, retries, rotation, one IP per browser context, and cutting bandwidth in headless browsers.

Before you start: get your proxy line

The CleanProxies dashboard gives each proxy as a line, usually HOST:PORT:USERNAME:PASSWORD. Most code wants a URL instead:

http://USERNAME:PASSWORD@HOST:PORT

Our free proxy list formatter converts a whole list between formats. If your password contains characters such as @ or :, URL-encode them (@ becomes %40).

Which CleanProxies product to use:

• US ISP proxies for logins, long sessions and heavy repeat fetches: each line is one static, dedicated IP on a Tier-1 US carrier (Cox, Spectrum, Frontier, Verizon, Comcast, RCN, AT&T or T-Mobile), with unlimited bandwidth and threads.

• Rotating residential for scraping many pages: one gateway endpoint that gives you a new IP per request, or holds one IP for a sticky session of up to 90 minutes.

• Mobile proxies for mobile-first platforms: real 4G/5G carrier IPs.

Replace the capitalised placeholders in every example with your own values.

Which tool for which job

Start with plain HTTP requests wherever the data is in the HTML or a JSON endpoint: it's faster and uses a fraction of the bandwidth. Reach for a headless browser only when the page needs JavaScript to render the data, or when you have to act like a logged-in user.

Python Requests: a session with retries

A Session reuses connections and settings, and an HTTPAdapter with Retry handles temporary errors for you:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

session = requests.Session()
session.trust_env = False  # ignore HTTP(S)_PROXY environment variables
session.proxies = {"http": PROXY, "https": PROXY}
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
})

retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)

response = session.get("https://api.ipify.org?format=json", timeout=30)
print(response.json())  # {"ip": "..."}: should be the proxy's IP

Notes:

• trust_env = False stops environment variables from overriding your proxy settings, a common source of "my real IP shows up" bugs.

• backoff_factor=1 waits longer between each retry, and Retry honours a Retry-After header on 429 responses by default.

• For SOCKS5, run pip install "requests[socks]" and use socks5h://USERNAME:PASSWORD@HOST:PORT so DNS lookups also go through the proxy.

Rotating gateways and connection reuse

Many rotating gateways pick an exit IP per connection, not per request. A Session keeps connections open, so several requests can leave from the same IP. If you need a new IP on every request, open a fresh connection each time:

def fetch_new_ip(url):
    with requests.Session() as s:
        s.trust_env = False
        s.proxies = {"http": PROXY, "https": PROXY}
        return s.get(url, timeout=30, headers={"Connection": "close"})

If you want the same IP for a multi-step flow, such as page 1, 2 and 3 of a search, use a sticky session instead. CleanProxies residential sticky sessions last up to 90 minutes. Our guide to rotating vs sticky sessions explains when to use which.

Spreading work across a list of ISP proxies

With static ISP proxies, give each worker its own IP:

from concurrent.futures import ThreadPoolExecutor
import requests

with open("proxies.txt") as f:
    lines = [line.strip() for line in f if line.strip()]

def to_url(line):
    host, port, user, password = line.split(":", 3)
    return f"http://{user}:{password}@{host}:{port}"

def check(line):
    url = to_url(line)
    r = requests.get("https://api.ipify.org", proxies={"http": url, "https": url}, timeout=30)
    return line.split(":")[0], r.text

with ThreadPoolExecutor(max_workers=10) as pool:
    for host, seen_ip in pool.map(check, lines):
        print(host, "->", seen_ip)

Async with HTTPX

For thousands of concurrent requests, an async client is lighter than threads. In current HTTPX versions the option is proxy:

import asyncio
import httpx

async def main():
    async with httpx.AsyncClient(proxy="http://USERNAME:PASSWORD@HOST:PORT", timeout=30) as client:
        r = await client.get("https://api.ipify.org?format=json")
        print(r.json())

asyncio.run(main())

For SOCKS5, install httpx[socks].

Playwright: one proxy for the browser, or one per context

Playwright's proxy option takes a server plus optional username and password. Its API docs say both HTTP and SOCKS proxies are supported, and that the credentials apply when an HTTP proxy requires authentication.

Python

from playwright.sync_api import sync_playwright

PROXY = {"server": "http://HOST:PORT", "username": "USERNAME", "password": "PASSWORD"}

with sync_playwright() as p:
    browser = p.chromium.launch(proxy=PROXY)
    page = browser.new_page()
    page.goto("https://api.ipify.org?format=json")
    print(page.inner_text("body"))
    browser.close()

One IP per context

A browser context is an isolated session with its own cookies and storage. Giving each context its own static ISP proxy lets one browser run several separate identities:

from playwright.sync_api import sync_playwright

with open("proxies.txt") as f:
    lines = [line.strip() for line in f if line.strip()]

with sync_playwright() as p:
    browser = p.chromium.launch()
    for line in lines[:3]:
        host, port, user, password = line.split(":", 3)
        context = browser.new_context(
            proxy={"server": f"http://{host}:{port}", "username": user, "password": password},
            locale="en-US",
            timezone_id="America/New_York",
        )
        page = context.new_page()
        page.goto("https://api.ipify.org?format=json")
        print(host, page.inner_text("body"))
        context.close()
    browser.close()

Set locale and timezone_id to match where the IP is. A US IP with a browser that reports a European timezone is an easy signal for anti-bot systems.

Node.js

import { chromium } from "playwright";

const browser = await chromium.launch({
  proxy: { server: "http://HOST:PORT", username: "USERNAME", password: "PASSWORD" },
});
const page = await browser.newPage();
await page.goto("https://api.ipify.org?format=json");
console.log(await page.textContent("body"));
await browser.close();

Puppeteer: launch flag plus page.authenticate

Puppeteer passes the proxy to Chrome as a launch argument, and supplies credentials with page.authenticate():

import puppeteer from "puppeteer";

const browser = await puppeteer.launch({
  args: ["--proxy-server=http://HOST:PORT"],
});
const page = await browser.newPage();
await page.authenticate({ username: "USERNAME", password: "PASSWORD" });
await page.goto("https://api.ipify.org?format=json");
console.log(await page.$eval("body", (el) => el.innerText));
await browser.close();

Puppeteer's docs note that authenticate() turns on request interception internally, which can affect performance slightly.

For a different proxy per context in recent Puppeteer versions, pass proxyServer when you create the context:

const context = await browser.createBrowserContext({ proxyServer: "http://HOST2:PORT2" });
const page2 = await context.newPage();
await page2.authenticate({ username: "USERNAME2", password: "PASSWORD2" });

The SOCKS5 catch in Chrome

Chromium's own proxy documentation says no authentication methods are supported for SOCKSv5 in Chrome. Both Playwright (with Chromium) and Puppeteer drive Chrome, so a socks5://USERNAME:PASSWORD@... line won't authenticate. You have two options:

1. Use the HTTP endpoint with username and password, as in the examples above. Every CleanProxies product supports HTTP(S).

2. Use SOCKS5 with IP whitelisting: add your server's IP in the dashboard, then pass socks5://HOST:PORT with no credentials.

On the plus side, Chrome always resolves hostnames on the proxy side for SOCKS5, so your DNS lookups don't leak. For a full comparison of the two protocols, read SOCKS5 vs HTTP proxies.

Cut bandwidth in headless browsers

On per-GB plans, everything the browser downloads costs money. The median home page weighs 2.56 MB on mobile (HTTP Archive Web Almanac 2025), and most of that is images, fonts and media you don't need for data collection.

Playwright (Python):

def block_heavy(route):
    if route.request.resource_type in {"image", "media", "font"}:
        route.abort()
    else:
        route.continue_()

page.route("**/*", block_heavy)

Puppeteer:

await page.setRequestInterception(true);
page.on("request", (req) => {
  if (["image", "media", "font"].includes(req.resourceType())) req.abort();
  else req.continue();
});

Don't block resources on sites where you log in or check out; missing assets can look unusual and break pages.

Testing your setup before you scale

Before you point a script at thousands of pages, run four quick checks:

1. Confirm the exit IP. Every example above requests api.ipify.org, which returns the address websites see. It should be the proxy's IP, not your server's. You can also open our What Is My IP tool inside a Playwright or Puppeteer page.

2. Confirm location and network. Paste a sample of lines into the proxy checker to see the country, ISP and response time. For US work, the ISP should be a named consumer carrier such as Comcast, Spectrum or Verizon, not a hosting company.

3. Run a small real batch. Fetch 50 to 100 pages from your actual target at the concurrency you plan to use, and log status codes, response times and bytes downloaded. If you see 403s or CAPTCHA pages, slow down before adding IPs.

4. Start with the $4.99 trial. The CleanProxies trial gives you 5 US ISP proxies for 1 hour, or 100 MB of residential or mobile data, enough to run every check above on the same IPs you'll use in production.

Keep those logs. When something breaks later, comparing against a known-good baseline tells you quickly whether the problem is the target, your code or the network.

Troubleshooting

Our 8 common proxy mistakes covers the behaviour side in more depth, and the proxy checker confirms a line works before you put it in code.

What the numbers mean in practice

• Python is the default scraping language for most teams. It was used by 57.9% of respondents in the Stack Overflow 2025 Developer Survey, up 7 percentage points in a year. JavaScript, which Puppeteer and Playwright's Node API use, was at 66%.

• Sites are filtering more traffic. Bots made up 53% of internet traffic in 2025 and bad bots 40% (Thales 2026 Bad Bot Report), so clean carrier IPs and consistent browser settings matter.

• Pages are heavy. At 2.56 MB for a median mobile home page, blocking images, media and fonts is the single biggest saving on per-GB plans. On CleanProxies US ISP proxies, bandwidth is unlimited, so a headless browser that loads every asset costs no more than one that doesn't.

Which proxy should you pick for your scripts?

Pick CleanProxies. It gives your code the IP types it needs from one account and one dashboard:

• Logged-in flows and one identity per Playwright context: US ISP proxies, one dedicated Tier-1 carrier IP per context, from $0.75/IP.

• Wide crawls with Requests or HTTPX: rotating residential through one gateway, as low as $0.63/GB, data never expires.

• Mobile-first targets: mobile proxies, as low as $0.79/GB.

• Your script's host: a VPS in the same Virginia data center, so the hop from code to proxy is as short as possible.

Every product supports HTTP(S) and SOCKS5 with username/password or IP whitelisting, so the examples above work unchanged.

Collect responsibly

Respect each site's terms and robots rules, keep request rates reasonable, and collect only public data. Where a site offers an official API, use it. For more on large-scale setups, see our web scraping use case.

CleanProxies pricing

The $4.99 trial is enough to run every example in this guide. Full tiers are on the pricing page.

See pricing · Buy US ISP proxies · Test our free proxy checker

Frequently Asked Questions

Team CleanProxies

Proxy Infrastructure Team

Share