Skip to content
Get Started
G2G2
4.7/5
Excellent
Trustpilot

What Is a Web Crawler?

A web crawler (also called a spider or bot) is a program that discovers web pages by fetching a page, collecting its links and following them, over and over. Search engines use crawlers such as Googlebot to find and index the web; companies use them to map their own sites or collect public data.

Try Now for $4.99Continue with Google
  • Session Control
  • High Speed
  • Privacy First
Best fit for Web crawlerRotating Residential Proxiesfrom$0.63/GB
how it works

How a crawler works

A crawler starts from seed URLs and keeps a queue of pages to visit, called the frontier. For each URL it checks robots.txt, fetches the page, extracts links, normalises them, drops ones it has already seen and adds the rest to the queue. Sitemaps give it extra URLs directly. A scheduler decides what to fetch next and how fast each host may be hit.

Crawling and scraping are often mixed up. A crawler is about discovery: finding URLs. A scraper is about extraction: pulling specific data out of pages. Most real projects do both.

  • Web crawler

    This term
    Goal
    Discover and visit pages
    Output
    A list or index of URLs and their content
    Scope
    Whole sites or the web
    Examples
    Googlebot, Bingbot, SEO audit tools
  • 2

    Web scraper

    Goal
    Extract specific data from pages
    Output
    Structured records: prices, titles, reviews
    Scope
    Chosen pages and fields
    Examples
    Price monitors, SERP trackers
try it

Check robots.txt before you crawl

robots.txt is the long-standing convention for telling crawlers what they may fetch, standardised as RFC 9309 in 2022. Python's standard library reads it for you:

Check robots.txt before fetchingPython
from urllib import robotparser

rp = robotparser.RobotFileParser("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("MyCrawler/1.0", "https://example.com/products/"))
print(rp.crawl_delay("MyCrawler/1.0"))  # None if the site sets none
why it matters

Why crawlers use proxies

Large crawls send many requests, and sites limit requests per IP. Proxies spread the load across IPs and let a crawler fetch pages as they appear in other countries. They do not change the rules of politeness.

CleanProxies rotating residential proxies rotate the IP on every request across 190+ countries; static ISP proxies suit steady crawls of the same sites with unlimited bandwidth.

Web crawler: what it means for you
  • Limit concurrency per host, not just overall; spreading a crawl over many IPs does not entitle it to hit one site harder.
  • Honour robots.txt, crawl delays, 429 responses and Retry-After.
  • Cache and deduplicate, so you never fetch the same page twice without need.
  • Name your crawler in the User-Agent with a contact URL if you crawl as a search or research bot.

Rotating residential proxies from $0.63/GB

The CleanProxies proxy that fits best here, with a $4.99 trial to test it first.

Go deeper

Guides from the blog

Setup guides and best practices from the CleanProxies blog.

Get the latest insightsAll Blogs

Related terms

Terms that come up alongside Web crawler.

All glossary terms

Web crawler FAQs

Straight answers about Web crawler.

A crawler discovers pages by following links; a scraper extracts specific data from pages. Many tools do both: crawl to find product pages, then scrape each one.

Got more questions?All FAQs