Web scraping at scale: architecture, proxy routing and ethics

A scraper that fetches ten thousand pages is a script. A scraper that fetches ten million pages every week, survives site redesigns, and produces data someone will make decisions from is a distributed system. This article is about the second thing: the architecture, the failure modes and the operational discipline that separate a pipeline you can trust from one that quietly lies to you.

Separate fetch, parse and load

The single highest-leverage decision in a crawling system is to keep these three stages independent, each with its own storage boundary.

1. Fetch — retrieve bytes and write them, unmodified, to object storage. Nothing else.
2. Parse — read stored bytes, extract structured records, write to a staging table.
3. Load — validate, deduplicate, reconcile with history, publish.

The reason is economic. Fetching is the expensive part: it costs proxy bandwidth, it takes wall-clock time, and it consumes goodwill with the target. Parsing is nearly free. When a site changes its markup — and it will — a coupled pipeline forces you to re-crawl everything. A decoupled one lets you fix the selector and reprocess yesterday's archive in minutes.

s3://crawl-raw/{source}/{yyyy-mm-dd}/{sha256(url)}.html.gz
    ├── stored with metadata: url, fetched_at, status, proxy_pool, response_headers
    └── immutable; never overwritten

postgres.staging_products      ← parser output, one row per (url, crawl_date)
postgres.products              ← loader output, deduplicated, versioned

The work queue

Move from "a list of URLs in a loop" to a durable queue as soon as you have more than one worker. The queue gives you retries, backpressure, priority and observability for free.

-- A queue table is enough up to a few million rows a day
CREATE TABLE crawl_queue (
    id           BIGSERIAL PRIMARY KEY,
    url          TEXT NOT NULL,
    url_hash     BYTEA GENERATED ALWAYS AS (sha256(url::bytea)) STORED,
    source       TEXT NOT NULL,
    priority     SMALLINT     DEFAULT 5,
    state        TEXT         DEFAULT 'pending',   -- pending|leased|done|failed
    attempts     SMALLINT     DEFAULT 0,
    lease_until  TIMESTAMPTZ,
    last_error   TEXT,
    created_at   TIMESTAMPTZ  DEFAULT now(),
    UNIQUE (url_hash)
);
CREATE INDEX ON crawl_queue (state, priority, created_at)
    WHERE state IN ('pending', 'leased');

Leases matter more than they look. A worker claims a batch with a deadline; if it dies, the lease expires and the work returns to the pool automatically. No orphaned URLs, no manual cleanup.

-- Atomic claim: SKIP LOCKED lets many workers pull concurrently
UPDATE crawl_queue SET
    state = 'leased',
    lease_until = now() + interval '5 minutes',
    attempts = attempts + 1
WHERE id IN (
    SELECT id FROM crawl_queue
    WHERE state = 'pending'
       OR (state = 'leased' AND lease_until < now())
    ORDER BY priority, created_at
    LIMIT 50
    FOR UPDATE SKIP LOCKED
)
RETURNING id, url, source;

Adaptive concurrency

Fixed concurrency is always wrong. Too low and you waste time; too high and you trigger rate limiting, then spend hours recovering. Let the target tell you the right number.

An additive-increase / multiplicative-decrease controller — the same idea behind TCP congestion control — converges on the ceiling without crossing it:

class AdaptiveLimiter:
    def __init__(self, start=4, ceiling=64):
        self.limit, self.ceiling = start, ceiling
        self.ok_streak = 0

    def success(self):
        self.ok_streak += 1
        if self.ok_streak >= 20:                 # earn the increase
            self.limit = min(self.ceiling, self.limit + 1)
            self.ok_streak = 0

    def throttled(self):                          # 429, 503, or a challenge
        self.limit = max(1, int(self.limit * 0.6))
        self.ok_streak = 0
        return 2 ** min(6, self.backoff_step())   # seconds to sleep

Track the limit per host, not globally. Two targets on the same crawl have nothing to do with each other, and one aggressive site should not slow down the rest of your run.

Politeness as engineering

Being considerate is not only ethical; it is the cheapest way to stay unblocked.

• Honour robots.txt. Parse it, cache it for a day, and respect Crawl-delay when present.
• Identify yourself where appropriate — a User-Agent with a contact URL turns a potential block into an email.
• Respect Retry-After. If the server tells you when to come back, come back then.
• Use conditional requests. If-Modified-Since and If-None-Match turn most re-crawls into a 304 with no body — cheaper for both sides.
• Crawl off-peak in the target's local timezone.
• Cap per-host concurrency even when the target tolerates more. You are not the only crawler they have.

from urllib.robotparser import RobotFileParser
from functools import lru_cache

@lru_cache(maxsize=512)
def robots_for(origin: str) -> RobotFileParser:
    rp = RobotFileParser()
    rp.set_url(origin + "/robots.txt")
    rp.read()
    return rp

def allowed(url: str, ua: str = "MyCrawler") -> bool:
    origin = "://".join(urlsplit(url)[:2])
    return robots_for(origin).can_fetch(ua, url)

Proxy routing that pays for itself

Do not send everything through your most expensive pool. Route per domain, based on measured success rates, and escalate only on failure. The pattern and the arithmetic are covered in residential versus datacentre proxies; here is the operational shape:

ROUTES = {          # learned from the last 24h of metrics, refreshed hourly
    "shop.example.com":    ["datacenter"],
    "retailer.example":    ["isp", "residential"],
    "fortress.example":    ["residential"],
}

def pools_for(host):
    return ROUTES.get(host, ["datacenter", "isp", "residential"])

Three rules keep this healthy:

• One sticky session per logical identity, held for the entire flow.
• Block images, fonts and media when you only need the DOM — on per-GB billing this alone often cuts spend by half.
• Record the pool used on every stored response, so you can attribute cost and success rate later.

# Playwright: refuse the bytes you are not going to parse
await page.route("**/*", lambda route: (
    route.abort() if route.request.resource_type in
        {"image", "media", "font", "stylesheet"} else route.continue_()
))

Validate on content, never on status

The most damaging failure mode in scraping is the soft block: HTTP 200, a well-formed page, and no data. Your monitoring shows green while your warehouse fills with nulls.

Defend with layered assertions on every response:

def validate(html: str, url: str) -> None:
    if len(html) < 2_000:
        raise SoftBlock(f"suspiciously small: {len(html)}B")

    lowered = html.lower()
    for marker in ("captcha", "access denied", "are you a robot",
                   "unusual traffic", "cf-browser-verification"):
        if marker in lowered:
            raise SoftBlock(f"challenge marker: {marker}")

    doc = HTMLParser(html)
    if doc.css_first("h1.product-title") is None:
        raise ParseFailure("expected selector missing")

    price = doc.css_first("[data-price]")
    if price and not PRICE_RE.match(price.text(strip=True)):
        raise ParseFailure(f"implausible price: {price.text()!r}")

Then add distribution-level checks at load time — the errors that no single-record validation can catch:

• Row count within ±20% of the trailing seven-day median.
• Null rate per column below its historical maximum.
• Median price within a plausible band of yesterday's.
• Fewer than 5% of records unchanged in a dataset that should change daily (a sign you are re-reading a cache).

Handling schema drift

Sites redesign. Class names change, markup gets restructured, a field moves inside a JSON blob. Build for it:

1. Multiple selectors per field, tried in order, with the winning strategy recorded.
2. Prefer structured sources. JSON-LD (application/ld+json), microdata and embedded state blobs are far more stable than CSS classes.
3. Alert on strategy shifts. If a field suddenly starts resolving via fallback #3, something changed — investigate before the fallback also breaks.

PRICE_STRATEGIES = [
    ("jsonld",   lambda d: jsonld_field(d, "offers.price")),
    ("microdata",lambda d: d.css_first('[itemprop="price"]')),
    ("data-attr",lambda d: d.css_first("[data-price]")),
    ("css",      lambda d: d.css_first("span.price-now")),
]

def extract_price(doc):
    for name, fn in PRICE_STRATEGIES:
        try:
            node = fn(doc)
            if node is not None:
                metrics.increment("price.strategy", tags={"used": name})
                return clean_price(node)
        except Exception:
            continue
    raise ParseFailure("no price strategy matched")

When you actually need a browser

Headless browsers cost roughly 50–200× more CPU and memory per page than an HTTP request. Use them only when the data genuinely is not available otherwise, and check these first:

1. Is there an internal API? Open the network tab and look for the XHR/fetch call that populates the page. It usually returns clean JSON.
2. Is the state embedded? __NEXT_DATA__, __NUXT__, Redux preload blobs and JSON-LD are all in the initial HTML.
3. Does a plain fetch already contain it? Server-rendered frameworks often ship the full content and hydrate afterwards.

If you do need a browser, amortise the cost: reuse contexts, run several pages per browser instance, block subresources, and cap the pool by available memory rather than by optimism.

browser = await pw.chromium.launch(args=[
    "--disable-dev-shm-usage",     # avoid /dev/shm exhaustion in containers
    "--disable-gpu",
    "--no-sandbox",
])
# One context per identity; several pages per context; recycle every N pages

What to measure

Legal and ethical boundaries

Scraping sits in genuinely contested legal territory, and it varies by jurisdiction, by what you collect and by how you use it. General principles that keep most projects on defensible ground:

• Public data only. Do not circumvent authentication, paywalls or access controls.
• Personal data is regulated. Names, emails and profiles fall under GDPR, CCPA and similar regimes regardless of being publicly visible. You need a lawful basis.
• Contracts matter. Terms of service can bind you, especially where you accepted them by registering.
• Do not degrade the service. Volume that materially affects a site's availability moves you from "crawler" to "problem" quickly.
• Copyright survives copying. Extracting facts differs from republishing creative content.

Get advice specific to your jurisdiction and your dataset before a project goes to production. This section is general information, not legal advice.

Production readiness checklist

1. Raw responses archived immutably, with metadata, before parsing.
2. Durable queue with leases, retry limits and a dead-letter path.
3. Per-host adaptive concurrency and per-host proxy routing.
4. Content-level validation on every response; distribution checks at load.
5. Multi-strategy extraction with alerting on strategy shifts.
6. Metrics for success rate, challenge rate, latency, bytes and cost — per host.
7. Idempotent workers: re-running a batch never duplicates rows.
8. A documented kill switch, and someone who knows they own it.
9. Retention policy for raw archives and for any personal data.
10. A written note of the legal basis for each source you crawl.

The unglamorous conclusion

Teams tend to spend their budget on proxies and their attention on parsers. The systems that survive spend attention on queues, archives, validation and observability — and then discover they need less proxy budget than they thought, because they are no longer re-crawling to fix mistakes.

Build the boring parts first. For choosing the pools underneath, see residential versus datacentre proxies; for the hardware to run it on, see the dedicated server buying guide.