Most scraping setups start the same way. Someone writes a Python script, fires off requests in a loop, parses the HTML, and dumps everything into a CSV. It works fine until it doesn’t. The moment you’re scraping thousands of pages per hour across dozens of domains, that single-script approach turns into a latency nightmare with unpredictable throughput and silent failures. Picking the right scraping tools matters, but tools alone won’t save you if the underlying infrastructure can’t keep up.
One Endpoint Outside, a Pipeline Inside
Your users should hit one endpoint, something like POST /scrape, while internally the work fans out through a pipeline with distinct stages: validation, scheduling, fetching, optional rendering, extraction, normalization, delivery. The reason this decomposition matters is measurability. When each stage has its own timing span, you stop guessing where latency lives and start measuring it.
Keep your control plane (template registry, rate-limit policies, credentials) separate from the data plane (fetch/render/extract workers). This lets you deploy new templates or tweak policies without redeploying the workers that are processing requests.
For most teams, a modular monolith is the right starting point. One deployable service, internal modules for each stage, lowest overhead. Graduate to microservices when you need to scale hot components independently, like a renderer pool that’s eating CPU while fetchers sit idle. Serverless fits spiky async workloads, but cold starts can wreck latency SLOs, and headless rendering in Lambda-style environments gets awkward fast. If you’re exploring AI-driven scraping workflows, MCP server integrations can simplify the endpoint layer, but the pipeline principles below still apply.
Scraping Templates That Actually Hold Up
A scraping template is more than a bag of CSS selectors. A good one encodes acquisition mode (HTTP fetch vs. browser render vs. direct API call), extraction rules (selectors, JSONPath, regex), render-done conditions (target selector present, network idle, DOM stable), and politeness config (per-domain rate limits, robots.txt compliance). Everything the pipeline needs to process a page type without human intervention.
The part most teams skip is versioning and validation. Use SemVer for templates so production workloads stay reproducible. And validate every output against a JSON Schema definition before it leaves the pipeline. Without this, a site redesign can silently corrupt your data for hours before anyone notices. Schema validation turns that into a loud, immediate failure you can fix.
Where the Milliseconds Actually Go
This is where most of the performance wins live. Every scrape job walks through a predictable sequence. DNS lookup, TCP connect, TLS handshake, HTTP request, time-to-first-byte, download, optional rendering, extraction, serialization. Each segment is measurable with OpenTelemetry spans, and each one has specific optimizations.
Connection Reuse, HTTP/2, and DNS
The single biggest quick win is connection reuse. If you’re scraping multiple URLs on the same host, reusing TCP and TLS connections eliminates repeated handshakes. In Python, HTTPX’s async client handles connection pooling automatically when you use a Client instance rather than the top-level API (which opens a new connection per request). In Node.js, undici gives you explicit control over pooling configuration and assumes persistent connections by default.
HTTP/2 adds multiplexing and header compression on a single connection. For scraping workflows that make many concurrent requests to the same origin, this reduces latency measurably. HTTPX supports HTTP/2 but doesn’t enable it by default, so you’ll need to opt in. HTTP/3 goes further by running over QUIC, which improves performance under packet loss and cuts connection setup time. Support is still uneven across clients and targets, but it’s worth enabling opportunistically where available.
DNS lookups are an underrated bottleneck at high concurrency. Every unique domain needs resolution, and if your resolver isn’t caching aggressively, you’re adding milliseconds to every request. Prefer long-lived processes that benefit from OS-level and library-level DNS caches. Avoid creating new resolver instances per request.
Geo-Distribution and Proxy Placement
If your fetchers sit in one region but your targets are globally distributed, round-trip time dominates your latency budget. Moving fetchers closer to target sites is one of the most impactful optimizations you can make.
This is where proxy infrastructure becomes a performance tool, not just an anonymity layer. A geo-distributed rotating proxy network with nodes near your target origins shaves tens or hundreds of milliseconds off each request. Decodo’s residential proxy network covers 195+ locations with over 125 million IPs, which means you can route requests through nodes that are geographically close to whatever site you’re scraping. That proximity directly reduces RTT and keeps your throughput consistent across regions.
For your own API endpoint (the part your consumers hit), services like AWS Global Accelerator or edge placement features like Cloudflare Workers Smart Placement can route traffic to the nearest processing node based on measured latency.
Measuring Each Segment
Instrument every stage with spans. OpenTelemetry’s Collector pipeline (receive → process → export) is built for exactly this kind of multi-service telemetry. Use W3C Trace Context headers (traceparent, tracestate) to propagate trace IDs across internal calls so you can correlate API request → worker → renderer in a single trace.
Your target metric set:
- Scrape request duration (histogram) – labeled by mode, template, status class
- Queue wait time (histogram) – how long jobs sit before a worker picks them up
- Fetch sub-timings (histograms) – DNS, TLS, TTFB broken out individually
- Renderer active sessions (gauge) and CPU seconds (counter)
- Errors and retries (counters) – broken down by type and domain
Be careful with Prometheus label cardinality, though. Per-domain labels on high-volume metrics will blow up your time-series count.
Concurrency Without the Chaos
Increase concurrency until you hit a limiting resource, then apply backpressure so latency stays predictable.
Bounded Queues and Backpressure
Unbounded queues are a latency trap. Without backpressure, your queue grows without limit, memory balloons, and end-to-end latency becomes unpredictable because jobs sit waiting longer and longer. The fix is simple. In Python, asyncio.Queue(maxsize=1000) causes put() to block when the queue is full. That’s your backpressure primitive. The producer slows down automatically.
Node.js streams have a built-in backpressure mechanism that prevents downstream overload. If you’re building a custom pipeline, respect it.
For CPU-bound work like heavy HTML parsing or compression, offload to worker pools. Node’s worker_threads handles this. Python’s concurrent.futures gives you ThreadPoolExecutor and ProcessPoolExecutor behind a clean interface.
Rate Limits and 429 Handling
Respect per-domain rate limits. When you exceed them, you’ll get 429 Too Many Requests, often with a Retry-After header telling you how long to wait. Don’t ignore it. If you’re scraping sites like Amazon that enforce aggressive rate limiting, this matters even more.
Combine rate limiting with retries that use exponential backoff and jitter. Uncontrolled retries can overload both your own system and the target. Only retry on transient failures. A 404 isn’t going to succeed on the second try.
If a domain repeatedly fails with timeouts, 5xx responses, or CAPTCHAs, trip a circuit breaker. Stop sending traffic to it, protect your system from cascading failures, and re-probe slowly to check if it’s recovered.
Skip the Browser When You Can
Headless rendering is the single most expensive operation in a scraping pipeline. It eats CPU and RAM. The fastest scraper is the one that doesn’t render when it doesn’t have to.
The “Auto” Mode Pattern
Build your templates with a three-step acquisition strategy:
- Try HTTP-only extraction first. If the required fields are present in the raw HTML response, you’re done
- If critical selectors return empty, fall back to rendering. Let the template decide when a browser is actually needed
- Record which path was taken. Track your render fallback rate over time so you can see which templates are costing you the most
The goal is to push that HTTP-only success rate as high as possible. Many sites that appear to be fully client-rendered actually embed their data in JSON-LD, __NEXT_DATA__ blobs, or inline <script> tags. Extracting from those is orders of magnitude cheaper than spinning up a browser.
Rendering Faster When You Must
When rendering is unavoidable, cut the waste. Block non-essential resource types like images and fonts. Playwright’s route API and Puppeteer’s request interception both support this. On content-heavy pages, blocking images alone can cut render time significantly.
Ditch fixed sleep() calls. Instead, use done conditions. Wait for a target selector to appear, a specific network request to complete, or the DOM node count to stabilize over a short interval.
Keep a pool of warm browser contexts. Cold-starting a browser per job adds seconds of overhead. Reuse browser instances and isolate at the context level rather than launching a full browser per request. For low-latency SLOs, this is non-negotiable.
Getting Data Out
For synchronous API responses, JSON is the obvious default. For streaming large result sets, use NDJSON or JSON Text Sequences so consumers can process records one at a time without buffering. For bulk analytical exports, Apache Parquet cuts storage and speeds up downstream queries. For async workflows, model webhook events with CloudEvents and publish your API contract via OpenAPI with idempotency keys on async jobs so retries don’t create duplicates.
