← Blog

BLOG

Scraping Browser: A Production Runtime for Web Data

A scraping browser runs real Chromium in the cloud for agents and pipelines. Learn how it works, where it breaks, and how to wire one up over CDP.

9 min readRemote Browser
Illustrated browser documents with an extracted structured data panel
From rendered pages to useful informationREMOTE BROWSER / ILLUSTRATION

A scraping browser is a real Chromium instance running on remote infrastructure that you drive programmatically to load pages, execute JavaScript, and extract data. It differs from an HTTP client like requests or fetch because it renders the page the way a user's browser would: it runs client-side scripts, resolves dynamic content, and exposes the full DOM through the Chrome DevTools Protocol (CDP). If your target site ships data through React hydration, lazy-loaded lists, or a login wall, a scraping browser is often the only reliable way to get at it.

This guide covers what a scraping browser actually is, where the naive approach fails in production, and how to wire one up with Playwright over CDP against hosted Chromium. The code is real and runnable; the trade-offs are the ones you hit after the first hundred requests.

What a scraping browser is (and what it is not)

A scraping browser is not a headless HTTP library with a JavaScript engine bolted on. It is a full browser process — Chromium, in most cases — with a remote debugging endpoint you connect to. You get:

  • A real DOM and rendering pipeline. Layout, CSS, and script execution happen as they do for a human visitor.
  • CDP access. You can intercept network requests, read console output, evaluate JavaScript in page context, and capture screenshots.
  • Session state. Cookies, localStorage, and IndexedDB persist for the life of the session, which matters for multi-step flows.
  • Isolation. Each session runs in its own sandbox, so one job's cookies and cache don't leak into another's.

What it is *not* is a magic anti-bot bypass. A scraping browser gives you the same surface a normal browser has. Whether a site blocks you depends on IP reputation, request patterns, TLS fingerprints, and behavioral signals — not on the fact that you're using Chromium. Treat stealth as a configuration problem, not a guarantee. Remote Browser exposes configurable browser settings and proxy options, but no runtime can promise you'll never be challenged.

The other thing it is not: a substitute for understanding the target. If the data is in a JSON endpoint the page calls, hitting that endpoint directly is faster and cheaper than rendering the page. Use a scraping browser when the data only exists after rendering, or when you need to interact with the page to reach it.

Why local scraping breaks down

Running Chromium on your laptop or a single CI box works until it doesn't. The failure modes are predictable:

Resource contention. Each Chromium instance consumes a substantial amount of RAM. Ten parallel jobs on a 4 GB box will thrash. Scaling means provisioning more machines, and provisioning means you're now maintaining browser infrastructure instead of writing scrapers.

Version drift. Your local Chrome updates, your Playwright version pins a different Chromium build, and a selector that worked yesterday fails today. Pinning versions helps, but now you own the update treadmill.

IP and rate limits. A single egress IP hitting a site at volume gets throttled or blocked. Rotating proxies locally is possible but adds another moving part.

No observability. When a job fails at 3 a.m., you have logs and nothing else. You can't see what the page looked like, what the network returned, or where the click landed.

State leakage. Long-running scrapers accumulate cookies and cache. A session that logged into one account can contaminate the next job.

A hosted scraping browser addresses these by moving the browser process off your machine. You get a CDP endpoint per session, isolated state, and infrastructure that scales independently of your application. For a deeper treatment of the runtime layer, see Remote browsers for AI agents.

How a scraping browser session works

The lifecycle is the same whether you run Chromium locally or in the cloud:

  1. Provision a session. The provider starts a Chromium process and returns a WebSocket debugger URL.
  2. Connect over CDP. Your client (Playwright, Puppeteer, or a raw CDP client) attaches to that URL.
  3. Drive the page. Navigate, wait for selectors, evaluate scripts, extract data.
  4. Tear down. Close the connection and release the session.

The difference in a hosted setup is that steps 1 and 4 are API calls, and the browser lives somewhere else. That's it. The programming model is identical, which is why migrating an existing Playwright script is mostly a matter of swapping launch() for connectOverCDP().

Connecting with Playwright over CDP

Playwright's `connectOverCDP` attaches to an existing browser rather than launching a new one. Here's a complete TypeScript example that provisions a session, connects, extracts data, and cleans up:

import { chromium, type Browser, type Page } from "playwright";

interface Session {
  id: string;
  cdpUrl: string;
}

// Provision a hosted Chromium session. Endpoint shape is illustrative;
// check /documentation for the current API.
async function createSession(): Promise<Session> {
  const res = await fetch("https://api.remote-browser.dev/v1/sessions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.REMOTE_BROWSER_API_KEY}`,
    },
    body: JSON.stringify({
      // Configurable browser settings: proxy, viewport, locale, timezone.
      proxy: { country: "us" },
      viewport: { width: 1440, height: 900 },
    }),
  });
  if (!res.ok) throw new Error(`session create failed: ${res.status}`);
  return (await res.json()) as Session;
}

async function scrape(url: string): Promise<Record<string, string>[]> {
  const session = await createSession();
  let browser: Browser | undefined;

  try {
    browser = await chromium.connectOverCDP(session.cdpUrl);
    const context = browser.contexts()[0] ?? (await browser.newContext());
    const page: Page = await context.newPage();

    // Block images and fonts to cut bandwidth on data-only jobs.
    await page.route("**/*", (route) => {
      const type = route.request().resourceType();
      if (type === "image" || type === "font") return route.abort();
      return route.continue();
    });

    await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30_000 });

    // Wait for the client-rendered list to appear, not a fixed sleep.
    await page.waitForSelector("[data-testid='result-row']", { timeout: 15_000 });

    const rows = await page.$$eval("[data-testid='result-row']", (els) =>
      els.map((el) => ({
        title: el.querySelector("h3")?.textContent?.trim() ?? "",
        price: el.querySelector(".price")?.textContent?.trim() ?? "",
        href: (el.querySelector("a") as HTMLAnchorElement)?.href ?? "",
      }))
    );

    return rows;
  } finally {
    await browser?.close();
    await fetch(`https://api.remote-browser.dev/v1/sessions/${session.id}`, {
      method: "DELETE",
      headers: { Authorization: `Bearer ${process.env.REMOTE_BROWSER_API_KEY}` },
    });
  }
}

scrape("https://example.com/search?q=widgets").then(console.log);

Two details matter here. First, waitForSelector beats waitForTimeout every time — fixed sleeps are either too short (flaky) or too long (slow and expensive). Second, the finally block releases the session even on error. Leaked sessions are the most common cause of surprise bills in hosted browser setups. See pricing for how session time is metered.

Production criteria for choosing a scraping browser

Not all hosted browsers are equivalent. When you're evaluating one for scraping workloads, these are the axes that actually matter:

CriterionWhy it matters for scrapingWhat to check
CDP compatibilityDetermines whether Playwright/Puppeteer work unmodifiedFull CDP surface, not a subset
Session isolationPrevents cookie/cache leakage between jobsPer-session sandbox, fresh profile by default
Persistent profilesNeeded for logged-in scraping and multi-step flowsOpt-in profile reuse with explicit IDs
Proxy supportIP reputation drives block ratesConfigurable egress, per-session or per-request
Live viewerDebugging failures without re-runningReal-time view of the session
Concurrency modelDetermines throughput and costHow parallel sessions are counted
Egress locationSome sites are geo-restrictedRegion selection at session create
ObservabilityPost-mortem debuggingConsole logs, network traces, recordings

A few of these deserve elaboration.

CDP compatibility is non-negotiable. Some providers expose a limited API that wraps CDP rather than passing it through. If page.route(), page.evaluate(), or request interception behave differently than they do locally, you've traded portability for lock-in. Test with a script that uses interception and evaluate before committing.

Persistent profiles cut both ways. Reusing a profile keeps you logged in, which is essential for scraping behind auth. But it also means state accumulates. Use named profiles per account or per job type, and rotate them deliberately. Remote Browser supports persistent profiles as an explicit opt-in; the default is a clean session.

The live viewer is a debugging tool, not a feature. When a selector fails in production, being able to watch the session — or replay a recording — saves hours. It's the difference between "the job failed" and "the job failed because the site showed a consent modal."

Where scraping browsers fail, and what to do

Even a well-configured scraping browser hits walls. The common ones:

  • Bot challenges. Cloudflare, DataDome, and PerimeterX will challenge or block automated traffic. A scraping browser doesn't solve this by existing; you need realistic browser settings, good IPs, and human-like pacing. Some sites will remain off-limits regardless.
  • Rate limits. Hitting a site faster than a human would triggers throttling. Add jitter, respect Retry-After, and back off exponentially.
  • Selector fragility. Class names change. Prefer stable attributes (data-testid, ARIA roles) and fail loudly when a selector disappears rather than silently returning empty arrays.
  • Memory growth. Long-lived sessions accumulate memory. Recycle sessions every N pages rather than running one browser for hours.
  • Cost creep. Browser time is billed by the hour. A job that waits 30 seconds for a selector that never appears costs the same as one that succeeds. Set aggressive timeouts.

The pattern that holds up: short-lived sessions, explicit waits, request interception to skip non-essential assets, and structured error handling that captures a screenshot and the page HTML on failure.

When to use a scraping browser vs. something simpler

Use the simplest tool that gets the data:

  • Static HTML or a public API? Use an HTTP client. No browser needed.
  • Data rendered client-side, no interaction required? A scraping browser with request interception is the right call.
  • Multi-step flow with login and state? A scraping browser with a persistent profile.
  • High-volume, stable endpoints? Negotiate API access or use a data provider. Rendering large volumes of pages is expensive.

The scraping browser is the right tool when the page's behavior — not just its HTML — is what you need. For a broader look at running Chromium without local setup, see remote browser online and remote web browser.

Getting started

The fastest path is to provision a session, connect with connectOverCDP, and port an existing Playwright script. The API surface is small: create a session, get a CDP URL, connect, close. Everything else — interception, evaluation, screenshots — is standard Playwright.

Start with the documentation for the current session API and connection details, and check pricing to understand how browser time is metered before you scale up. If you're driving the browser from an agent rather than a script, the runtime considerations in remote control browser apply — agents need the same isolation and observability that scrapers do, plus tighter timeouts.

A scraping browser is infrastructure. Treat it like infrastructure: measure it, bound it, and replace it when it stops fitting. The code above is a starting point, not a finished pipeline — but it's the same shape every production scraper ends up with.