BLOG
Automated Web Browsing: A Production Runtime Guide
Automated web browsing for AI agents and scripts: how hosted Chromium, CDP, and session isolation work, and what to check before production.

Automated web browsing is the practice of driving a real browser programmatically — navigating pages, filling forms, extracting data, and clicking through multi-step flows — without a human at the keyboard. It covers everything from a Playwright test suite to an AI agent that books a flight. The hard part is rarely the automation code. It is the runtime: where the browser lives, how sessions are isolated, how you reconnect after a crash, and how you observe what happened. This guide explains how automated web browsing actually works in production and what to evaluate before you commit to a runtime.
What Automated Web Browsing Actually Means
At its core, automated web browsing is a control loop. A program sends commands to a browser, the browser executes them against a live page, and the program reads back state — the DOM, network responses, screenshots, console output. The protocol that carries those commands in Chromium is the Chrome DevTools Protocol (CDP), a JSON-over-WebSocket interface that exposes navigation, input, network interception, and runtime evaluation.
Two layers sit on top of CDP in practice:
- Automation libraries like Playwright, Puppeteer, and Selenium wrap CDP (or the WebDriver protocol) in a higher-level API. You write
page.click()and the library translates it into protocol calls. - Agent frameworks like browser-use or Stagehand sit above the libraries. They take a natural-language goal, decide which actions to take, and call the automation library on your behalf.
Automated web browsing is the umbrella term for all of this. The distinction that matters for production is not which library you pick — it is whether the browser process is something you own and babysit, or something a runtime manages for you.
Why Local Browsers Break Down at Scale
A single Playwright script on a laptop is fine. Ten of them running in CI is manageable. Many running concurrently, driven by agents that need persistent logins and clean IPs, is a different problem. The failure modes are predictable:
- Resource contention. Each Chromium instance consumes a large amount of RAM. Running many in parallel on one host causes swap thrashing and flaky timeouts.
- Session leakage. Cookies, localStorage, and cached credentials bleed between runs unless you isolate profiles. An agent that logs into one account can accidentally act as another.
- No observability. When a headless run fails at step 14 of 20, you have a stack trace and nothing else. You cannot see the page state at the moment of failure.
- Environment drift. Browser versions, fonts, and OS libraries differ between your laptop, CI, and production. A selector that works locally fails in the cloud.
- Reconnection gaps. If the process dies mid-session, the browser state is gone. There is no way to resume.
Hosted runtimes address these by moving the browser off your infrastructure. Remote Browser, for example, runs Chromium sessions on managed hosts and exposes them over CDP, so your code connects to a WebSocket endpoint instead of launching a local binary. The session lifecycle, isolation, and viewer are handled by the platform. You can read more about the architecture in Remote browsers for AI agents.
The Runtime Choices Compared
Before writing code, decide where the browser runs. The table below compares the common options on the dimensions that affect production reliability.
| Runtime | Isolation | Reconnect after crash | Live view | Profile persistence | Ops burden |
|---|---|---|---|---|---|
| Local Chromium (Playwright/Puppeteer) | Per-process, manual profiles | No — state lost | Via headed mode only | Manual user-data-dir | High at scale |
| Self-hosted Chromium in containers | Per-container | Partial, depends on orchestration | Custom tooling | Volume mounts | High |
| Hosted browser runtime (Remote Browser) | Per-session | Yes — reconnect over CDP | Built-in viewer | Persistent profiles | Low |
| Serverless browser functions | Per-invocation | No | Limited | Usually none | Low but constrained |
The trade-off is control versus operational load. Self-hosting gives you full control over the browser binary and network, but you own patching, scaling, and monitoring. A hosted runtime trades some of that control for a managed session lifecycle. For most agent workloads, the managed path wins because the browser is not the product — the agent is.
If you are evaluating hosted options specifically, Remote browser online covers what "online" actually means when the browser is a remote process rather than a local install.
Connecting to a Hosted Browser Over CDP
The connection model is the same whether the browser is local or remote: you get a CDP endpoint and attach to it. With Playwright, that is chromium.connectOverCDP(). The difference is the URL — a hosted runtime gives you a wss:// endpoint per session instead of a local port.
Here is a TypeScript example that creates a session, connects, navigates, and extracts data. The endpoint and session-creation call are illustrative; check the documentation for the exact API surface.
import { chromium } from "playwright";
// 1. Create a hosted session (endpoint shape per Remote Browser docs).
const sessionRes = await fetch("https://api.remote-browser.dev/v1/sessions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${process.env.REMOTE_BROWSER_API_KEY}`,
},
body: JSON.stringify({
// Persistent profile so logins survive across sessions.
profileId: "checkout-agent",
// Configurable browser settings; see docs for supported options.
proxy: { country: "us" },
}),
});
if (!sessionRes.ok) {
throw new Error(`Session create failed: ${sessionRes.status}`);
}
const { cdpUrl, sessionId } = await sessionRes.json();
// 2. Attach Playwright to the remote Chromium over CDP.
const browser = await chromium.connectOverCDP(cdpUrl);
const context = browser.contexts()[0] ?? (await browser.newContext());
const page = await context.newPage();
try {
await page.goto("https://example.com/orders", {
waitUntil: "domcontentloaded",
timeout: 30_000,
});
// Wait for the app to hydrate before reading state.
await page.waitForSelector("[data-testid='order-row']");
const orders = await page.$$eval("[data-testid='order-row']", (rows) =>
rows.map((row) => ({
id: row.getAttribute("data-order-id"),
status: row.querySelector(".status")?.textContent?.trim(),
})),
);
console.log(JSON.stringify({ sessionId, orders }, null, 2));
} finally {
// 3. Always release the session so it stops metering.
await browser.close();
await fetch(`https://api.remote-browser.dev/v1/sessions/${sessionId}`, {
method: "DELETE",
headers: { Authorization: `Bearer ${process.env.REMOTE_BROWSER_API_KEY}` },
});
}A few details matter here. connectOverCDP attaches to an existing browser rather than launching one, so you do not manage the process. browser.contexts()[0] reuses the default context the runtime created; creating a new one is fine but you lose the profile binding. The finally block is not optional — a session that is never closed keeps metering, and metering is what you pay for. See pricing for how browser-hours are counted.
Production Criteria for Automated Web Browsing
Once the connection works, the real questions begin. These are the criteria that separate a demo from a system that holds up.
Session isolation and profiles
Every agent task should run in a session that cannot see another task's cookies or storage. Isolation is the default in a hosted runtime: each session gets its own browser context. Persistent profiles are the deliberate exception — you opt in when an agent needs to stay logged in across runs. The design question is which tasks share a profile. A support agent reading a customer's account should not share a profile with a bulk scraper. Map profiles to trust boundaries, not to convenience.
Reconnection and session lifetime
Agents are long-running and failure-prone. A session that dies when your worker restarts is a session you cannot use for multi-hour workflows. Look for a runtime that lets you reconnect to a live session over CDP after a client disconnect, and that exposes a session ID you can persist in your own database. That is what makes resumable workflows possible: store the session ID, and on restart, reattach instead of starting over.
Observability
You need to see what the browser saw. A live viewer is the minimum — it lets a human watch a stuck agent and intervene. Beyond that, capture screenshots at decision points, log network requests that failed, and record console errors. If your runtime offers session recordings, use them for post-mortems. Debugging an agent without page state is guesswork.
Network and browser settings
Many sites behave differently by geography and client fingerprint. Hosted runtimes typically expose configurable browser settings — proxy region, user agent, viewport, locale, timezone. Treat these as configuration, not magic. Test against the sites you actually target, and do not assume a setting that works for one site generalizes to another. Where a runtime documents stealth-related settings, read the docs carefully; "configurable" is not the same as "undetectable."
Concurrency and cost control
Automated web browsing at scale is a cost problem as much as a reliability problem. Every open session meters. Set hard limits on concurrent sessions per tenant, close sessions aggressively in finally blocks, and alert on sessions that outlive their expected duration. The pricing page explains how usage is measured; build your own budget guardrails on top of it.
Where Agents Fit In
The rise of AI agents changed the shape of automated web browsing. A traditional script follows a fixed path; an agent decides the path at runtime. That flexibility is powerful and fragile in equal measure. Agents are slower per step, more likely to wander, and harder to test. The runtime has to compensate:
- Deterministic primitives. Give the agent a small set of reliable actions (navigate, click, type, extract) rather than letting it emit arbitrary JavaScript.
- Bounded sessions. Cap session duration and step count so a confused agent cannot burn hours.
- Human handoff. A live viewer plus a pause mechanism lets a human take over when the agent is stuck, then hand control back.
For a deeper look at how agents drive browsers and where the runtime boundary sits, see Remote control browser. The short version: the agent is the decision layer, the runtime is the execution layer, and keeping them separate is what makes each testable.
Common Mistakes to Avoid
A few patterns show up repeatedly in failed automated web browsing deployments:
- Launching a browser per request. Session creation has overhead. Reuse sessions across related steps within a task, and close them when the task ends.
- Ignoring `waitUntil` and selectors.
networkidleis unreliable on modern apps. Wait for specific elements or network responses instead. - Hardcoding timeouts. A 30-second timeout that works in staging fails under load. Make timeouts configurable and log when they fire.
- Skipping the `finally` block. Leaked sessions are the most common cause of surprise bills.
- Treating profiles as free. Shared profiles are a security boundary. Audit who can use which profile.
Getting Started
The fastest path is to connect an existing Playwright or Puppeteer script to a hosted session and see whether the connection model fits your workflow. Start with the documentation for session creation and CDP endpoints, then read Remote web browser for the broader runtime picture. If you are running agents rather than scripts, the AI agent runtime guide covers the agent-specific concerns.
Automated web browsing is not a library choice. It is an infrastructure decision. Pick a runtime that isolates sessions, survives client restarts, shows you what happened, and meters predictably — then spend your engineering time on the automation logic that actually differentiates your product.