BLOG
Web Browser Automation: A Production Runtime Guide
Web browser automation for AI agents and test suites: how hosted Chromium, CDP, and session isolation change reliability, cost, and operations.

Web browser automation is the practice of driving a real browser — loading pages, clicking, typing, extracting data — from code rather than a human. In 2026 that code is increasingly an AI agent, and the hard part is no longer writing the script. It is running that script reliably, at concurrency, against sites that change, behind IPs that get blocked, and on infrastructure you do not want to babysit. This guide covers what web browser automation actually requires in production, where local Chrome setups break down, and how to evaluate a hosted runtime before you commit.
If you already know you want a remote execution layer, start with Remote Browser for AI agents and come back here for the decision criteria.
What web browser automation means in practice
At its core, browser automation is a control loop: your program sends a command (navigate, click, type, evaluate JavaScript), the browser executes it, and your program reads the result. The transport for that loop is almost always the Chrome DevTools Protocol (CDP), a JSON-over-WebSocket interface that Chromium exposes on a debugging port.
Three layers sit on top of CDP:
- Libraries — Playwright, Puppeteer, and Selenium wrap CDP (or the WebDriver protocol) in ergonomic APIs. Playwright's
connectOverCDPand Puppeteer'sbrowserWSEndpointare the two most common entry points for attaching to a browser you did not launch. - Agents — frameworks like browser-use, Stagehand, and the agent-browser CLI take a natural-language task, decide which actions to take, and call the library underneath. They add planning, retries, and often a vision or DOM-extraction step.
- Runtimes — the process, machine, and network the browser actually runs on. This is where most production failures live, and it is the layer most teams under-invest in.
The first two layers are well documented and largely solved. The third is the one that decides whether your automation survives contact with real traffic.
Why local Chrome stops working at scale
A single Playwright script on a laptop is fine for development. The moment you run it in CI, on a server, or across dozens of parallel agent tasks, a predictable set of problems appears:
Resource contention. Each headless Chromium instance consumes a large amount of RAM and a meaningful slice of CPU. Ten parallel sessions on a 4 GB container will thrash. Twenty will crash. You end up writing a queue, a pool manager, and a health checker — infrastructure work that has nothing to do with the task you actually care about.
Environment drift. The browser version on your laptop, in CI, and on the production worker will diverge. A selector that works in Chrome 128 may behave differently in 131. Pinning versions helps, but now you own a browser distribution pipeline.
Network identity. Every request from a single datacenter IP looks like a bot to sites that care. Residential and mobile proxies, per-session IP rotation, and consistent cookie jars are not optional for anything touching commerce, travel, or social platforms. Wiring that into a local browser is possible but fiddly.
Session state. Logged-in workflows need persistent profiles — cookies, localStorage, IndexedDB. Local profiles are files on disk that you must back up, migrate, and isolate between tenants. One leaked profile is a security incident.
Observability. When an agent fails at step 14 of a 20-step task, you need to see what the browser saw. Screenshots, video, console logs, and network traces have to be captured continuously, not reconstructed after the fact.
None of these are reasons browser automation is hard in principle. They are reasons the *runtime* is hard, and they are exactly what a hosted browser abstracts away.
The runtime decision: local, self-hosted, or hosted
There are three realistic options, and they trade off differently depending on team size and workload shape.
| Dimension | Local Chrome | Self-hosted Playwright grid | Hosted browser runtime |
|---|---|---|---|
| Setup time | Minutes | Days to weeks | Minutes |
| Browser version management | Manual | You own it | Provider owns it |
| Concurrency | Limited by your machine | Limited by your cluster | Scales with your plan |
| Proxy / IP control | Manual per browser | You build it | Configurable per session |
| Persistent profiles | Files on disk | You build storage | Managed, isolated per session |
| Live debugging | Local DevTools | Custom tooling | Built-in viewer |
| Cost model | Hardware + your time | Fixed infra + ops | Per browser-hour |
| Best fit | Prototyping, single tasks | Large orgs with platform teams | Agents, CI, variable load |
The self-hosted column is not a strawman — for organizations with existing Kubernetes capacity and a platform team, running your own Chromium pool can be the right call. But it is a product in itself. If your differentiation is the agent or the test suite, not the browser fleet, hosting the fleet is usually a distraction.
Hosted runtimes win on the axes that matter for agents specifically: burst concurrency, per-session isolation, and the ability to attach from anywhere over CDP without shipping a browser binary to every worker.
How hosted Chromium sessions actually work
A hosted runtime exposes a browser as an API resource. The lifecycle is consistent across providers:
- Create a session. An API call returns a session ID and a CDP WebSocket endpoint, something like
wss://…/devtools/browser/<id>. - Connect. Your Playwright, Puppeteer, or Selenium client attaches to that endpoint. No local browser is launched.
- Drive. Your code or agent issues commands over CDP. The browser runs on the provider's infrastructure.
- Observe. A live viewer streams the session so a human can watch, intervene, or debug.
- Terminate. The session is torn down, and metering stops.
Here is the minimal Playwright path against a hosted endpoint:
import { chromium, Browser, Page } from "playwright";
async function runTask(cdpEndpoint: string): Promise<string> {
// Attach to the hosted browser instead of launching a local one.
const browser: Browser = await chromium.connectOverCDP(cdpEndpoint);
// Hosted sessions usually start with one default context.
const context = browser.contexts()[0] ?? (await browser.newContext());
const page: Page = context.pages()[0] ?? (await context.newPage());
try {
await page.goto("https://example.com/login", {
waitUntil: "domcontentloaded",
timeout: 30_000,
});
await page.getByLabel("Email").fill("agent@example.com");
await page.getByLabel("Password").fill(process.env.APP_PASSWORD!);
await page.getByRole("button", { name: "Sign in" }).click();
await page.waitForURL("**/dashboard", { timeout: 15_000 });
const heading = await page.getByRole("heading", { level: 1 }).innerText();
return heading;
} finally {
// Close the connection, not the remote browser, unless you own its lifecycle.
await browser.close();
}
}Two details matter here. First, connectOverCDP is Chromium-only — Playwright's own docs note that CDP attachment does not extend to Firefox or WebKit, so a hosted Chromium runtime covers Chromium workloads and nothing else. Second, browser.close() on a connected browser disconnects your client; whether it also terminates the remote session depends on the provider's semantics. Check that before you build cleanup logic around it.
Puppeteer users get the same shape via puppeteer.connect({ browserWSEndpoint }). Selenium attaches through a remote WebDriver URL if the provider exposes one. The transport differs; the mental model does not.
What to evaluate in a hosted runtime
Not all hosted browsers are equivalent, and the differences show up in month two, not day one. Use this as a checklist.
CDP fidelity. Can you attach with the library you already use, at the version you already run? A runtime that only exposes a proprietary action API forces a rewrite. Prefer one that gives you raw CDP plus a convenience layer.
Session isolation. Each session should have its own browser process, profile directory, and cookie jar. Shared state between sessions is a correctness bug and a security bug. Ask how profiles are scoped and whether they persist across sessions when you want them to.
Persistent profiles. For logged-in workflows, you need to save and reuse a profile — cookies, storage, extensions — across sessions. Confirm the profile is tied to your account, not a shared pool, and that you can delete it on demand.
Network controls. Configurable proxy settings per session, including residential options where the provider supports them, plus the ability to pin a session to a stable egress IP when a site's fraud checks require consistency.
Live viewer and recordings. A real-time view of the session is the fastest debugging tool you will use. Recordings and console/network logs are the second. Both should be available without extra instrumentation.
Usage controls and metering. Understand what a "browser-hour" means — wall-clock session time, active page time, or something else — and how concurrency is billed. Current rates and plan limits are on the pricing page; do not plan capacity from secondhand numbers.
Data handling. For regulated workloads, ask about retention, DPA availability, and whether session data is stored at rest. This is where enterprise procurement stalls, so resolve it early.
Where agents change the requirements
Classic automation runs a fixed script. An agent decides at runtime what to do next, which shifts the runtime requirements in three ways.
Latency compounds. An agent may take 30–100 actions per task. A 200 ms overhead per action becomes 6–20 seconds of pure waiting. Session startup time and CDP round-trip latency matter more than raw page-load speed.
Failures are semantic. A script that fails throws an exception. An agent that fails may confidently report success on the wrong page. The runtime needs to make the browser state observable — screenshots, DOM snapshots, current URL — so the agent's own verification step has something to check.
Concurrency is bursty. Agent workloads spike. A queue of 50 tasks arrives at once, then nothing for an hour. Fixed-capacity self-hosted pools handle this badly; metered hosted sessions handle it by design, provided the provider's concurrency limits match your burst profile.
For a deeper treatment of the agent-specific runtime layer, see remote browser for AI agents. For the operational side of running Chromium without a local install, remote browser online covers the setup path.
A practical production checklist
Before you move web browser automation from a prototype to something you depend on, confirm each of these:
- Version pinning. Your Playwright/Puppeteer version and the hosted Chromium version are compatible. Pin both in CI.
- Retry policy. Transient failures (navigation timeouts, 5xx) retry with backoff; semantic failures (wrong page, missing element) do not retry blindly.
- Session budget. Every task has a maximum session duration and a maximum action count. Runaway agents are expensive.
- Credential handling. Secrets are injected at runtime, never baked into profiles or scripts. Rotate on a schedule.
- Observability. Every session emits a recording, a final screenshot, and structured logs keyed by task ID.
- Cleanup. Sessions are explicitly terminated on success and failure. Orphaned sessions are the most common source of surprise bills.
- Egress consistency. If a site ties sessions to IPs, pin the egress IP for the duration of the workflow.
None of this is exotic. It is the same discipline you would apply to any distributed system, applied to a browser fleet you did not have to build.
Common failure modes and what they tell you
"It works locally but not in CI." Almost always a browser version mismatch or a missing system dependency. Hosted runtimes eliminate the class by giving every session the same image.
"The agent gets logged out mid-task." Profile or cookie isolation is misconfigured, or the session was recycled. Persistent profiles fix this if the provider scopes them correctly.
"We get blocked after 20 requests." Single-IP egress. You need per-session proxy configuration, and possibly residential IPs for the target site.
"Sessions hang and we get billed." No session timeout. Set one at the API level, not just in your client code.
"Debugging takes hours." No live viewer or recording. You are reconstructing browser state from logs, which is guesswork.
Each of these is a runtime problem, and each has a runtime fix. That is the argument for treating the browser as infrastructure rather than a library call.
Getting started
The shortest path is to pick one workflow — a login-and-extract task, a checkout smoke test, a form submission — and run it against a hosted session with your existing Playwright code. If connectOverCDP works unchanged, you have validated the core integration. From there, add persistent profiles, then proxies, then observability, in that order.
The documentation covers session creation, CDP endpoints, profile management, and the live viewer. If you are still deciding between a hosted runtime and running your own fleet, remote control browser walks through the control-plane trade-offs, and remote web browser covers the practical runtime setup.
Web browser automation is not hard because browsers are hard. It is hard because running browsers reliably, at concurrency, with clean network identity and observable state, is a systems problem. Solve the systems problem once, and the automation — scripted or agentic — becomes the easy part.