BLOG
Browser Automation: A Production Guide for AI Agents
Browser automation for AI agents: how hosted Chromium, CDP, and session isolation work in production, plus what separates demos from reliable systems.

Browser automation is the practice of driving a real browser programmatically — navigating pages, clicking, typing, reading the DOM — instead of calling APIs or scraping static HTML. For AI agents, it is the layer that turns a language model's intent into actual web actions. The hard part is not the first script. It is running that script reliably across many concurrent sessions, on pages that change, behind logins, and under time pressure. This guide covers how browser automation works for agents, where local setups break down, and what to evaluate before you put it in production.
What browser automation actually does
At its core, browser automation controls a browser process over a protocol. The dominant protocol is the Chrome DevTools Protocol (CDP), a JSON-over-WebSocket interface that exposes navigation, input, network, and DOM inspection. Libraries like Playwright and Puppeteer wrap CDP (or equivalent protocols for Firefox and WebKit) in a friendlier API.
A typical automation loop looks like this:
- Launch or connect to a browser instance.
- Create a page or context with the state you need (cookies, headers, viewport).
- Perform actions —
goto,click,fill,waitForSelector. - Extract results from the DOM, network responses, or screenshots.
- Tear down or reuse the session.
For deterministic scripts — regression tests, scheduled scrapes, form submissions — this loop is well understood. For AI agents, the loop is driven by a model that decides the next action at runtime. That shift changes the operational requirements substantially.
Why local browser automation stops scaling
Running Chromium on your laptop or a single CI runner works until it doesn't. The failure modes are predictable:
- Resource contention. Each Chromium instance consumes a large amount of memory. Ten parallel agents on one machine will thrash.
- Environment drift. Browser versions, fonts, and OS libraries differ between your laptop and your server. A selector that works locally fails in CI.
- State leakage. Shared browser profiles carry cookies between runs. One agent's login bleeds into another's session.
- No observability. When an agent fails at step 14, you have a stack trace and nothing else. You cannot see what the page looked like.
- Network constraints. Datacenter IPs get blocked. Residential routing requires proxy infrastructure you now have to operate.
None of these are exotic. They are the normal cost of running stateful, resource-heavy processes. The question is whether you want to own that cost or rent it.
Hosted Chromium as the runtime layer
A hosted browser runtime moves the Chromium process off your infrastructure. You get a WebSocket endpoint, connect over CDP, and drive a real browser running in a provider's datacenter. Remote Browser is one such runtime: it exposes hosted Chromium sessions with CDP access, Playwright/Puppeteer/Selenium compatibility, a live viewer, persistent profiles, configurable proxy and browser settings, and per-session isolation.
The architectural difference matters more than the marketing. When the browser is remote:
- Your agent process is stateless. It holds a connection string, not a browser. Scaling agents means scaling lightweight workers.
- Sessions are isolated by default. Each session gets its own browser context. Cookies and storage do not leak across runs unless you deliberately attach a persistent profile.
- You can watch. A live viewer lets a human inspect an in-flight session — useful for debugging agent failures and for human-in-the-loop approval steps.
- Browser versions are pinned by the provider. You stop debugging font rendering differences between environments.
The trade-off is latency and dependency. Every action round-trips over the network, and you are trusting a third party with session data. For most agent workloads, the latency is acceptable — page loads dominate, not protocol messages — but it is a real consideration for tight interaction loops.
Connecting over CDP with Playwright
The connection pattern is the same whether you use Playwright, Puppeteer, or a raw CDP client. With Playwright, connectOverCDP attaches to an existing browser rather than launching one. The Playwright CDP documentation covers the full API surface.
import { chromium, Browser, Page } from 'playwright';
interface SessionConfig {
cdpEndpoint: string; // wss://... from your runtime's session API
timeoutMs?: number;
}
async function runAgentTask(
config: SessionConfig,
task: (page: Page) => Promise<void>
): Promise<void> {
let browser: Browser | undefined;
try {
browser = await chromium.connectOverCDP(config.cdpEndpoint, {
timeout: config.timeoutMs ?? 30_000,
});
// connectOverCDP attaches to the existing browser; reuse its default context
const context = browser.contexts()[0] ?? (await browser.newContext());
const page = context.pages()[0] ?? (await context.newPage());
page.setDefaultTimeout(15_000);
await task(page);
} catch (err) {
// Surface the endpoint and error class; agent failures are usually
// navigation timeouts or selector misses, not protocol errors.
console.error('Automation failed', {
endpoint: config.cdpEndpoint,
error: err instanceof Error ? err.message : String(err),
});
throw err;
} finally {
// Close the connection, not the browser. The runtime owns the session
// lifecycle; closing here would kill a session you may want to inspect.
await browser?.close();
}
}
// Usage
await runAgentTask(
{ cdpEndpoint: process.env.REMOTE_BROWSER_CDP_URL! },
async (page) => {
await page.goto('https://example.com/login');
await page.fill('#email', process.env.TEST_EMAIL!);
await page.fill('#password', process.env.TEST_PASSWORD!);
await page.click('button[type=submit]');
await page.waitForURL('**/dashboard');
}
);Two details in that snippet are worth calling out. First, connectOverCDP does not create a browser — it attaches to one, so browser.close() disconnects your client without terminating the remote session. That is usually what you want: you can reconnect and inspect a failed run. Second, the default context already exists on the remote side; creating a new one is fine but unnecessary unless you need isolation within a single session.
For a deeper walkthrough of the connection model, see Remote Browser for AI agents.
Comparing automation approaches
The right runtime depends on workload shape. Here is how the common options differ on the dimensions that matter in production.
| Approach | Isolation | Scaling model | Debuggability | Best fit |
|---|---|---|---|---|
| Local Chromium (Playwright/Puppeteer) | Manual — you manage profiles | Vertical; limited by host RAM | Local traces, screenshots | Development, small test suites |
| Self-hosted browser fleet | You build it | Horizontal, but you operate the fleet | Depends on your tooling | Teams with existing infra expertise |
| Hosted browser runtime (Remote Browser) | Per-session by default | Horizontal; sessions are independent | Live viewer, CDP reconnect | Agent workloads, bursty traffic |
| Browser extension automation | Shares the user's real browser | None — one browser | Manual | Personal automation, not agents |
| Headless-only scraping services | Varies | Horizontal | Limited | Static content extraction |
The extension row is worth a note because it comes up often. Extension-based automation drives the user's actual browser, which means it inherits their logins and their fingerprint — but it also means one browser, one user, no concurrency, and no clean way to run it on a server. It is a personal productivity tool, not an agent runtime.
Production criteria to evaluate
If you are choosing a runtime — hosted or self-built — these are the criteria that predict whether it holds up.
Session isolation. Can two concurrent sessions share cookies accidentally? Isolation should be the default, with persistent profiles as an explicit opt-in. Remote Browser's documentation covers how profiles are scoped.
CDP fidelity. Some hosted browsers expose a restricted subset of CDP. If your agent relies on network interception, Page.addScriptToEvaluateOnNewDocument, or performance metrics, verify those domains are available before committing.
Reconnection. Agent runs fail mid-task. Can you reconnect to the same session and inspect state, or does the session die with the connection? Reconnectable sessions turn debugging from guesswork into inspection.
Profile persistence. Logged-in workflows need cookies that survive across sessions. Persistent profiles should be addressable by ID and isolated from each other.
Network configuration. Proxy settings, geolocation, and timezone are configurable browser settings in most runtimes. Be precise about what you need — "stealth" is a vague term that covers everything from header normalization to TLS fingerprinting, and providers implement different subsets.
Usage controls. Concurrency limits and session duration caps are operational guardrails. Check /pricing for current limits rather than assuming a number.
Observability. A live viewer and session recordings are the difference between "the agent failed" and "the agent failed because the page showed a CAPTCHA at step 9."
Where agents differ from scripts
A deterministic Playwright test knows every selector in advance. An agent does not. That difference has concrete implications:
- Timeouts need to be generous and layered. A model may take seconds to decide the next action. Set per-action timeouts, not just a global one.
- Selectors should be resilient. Prefer role-based and text-based locators over brittle CSS paths. Agents often generate selectors from page content, and those selectors break on the next deploy.
- Failure recovery is part of the loop. The agent needs to see errors — a screenshot, a DOM snapshot, an accessibility tree — and decide whether to retry, backtrack, or escalate.
- Human handoff matters. Some steps (CAPTCHAs, 2FA, payment confirmation) should pause for a human. A live viewer makes that practical.
If you are building the agent loop itself, the runtime is the substrate. If you are evaluating whether hosted Chromium improves task success, the accurate web agent analysis covers the mechanism.
Common failure modes and fixes
Navigation timeouts on slow pages. Increase page.setDefaultNavigationTimeout and wait for specific network idle conditions rather than fixed sleeps.
Detached element errors. The DOM changed between locating and clicking. Re-query the element inside a retry wrapper, or use locators that auto-wait.
Session exhaustion. Long-running agents hold sessions open. Set a maximum session duration and design tasks to checkpoint so a fresh session can resume.
Proxy misconfiguration. Verify the proxy is reachable from the browser's network namespace, not just your agent's. A proxy that works from your laptop may not be routable from a datacenter.
Cookie leakage across runs. If two agents see each other's logins, isolation is misconfigured. Check whether you are attaching a shared persistent profile by accident.
Getting started
The shortest path to a working setup:
- Create a session through the runtime API and capture the CDP WebSocket endpoint.
- Connect with
chromium.connectOverCDP(endpoint). - Run your task against the attached page.
- Disconnect without closing the browser; inspect via the live viewer if something failed.
- Persist a profile if the workflow needs to stay logged in.
For a broader look at running Chromium without local setup, see remote browser online. For the control-plane side — driving sessions from code and agents — see remote control browser.
Browser automation for agents is not a solved problem, but the runtime layer is. Hosted Chromium removes the infrastructure burden and gives you the isolation, observability, and reconnection that agent workloads demand. The remaining work — resilient selectors, sensible timeouts, human handoff — is the part that actually determines whether your agent succeeds.