Control Web Browser: A Practical Guide for AI Agents

To control a web browser programmatically, you need a transport layer that lets your code issue navigation, input, and inspection commands to a real browser engine. For AI agents, the most reliable path is the Chrome DevTools Protocol (CDP) over a WebSocket connection to a hosted Chromium instance. This guide explains how that works, where local setups break down, and how to wire an agent to a remote browser that behaves predictably in production.

If you are evaluating runtimes, the short answer is: use CDP as the control plane, connect with Playwright or Puppeteer, and run the browser on infrastructure that isolates sessions, persists profiles, and meters usage. Everything below explains the trade-offs behind that recommendation.

What "Control Web Browser" Actually Means

The phrase covers several distinct capabilities, and conflating them causes most integration problems:

• Navigation control — loading URLs, following redirects, waiting for network idle.
• DOM interaction — clicking, typing, selecting, scrolling, and reading rendered content.
• State inspection — evaluating JavaScript, reading cookies, capturing screenshots, and recording network traffic.
• Session management — persisting logins, isolating cookies between tasks, and resuming work after a crash.
• Observability — watching a session live, replaying it, and debugging failures after the fact.

A local Chrome window driven by a script handles the first three. The last two are where teams hit walls, because they require infrastructure: storage for profiles, a viewer for live sessions, and a control plane that can start and stop browsers on demand.

The Control Plane: CDP Is the Interface

The Chrome DevTools Protocol is the same interface Chrome's own DevTools uses. It exposes domains like Page, Runtime, Network, and Input over a WebSocket. Any tool that speaks CDP — Playwright, Puppeteer, Selenium via a bridge, or a custom client — can drive any CDP-compatible browser, local or remote.

That matters because it decouples your agent logic from where the browser runs. The Chrome DevTools Protocol documentation lists the full domain surface. In practice, agent frameworks use a small subset: navigate, query selectors, dispatch input events, and extract text.

The connection model looks like this:

1. A browser process starts with --remote-debugging-port (or a hosted provider exposes an equivalent endpoint).
2. The client fetches /json/version to get the webSocketDebuggerUrl.
3. The client opens a WebSocket and sends CDP commands as JSON.
4. Responses and events stream back over the same socket.

When the browser is remote, step 1 is replaced by a provider-issued WebSocket URL. Everything after that is identical.

Connecting with Playwright over CDP

Playwright's connectOverCDP method attaches to an existing browser rather than launching one. This is the standard way to control a hosted Chromium session from a TypeScript agent.

import { chromium, Browser, Page } from "playwright";

interface SessionInfo {
  cdpUrl: string;
}

async function connectToHostedBrowser(
  session: SessionInfo
): Promise<{ browser: Browser; page: Page }> {
  // The CDP URL is issued by the runtime when a session is created.
  const browser = await chromium.connectOverCDP(session.cdpUrl, {
    timeout: 30_000,
  });

  // Reuse the default context so cookies and storage persist
  // for the lifetime of the session.
  const context = browser.contexts()[0] ?? (await browser.newContext());
  const page = context.pages()[0] ?? (await context.newPage());

  await page.goto("https://example.com", { waitUntil: "domcontentloaded" });

  // Agents typically extract structured data rather than screenshots.
  const title = await page.title();
  const links = await page.$$eval("a", (nodes) =>
    nodes.slice(0, 10).map((n) => (n as HTMLAnchorElement).href)
  );

  console.log({ title, links });
  return { browser, page };
}

Two details matter for agents. First, connectOverCDP does not create a fresh browser — it attaches to one that already exists, so any state from a previous step is still there. Second, closing the Browser object disconnects the client but does not necessarily terminate the remote browser; session lifecycle is controlled by the runtime, not the client. Check the Playwright CDP guide for the exact semantics of your provider.

Local Chrome vs Hosted Chromium

The decision between running Chrome on your own machine and connecting to a hosted runtime is the central trade-off. Here is how the two compare on the dimensions that affect agents.

Local Chrome is fine for development and one-off scripts. It becomes a liability when an agent needs to run dozens of tasks in parallel, keep logins alive across runs, or recover from a crashed browser without human intervention. The remote browser overview covers the runtime layer in more depth.

Where Agents Differ from Test Suites

A Playwright test suite knows exactly what it will do. An agent does not. That difference changes the control requirements:

• Selectors are dynamic. Agents often generate selectors at runtime from page content, so the runtime must tolerate slow loads and re-renders. Prefer waitUntil: "domcontentloaded" plus explicit element waits over fixed sleeps.
• Sessions are long. A single task may span minutes and dozens of page transitions. Persistent profiles keep authentication state intact across those transitions.
• Failures are ambiguous. When an agent fails, you need to know whether the page changed, the network stalled, or the model chose the wrong action. Live viewers and session recordings answer that question faster than logs.
• Concurrency is bursty. Ten agents may run at once, then zero. A hosted runtime absorbs the burst without provisioning hardware.

If you are comparing agent frameworks, the browser use vs Playwright discussion is relevant: Playwright is the transport, while the agent framework decides what to do with it.

Production Criteria for a Control Layer

Before committing to a runtime, evaluate it against these criteria. They are ordered by how often they cause incidents.

1. CDP compatibility. Does the endpoint accept standard CDP clients without a proprietary shim? If you must rewrite your Playwright code, the abstraction is leaking.
2. Session isolation. Are cookies, storage, and cache scoped per session? Shared state between agents causes cross-contamination that is hard to reproduce.
3. Profile persistence. Can you save and reuse a logged-in profile across sessions? This is the difference between an agent that works once and one that works daily.
4. Live observability. Can you watch a session in real time and inspect the DOM while it runs? Debugging blind is expensive.
5. Network controls. Are proxy and browser settings configurable per session? Many sites behave differently by region.
6. Usage metering. Is billing per browser-hour, per session, or per request? Understand the unit before you scale. See /pricing for current details.
7. Lifecycle API. Can you create, inspect, and terminate sessions programmatically? Orphaned browsers are a common source of cost overruns.

A runtime that satisfies all seven is a control layer you can build on. One that satisfies three is a demo.

Common Failure Modes and Fixes

Most "the agent can't control the browser" reports trace back to a small set of causes.

The WebSocket URL expires. Hosted sessions often have a TTL. If your agent holds a connection open past the limit, commands start failing silently. Fix: reconnect on error and re-fetch the endpoint.

The page never reaches `load`. Single-page apps fire load before content renders. Fix: wait for a specific selector or a network-idle condition, not the default navigation event.

Cookies don't persist. This usually means the agent created a new context per step instead of reusing the session's default context. Fix: attach to browser.contexts()[0] rather than calling newContext().

Concurrent agents interfere. Two agents sharing one browser will overwrite each other's tabs and storage. Fix: one session per agent, enforced at the runtime level.

The browser is detected as automated. Some sites block headless or automation-flagged browsers. Configurable browser settings and proxy options help, but no runtime guarantees access to every site. Treat detection as a risk to manage, not a problem to eliminate.

Wiring an Agent to a Remote Browser

The integration pattern is consistent across frameworks:

1. Request a session from the runtime API. The response includes a CDP URL and a session ID.
2. Connect with chromium.connectOverCDP(cdpUrl).
3. Run the agent loop: observe page state, decide on an action, execute it via Playwright.
4. On completion or failure, terminate the session through the API so the browser is released.

The remote browser online guide walks through the session request step in detail. For teams building on MCP-based tooling, the same CDP endpoint works — the MCP server is just another client.

One practical note: keep the agent's action space small. A tool that exposes click, type, navigate, and extract is easier to debug than one that exposes raw CDP. The runtime handles the transport; the agent should handle intent.

When to Move Off Local Chrome

Move to a hosted runtime when any of these become true:

• You need more than a handful of concurrent sessions.
• Logins must survive across runs.
• You are spending time restarting browsers or clearing stale profiles.
• You cannot reproduce a failure because you cannot see what the agent saw.
• Your CI or server environment cannot run a headed browser.

Until then, local Chrome with Playwright is a reasonable starting point. The migration is mostly a change in connection code, not agent logic, because CDP is the same interface on both sides. The remote control browser article covers the migration path for teams already running automation locally.

Summary

Controlling a web browser from an agent comes down to three decisions: use CDP as the control plane, use Playwright or Puppeteer as the client, and run the browser somewhere that isolates sessions and meters usage. Local Chrome works for development; hosted Chromium is what holds up when agents run continuously, in parallel, and against sites that require persistent logins.

Start with the connection code above, verify your runtime against the seven production criteria, and check /pricing before you scale. The transport is the easy part — the runtime is what determines whether your agent works tomorrow.