Other Web: The Runtime Layer Behind Agent Browsers

"Other web" is the shorthand people use when they mean the web as an agent sees it: not the page you and I read, but the DOM tree, the network requests, the accessibility snapshot, and the CDP session that a program drives. When someone searches for *other web*, they usually want to know how an AI agent or automation script navigates a site that was built for humans. The answer is a browser runtime. This guide explains what the other web is, why a hosted Chromium runtime beats a local Chrome install for agents, and how to wire one up with Playwright over CDP.

What "Other Web" Actually Means

The phrase is loose, so pin it down. There are three layers people conflate:

• The human web. Rendered pixels, visual layout, mouse targets. Built for eyes and hands.
• The document web. HTML, JSON, RSS, sitemaps. Built for crawlers and APIs.
• The other web. A live browser session that an agent controls programmatically — DOM queries, clicks, form fills, navigation, screenshots, network interception. Built for code.

A web research tool, a chrome mcp server, a cursor browser agent, a manus browser operator, or a terminal browser all sit on that third layer. They differ in interface, but underneath they need the same thing: a real browser engine they can drive deterministically. That engine is Chromium, and the control surface is the Chrome DevTools Protocol (CDP).

The distinction matters because most reliability problems in agent browsing are not model problems. They are runtime problems. A local Chrome tab that a human is also using, a headless browser that crashes mid-task, a session that loses cookies between steps — these are infrastructure failures, and they show up as "the agent got confused." It didn't. The browser did.

Why Agents Need a Different Browser Than You Do

A person tolerates a slow page, a cookie banner, and a re-login. An agent at scale does not. The other web has different requirements:

• Determinism. The same navigation should produce the same DOM. Local Chrome with extensions, autofill, and a logged-in profile is not deterministic.
• Isolation. Two concurrent agents must not share cookies, localStorage, or a single tab. Session isolation is a hard requirement, not a nicety.
• Observability. When a step fails, you need a screenshot, a DOM snapshot, and a console log from that exact moment — not a guess.
• Persistence where you want it. Some workflows need a saved login across runs (a persistent profile). Others need a clean slate every time. You need both, selectable per session.
• Addressability. The browser must be reachable over a WebSocket endpoint from wherever your code runs — a laptop, a CI job, a cloud worker.

A local chrome.exe gives you none of these by default. A hosted Chromium runtime gives you all of them as primitives. That is the core argument for treating the browser as infrastructure rather than a desktop app.

The Runtime Choices, Compared

If you are evaluating how to give an agent a browser, you are really choosing a runtime. Here is how the common options stack up on the criteria above.

The trade-off is explicit. Self-hosting gives you maximum control and maximum maintenance. A hosted runtime trades some control for a managed lifecycle: sessions spin up on demand, expose a CDP endpoint, and tear down cleanly. For most agent teams, the second option is the one that survives contact with production. If you want the longer version of this argument, see Remote Browser for AI agents.

Browser Use vs Playwright Is the Wrong Question

A frequent search is *browser use vs playwright*. They are not competitors. Playwright is a driver — it speaks CDP and gives you a clean API for pages, locators, and assertions. Browser Use is an agent framework that decides *what* to do. The runtime is the third thing, and it is the one that determines whether either works at scale.

You can run Browser Use on top of Playwright on top of a hosted Chromium session. You can also run a hand-written Playwright script with no LLM at all. The runtime is orthogonal to both. Choosing a runtime first, then a driver, then an agent framework, keeps the decision tree honest.

Wiring an Agent to the Other Web

The mechanics are the same whether your agent is a chrome mcp client, an auto browser loop, or a scripted workflow. You get a CDP WebSocket endpoint from the runtime, connect Playwright to it, and drive pages.

import { chromium, type Browser, type Page } from "playwright";

// The runtime returns a CDP WebSocket URL when a session is created.
// Treat it as a secret; it grants full control of that browser.
const cdpEndpoint = process.env.REMOTE_BROWSER_CDP_URL;
if (!cdpEndpoint) throw new Error("Missing REMOTE_BROWSER_CDP_URL");

async function run(): Promise<void> {
  const browser: Browser = await chromium.connectOverCDP(cdpEndpoint);

  // A hosted session usually starts with one context already open.
  const context = browser.contexts()[0] ?? (await browser.newContext());
  const page: Page = context.pages()[0] ?? (await context.newPage());

  page.setDefaultTimeout(30_000);

  await page.goto("https://example.com/research", {
    waitUntil: "domcontentloaded",
  });

  // Prefer role/label locators over CSS — they survive markup churn.
  await page.getByRole("searchbox", { name: /search/i }).fill("quarterly report");
  await page.getByRole("button", { name: /search/i }).click();

  await page.waitForLoadState("networkidle");

  const results = await page
    .getByRole("listitem")
    .evaluateAll((nodes) =>
      nodes.slice(0, 10).map((n) => (n as HTMLElement).innerText.trim())
    );

  console.log(results);

  // Disconnect without killing the session if you want to resume later.
  await browser.close();
}

run().catch((err) => {
  console.error(err);
  process.exitCode = 1;
});

A few details that matter in practice:

• `connectOverCDP` vs `launch`. You are attaching to a browser that already exists. Do not call chromium.launch() against a hosted runtime; you will start a second, local browser and wonder why nothing works.
• Contexts, not tabs. Reuse the existing context so cookies and storage persist within the session. Creating a new context gives you a clean slate — useful, but only when you mean it.
• Timeouts are policy. Set them explicitly per page. Defaults hide slow navigations until they fail in production.
• Close vs stop. browser.close() disconnects your client. Whether the session is torn down depends on the runtime's lifecycle settings. Check the documentation for how session stop is triggered.

For the connection-level details — endpoint formats, reconnects, and what happens when a WebSocket drops — see Remote Browser online.

Production Criteria for the Other Web

Once the demo works, the questions change. These are the criteria that separate a runtime you can ship from one you cannot.

Session lifecycle and isolation

Every task should get its own session, or a deliberately reused one. Shared sessions leak state: a cookie from task A changes the DOM in task B, and your agent's behavior becomes non-reproducible. Look for per-session isolation as a default, with persistent profiles as an opt-in for workflows that genuinely need a saved login.

Profiles and state

A persistent profile is how you avoid re-authenticating on every run. The trade-off is that profiles accumulate state and can drift. Treat them as named, versioned resources — one profile per account or per workflow — rather than a single global profile. The profile list pattern in most runtimes lets you enumerate and reuse them explicitly.

Proxies and browser settings

Many sites behave differently by geography and IP reputation. Configurable browser settings — proxy selection, locale, timezone, user agent — let you match the environment the task expects. Be precise about what you are claiming here: these are configuration knobs, not guarantees of undetectability. If a site actively blocks automation, no setting fixes that; you need a different approach or explicit permission.

Live viewer and debugging

When an agent fails at step 7 of 12, a stack trace is not enough. You need to see the page. A live viewer that streams the session lets a human watch, pause, and take over. This is the single highest-leverage debugging feature for agent browsing, and it is the reason many teams move off local headless Chrome. See remote control browser for how takeover works in practice.

Usage controls

Browser time is a real cost. Metering per session, concurrency limits, and hard timeouts prevent a runaway loop from becoming a runaway bill. Check pricing for current rates and how usage is measured rather than assuming a number.

Compatibility

Your driver should not dictate your runtime. Playwright, Puppeteer, and Selenium all speak CDP, so a runtime that exposes a standard CDP endpoint works with all three. That portability is worth more than any single feature — it means you can change drivers without changing infrastructure.

Common Failure Modes and Fixes

Most of these are not model failures. They are the other web behaving like software — which is exactly what it is.

Where This Leaves You

The "other web" is not a metaphor. It is a concrete stack: a Chromium engine, a CDP session, a driver, and an agent on top. The reliability of your agent is bounded by the reliability of that stack, and the stack is easiest to make reliable when the browser is hosted, isolated, observable, and addressable over a standard protocol.

If you are starting from scratch, the sequence is: pick a runtime, connect with Playwright over CDP, build one workflow end to end, then add persistence, proxies, and metering as the workflow demands them. If you are moving off local Chrome, the migration is mostly a change of connection string and a shift in how you think about session lifecycle.

Start with the documentation to get a CDP endpoint, then read Remote Browser for AI agents for the architecture behind it. The other web is already there. The job is giving your agent a browser that can actually reach it.