Agent-Browser Schema: A Practical Guide for AI Agents

# Agent-Browser Schema: A Practical Guide for AI Agents

The agent-browser schema is the contract between an AI agent and a browser runtime. It describes what the agent wants to do — navigate, click, type, extract, wait — in a structured form that a runtime can execute deterministically. If you have used the agent-browser CLI from Vercel Labs, you have already seen a version of this schema: a small set of typed commands that map to Chrome DevTools Protocol (CDP) operations. This post explains what the schema contains, where it breaks down in production, and how to run it against hosted Chromium instead of a local browser.

If you are still deciding whether you need a remote runtime at all, start with Remote Browser for AI Agents. If you already know you need one, keep reading.

What the agent-browser schema actually is

A schema, in this context, is not a database schema. It is a command vocabulary. The agent emits a structured object — usually JSON — and the runtime translates it into browser actions. The schema exists because free-form natural language is a poor transport format for browser control. An LLM can say "click the login button," but the runtime needs to know *which* element, *how* to wait, and *what* to do if the click fails.

The agent-browser CLI popularized a minimal version of this. Its commands look roughly like:

• open <url> — navigate the active page
• click <selector> — dispatch a click on a resolved element
• type <selector> <text> — focus an element and enter text
• read <selector> — extract text or attributes
• screenshot — capture the current viewport
• wait <condition> — block until a selector, network state, or timeout

Each command is a schema entry. The agent does not need to know about CDP domains, frame trees, or execution contexts. The runtime owns that translation.

Why a schema beats raw CDP for agents

Raw CDP is powerful but verbose. A single click involves DOM.getDocument, DOM.querySelector, DOM.getBoxModel, Input.dispatchMouseEvent (twice, for press and release), and error handling for each. That is fine for a Playwright maintainer. It is not fine for an LLM that has a token budget and a task deadline.

A schema compresses that sequence into one intent. The trade-off is that the schema must anticipate the failure modes the agent will hit. A click command that does not specify a wait strategy will race the page. A read command that does not specify a frame will read the wrong document. Good schemas make these choices explicit.

The fields that matter in production

Most agent-browser schemas converge on the same core fields. The differences show up in the optional ones, and those are where production reliability lives.

Two fields deserve extra attention. First, waitFor. Agents that skip it produce flaky runs that look like model failures but are actually timing failures. Second, frame. Single-page apps embed iframes for payments, auth, and chat widgets. An agent that cannot address a frame will silently act on the wrong element.

A minimal TypeScript schema

Here is a compact schema definition you can drop into a TypeScript project. It is intentionally narrow — six actions cover most agent workloads.

type AgentAction =
  | { action: "open"; url: string; waitFor?: WaitCondition }
  | { action: "click"; selector: string; frame?: string; timeout?: number }
  | { action: "type"; selector: string; text: string; frame?: string }
  | { action: "read"; selector: string; extract?: "text" | "value" | "href" }
  | { action: "wait"; condition: WaitCondition; timeout?: number }
  | { action: "screenshot"; fullPage?: boolean };

type WaitCondition =
  | { kind: "selector"; selector: string }
  | { kind: "response"; urlPattern: string; status?: number }
  | { kind: "timeout"; ms: number };

The WaitCondition union is the important part. It forces the agent to declare *what* it is waiting for rather than sleeping for an arbitrary duration. That single constraint removes a large class of flaky runs.

Running the schema against hosted Chromium

A schema is only useful if something executes it. Locally, you can wire the schema to Playwright and launch Chromium on your machine. That works until you need concurrency, persistent profiles, or a browser that survives your laptop sleeping.

The production path is to connect the same schema to a hosted Chromium session over CDP. Playwright supports this directly via chromium.connectOverCDP(). The Playwright CDP documentation covers the connection semantics; the short version is that you pass a WebSocket endpoint and get a Browser object back.

import { chromium } from "playwright";

const endpoint = process.env.REMOTE_BROWSER_WS_URL!;

const browser = await chromium.connectOverCDP(endpoint);
const context = browser.contexts()[0] ?? (await browser.newContext());
const page = context.pages()[0] ?? (await context.newPage());

async function run(action: AgentAction) {
  switch (action.action) {
    case "open":
      await page.goto(action.url, { waitUntil: "domcontentloaded" });
      break;
    case "click":
      await page.locator(action.selector).click({ timeout: action.timeout ?? 10_000 });
      break;
    case "type":
      await page.locator(action.selector).fill(action.text);
      break;
    case "read":
      return await page.locator(action.selector).innerText();
    case "wait":
      if (action.condition.kind === "selector") {
        await page.locator(action.condition.selector).waitFor();
      }
      break;
    case "screenshot":
      return await page.screenshot({ fullPage: action.fullPage ?? false });
  }
}

await run({ action: "open", url: "https://example.com" });
await run({ action: "click", selector: "a[href='/login']" });
await browser.close();

Two details matter here. First, connectOverCDP does not launch a browser — it attaches to one that already exists. That means the session lifecycle is owned by the runtime, not your script. Second, browser.contexts()[0] may already contain pages if the session was created with a persistent profile. Handle that case explicitly rather than assuming a blank slate.

If you want the same pattern without managing the WebSocket endpoint yourself, see Remote Browser Online for the connection flow and Remote Control Browser for the control-plane model.

Where schemas break in production

Schemas fail in predictable ways. Knowing the failure modes is more useful than knowing the happy path.

Selector drift

Generated selectors — hashed class names, positional XPath — break on the next deploy. The schema does not cause this, but it exposes it. Mitigations: prefer data-testid attributes, use role-based locators where the DOM supports them, and add a waitFor that confirms the element is the one you expect, not just present.

Frame ambiguity

An agent that clicks "Pay" on a checkout page may be clicking a button inside a payment iframe or a decoy outside it. The schema needs a frame field, and the runtime needs to resolve it deterministically. Hosted runtimes that expose the full frame tree over CDP make this tractable; runtimes that flatten frames do not.

Session state leakage

If two agent runs share a browser context, cookies and localStorage bleed between them. The schema does not prevent this — the runtime must. Look for session isolation as a first-class feature, not a configuration flag you have to remember to set.

Timeout stacking

Per-action timeouts are good. Per-action timeouts that exceed the task budget are not. If your task budget is 60 seconds and each action allows 30, a single retry can blow the budget. Set action timeouts as a fraction of the task budget and let the orchestrator decide when to abandon.

Extraction ambiguity

read is underspecified. Does it return textContent, innerText, or value? For an <input>, those differ. For a hidden element, innerText may be empty while textContent is not. Make the extraction mode explicit in the schema and default to the least surprising option.

Schema design criteria for production

If you are designing or adopting an agent-browser schema, evaluate it against these criteria.

• Small verb set. Every verb is code you maintain and a failure mode you debug. Six to ten actions cover most workloads.
• Explicit waits. The schema should require a wait condition, not offer it as an optional afterthought.
• Frame addressing. First-class, not bolted on.
• Deterministic extraction. Specify the property, not just the element.
• Structured errors. A failed action should return a typed error with the selector, the frame, and the last known page state — not a stack trace.
• Runtime-agnostic. The schema should not assume a specific browser vendor. It should map cleanly to CDP, and by extension to Playwright, Puppeteer, and Selenium.

The last point is what makes a schema portable. If your schema only works against one runtime, you have coupled your agent logic to your infrastructure. That coupling is expensive to undo.

Schema vs. raw Playwright vs. raw CDP

Teams often ask whether they need a schema at all. The answer depends on who is emitting the commands.

For a single script that runs once, raw Playwright is fine. For an agent that decides its own next action, a schema is the right abstraction. For an agent that runs thousands of times a day, a schema plus a hosted runtime is the only configuration that scales without a dedicated infra team.

Connecting the schema to a hosted runtime

Remote Browser exposes hosted Chromium sessions over CDP, which means any schema that maps to CDP works without modification. The runtime handles session isolation, persistent profiles, configurable browser settings, and a live viewer for debugging. You bring the schema; the runtime brings the browser.

The practical setup is three steps:

1. Define your schema as a TypeScript union or a JSON Schema document.
2. Write an executor that maps each action to a Playwright call.
3. Point the executor at a hosted session endpoint instead of a local launch.

Step three is the one that changes your operational profile. Local Chromium ties your agent to a machine. A hosted session ties it to an endpoint, which means your agent can run from a queue worker, a serverless function, or a CI job without carrying a browser binary.

For current session limits and pricing, see /pricing. For the full connection reference, including CDP endpoints and profile handling, see /documentation.

What to do next

Start by writing your schema down. Even a rough union type forces you to decide what your agent can and cannot do. Then run it locally against Playwright to validate the happy path. When you hit the first flaky selector or the first iframe you cannot address, you will know whether your schema is the problem or your runtime is.

Most teams find it is the runtime. A schema that assumes a stable, isolated, always-available browser is a schema that works. A schema that assumes the browser is on someone's laptop is a schema that will fail in production. The agent-browser schema is the contract; the hosted runtime is what makes the contract enforceable.