BLOG
Computer Use Agents: Runtime Architecture for Production
Computer use agents drive real browsers to complete tasks. Learn how they work, where they break, and how to run them on hosted Chromium.

Computer use agents are AI systems that complete tasks by operating a real graphical interface — clicking, typing, scrolling, and reading what is on screen — rather than calling a purpose-built API. In a browser context, that means an agent perceives a rendered page, decides on an action, and executes it against a live Chromium instance. The model is the decision layer; the browser is the execution layer. Most production failures in computer use agents trace back to the second layer, not the first.
This guide covers how computer use agents actually work, the runtime properties that determine whether they succeed in production, and how to wire one to a hosted browser over CDP. If you are still deciding between a local and remote runtime, the remote browser for AI agents overview is a useful starting point.
What a computer use agent actually is
The term is used loosely, so it helps to be precise. A computer use agent is a loop with four stages:
- Perception. The agent receives a representation of the current screen — a screenshot, an accessibility tree, a DOM snapshot, or some combination.
- Reasoning. A model interprets that representation against the task and selects the next action.
- Action. The action is translated into a concrete input event: a click at coordinates, a keystroke, a navigation, a form fill.
- Observation. The agent waits for the page to settle and loops back to perception.
What distinguishes computer use agents from classic automation is that the action space is open-ended. A Playwright script knows exactly which selector it will click. A computer use agent decides at runtime, which is why it can handle interfaces it was never explicitly programmed for — and why it can also fail in ways a script never would.
The perception channel matters more than most teams expect. Screenshot-only agents are flexible but expensive per step and prone to misreading dense UIs. Accessibility-tree agents are cheaper and more deterministic but miss anything not exposed semantically. Many production systems combine both: a structured tree for the common case, a screenshot as a fallback when the tree is ambiguous.
Why the runtime decides whether they work
A computer use agent is only as reliable as the browser it drives. Four runtime properties dominate outcomes.
Session stability. Agents run long. A task that takes forty model calls can take minutes, and any browser crash mid-task loses the entire trajectory. A runtime that keeps sessions alive, isolates them from other tenants, and survives worker restarts is not a nice-to-have — it is the difference between a demo and a product.
State continuity. Many real tasks span multiple sessions: log in once, come back tomorrow, resume where you left off. Persistent profiles — saved cookies, local storage, and session state attached to a named profile rather than a single browser process — turn a stateless agent into one that can operate an account over time. The browser session API covers how profiles and sessions are managed on Remote Browser.
Determinism of the control surface. The agent's actions must map cleanly onto browser events. CDP gives you that: Input.dispatchMouseEvent, Input.dispatchKeyEvent, Page.navigate, and the rest are the same primitives a human's input ultimately produces. Runtimes that abstract too far away from CDP make debugging painful, because you cannot replay exactly what the agent did.
Observability. When an agent fails, you need to see the screen it saw. A live viewer plus session recordings turns a black-box failure into a reproducible one. Without it, teams end up guessing at model behavior when the actual bug was a page that never finished loading.
Local Chrome vs hosted Chromium for computer use agents
The default starting point for most teams is a local Chrome instance driven by Playwright or Puppeteer. It works, and for a single developer iterating on prompts it is the fastest path. It stops working when you need concurrency, isolation, or uptime.
| Criterion | Local Chrome | Hosted Chromium (Remote Browser) |
|---|---|---|
| Setup per session | Install, launch, manage lifecycle | One API call, session returned with CDP endpoint |
| Concurrency | Limited by the host machine | Scales with the provider's capacity |
| Isolation between agents | Shared process tree, shared profile risk | Session-level isolation |
| Persistence across runs | Manual profile directory management | Named persistent profiles |
| Proxy and browser settings | Configured per launch, easy to drift | Configurable per session |
| Live debugging | Attach a local devtools window | Built-in live viewer |
| Failure recovery | Process dies, state lost | Session lifecycle managed by the runtime |
| Cost model | Your compute, your ops time | Metered per browser-hour — see /pricing |
The trade-off is real in both directions. Local Chrome gives you total control and zero marginal cost if you already have the hardware. Hosted Chromium gives you operational properties — isolation, persistence, observability, scale — that are expensive to build yourself and easy to underestimate until you hit them in production.
A useful rule: if your agent runs more than a handful of concurrent sessions, needs to survive restarts, or needs to operate logged-in accounts over days, the local model becomes an infrastructure project. That is the point at which a hosted runtime pays for itself.
Connecting a computer use agent over CDP
The integration path is the same one Playwright and Puppeteer already use. You request a session, receive a CDP WebSocket endpoint, and connect. The agent's action layer then issues input events against that connection.
import { chromium, Browser, Page } from "playwright";
interface SessionResponse {
id: string;
cdpUrl: string;
}
async function openAgentSession(): Promise<{ browser: Browser; page: Page }> {
// Request a hosted Chromium session from the Remote Browser API.
const res = await fetch("https://api.remote-browser.dev/v1/sessions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${process.env.REMOTE_BROWSER_API_KEY}`,
},
body: JSON.stringify({
profile: "agent-account-01", // persistent profile name
viewport: { width: 1280, height: 800 },
}),
});
if (!res.ok) {
throw new Error(`Session creation failed: ${res.status}`);
}
const session: SessionResponse = await res.json();
// Connect over CDP to the hosted browser.
const browser = await chromium.connectOverCDP(session.cdpUrl);
const context = browser.contexts()[0];
const page = context.pages()[0] ?? (await context.newPage());
return { browser, page };
}
async function agentStep(page: Page, action: { type: string; x?: number; y?: number; text?: string }) {
switch (action.type) {
case "click":
await page.mouse.click(action.x!, action.y!);
break;
case "type":
await page.keyboard.type(action.text!);
break;
case "navigate":
await page.goto(action.text!, { waitUntil: "domcontentloaded" });
break;
default:
throw new Error(`Unsupported action: ${action.type}`);
}
// Let the page settle before the next perception step.
await page.waitForLoadState("networkidle").catch(() => {});
}
export { openAgentSession, agentStep };Two details in that snippet matter more than they look.
First, connectOverCDP attaches to an existing browser rather than launching one. That is the correct primitive for hosted runtimes — you are not managing the browser process, only the connection to it. The Playwright CDP documentation is the authoritative reference for the connection semantics.
Second, the settle step. Computer use agents frequently act before the page has finished updating, which produces clicks on elements that are about to disappear. networkidle is a blunt instrument, but it is better than nothing; production systems usually combine it with an explicit wait for a specific DOM condition the agent expects.
Production criteria to evaluate before you commit
When you compare runtimes for computer use agents, evaluate against these criteria rather than feature lists.
- Session lifecycle control. Can you create, inspect, and terminate sessions programmatically? Can you attach to a running session to debug it?
- Profile persistence. Are profiles named and durable, or does every session start clean? Can two agents share a profile safely, or do you need isolation guarantees?
- Proxy and browser settings. Are proxy configuration and browser-level settings configurable per session, and do they stay consistent across sessions in a profile? Ask the provider to document exactly which browser settings can be configured and how they persist.
- Concurrency model. How many sessions can you run at once, and how is that enforced? Check /pricing for current limits rather than assuming.
- Observability. Is there a live viewer? Are sessions recorded? Can you retrieve a recording after a failure?
- Protocol fidelity. Does the runtime expose real CDP, or a proprietary abstraction? Real CDP means your existing Playwright, Puppeteer, and Selenium tooling works unchanged.
That last point deserves emphasis. A runtime that speaks CDP lets you keep the debugging tools you already know — the DevTools protocol inspector, Playwright traces, Puppeteer's page.evaluate for ad-hoc inspection. A runtime with a custom control API forces you to rebuild that tooling. For computer use agents, where debugging is the main activity, protocol fidelity is a productivity multiplier.
Where computer use agents still struggle
Honest production teams plan around these failure modes rather than pretending they do not exist.
Long-horizon tasks drift. Each step introduces a small probability of error, and errors compound. A twenty-step task with 95% per-step reliability succeeds roughly a third of the time. Mitigations include checkpointing intermediate state, breaking tasks into verifiable sub-goals, and having the agent re-perceive after every action rather than batching.
Dynamic pages break coordinate assumptions. Agents that click by pixel coordinates fail when layout shifts. Prefer semantic targeting — accessibility tree nodes, stable selectors — and reserve coordinate clicks for canvas-like interfaces where nothing else works.
Authentication and anti-bot friction. Logged-in workflows require persistent profiles and, often, proxy configuration. These are runtime concerns, not model concerns. A runtime that handles profile persistence and proxy configuration removes an entire class of failures from your agent's responsibility.
Cost per task is hard to predict. Screenshot-heavy agents burn tokens fast. Structured perception reduces cost but loses flexibility. Measure cost per successful task, not cost per step, and tune the perception channel accordingly.
Getting started
The shortest path to a working computer use agent is to separate the two concerns cleanly: let the model own perception and reasoning, and let a hosted runtime own the browser. That division means you can swap models without touching infrastructure, and scale infrastructure without retraining prompts.
Start by creating a session, connecting over CDP, and running a single-step agent against a page you control. Then add profile persistence and a live viewer so you can watch what the agent does. The documentation walks through session creation, profile management, and CDP connection in detail, and /pricing has current usage details. If you want the broader context on why hosted runtimes exist at all, remote browsers and remote control cover the architectural case.
Computer use agents are a genuine capability shift, but the model is only half the system. The runtime is the other half, and it is the half you can actually engineer for reliability today.