BLOG
Computer Use AI: What It Is and the Runtime It Needs
Computer use AI lets models drive a real browser. Learn how it works, where it breaks in production, and how to wire it to hosted Chromium over CDP.

Computer use AI is the class of models and agents that operate a graphical interface the way a person does: they look at a screen, decide on an action, and click, type, or scroll. In practice, almost all production computer use AI today runs against a browser rather than a desktop, because the browser is where the work lives. The model is only half the system. The other half is the runtime that gives it a real Chromium instance, a stable connection, and a way to observe what happened. This guide explains how computer use AI actually works, where it fails once you leave a demo, and how to wire it to hosted Chromium over the Chrome DevTools Protocol (CDP).
What "computer use AI" actually means
The term covers two related but distinct things, and conflating them causes most of the confusion in this space.
Model-level computer use. A vision-language model is given a screenshot and a task, and it returns an action: click at coordinates (x, y), type a string, press a key, scroll. The model has no persistent memory of the machine and no direct access to the DOM. It reasons over pixels. Anthropic's computer use capability and similar offerings from other labs fit here.
Agent-level computer use. A framework wraps that model in a loop: capture state, ask the model for an action, execute it against a real machine, capture the new state, repeat until the task is done or a step budget is exhausted. The framework owns retries, error handling, and the connection to the browser.
Most teams building on computer use AI are building the second thing. The model is a dependency you swap; the agent loop and the runtime are the parts you own and have to operate. That distinction matters because the runtime is where reliability is won or lost, and it is the part most tutorials skip.
Why the browser is the default target
A general-purpose computer use model can drive any application with a window. In production, though, the overwhelming majority of tasks — filling forms, pulling data from a dashboard, checking out, reconciling records across SaaS tools — happen in a browser. That narrows the problem in useful ways:
- The browser exposes a control protocol. CDP gives you programmatic access to navigation, input, network, and the DOM. You are not limited to screenshots.
- Sessions are cheap to isolate. A browser context is a clean, disposable unit. You can spin one up per task and throw it away.
- The environment is reproducible. A pinned Chromium build behaves the same on every run, unlike a user's desktop with arbitrary installed software.
That last point is why computer use AI and browser automation have converged. If you can make the browser deterministic, the model's job gets easier and your failure rate drops.
The architecture of a computer use AI agent
A working system has five layers. Each one is a place where things break.
- The model. Takes observations (screenshots, accessibility trees, or both) and emits actions.
- The agent loop. Manages the observe-decide-act cycle, step limits, and recovery.
- The transport. Connects the loop to a browser — CDP over WebSocket is the standard.
- The browser. A real Chromium instance with a profile, network settings, and isolation.
- The infrastructure. Where that browser runs, how sessions are metered, and how you watch them.
Demos usually collapse layers 3 through 5 into "a local Chrome window." Production cannot. You need the browser to run somewhere durable, survive a worker restart, and be observable when something goes wrong at 2 a.m.
Where computer use AI breaks in production
The failure modes are predictable, and they cluster around the runtime rather than the model.
Coordinate drift. A model that clicks at pixel coordinates is sensitive to viewport size, device pixel ratio, zoom level, and page layout shifts. If the browser resizes between the screenshot and the click, the action lands somewhere else. Fixing viewport dimensions and disabling layout-shifting ads or cookie banners helps; so does giving the model the accessibility tree alongside the screenshot so it can target elements semantically.
Session loss. A local Chrome window dies when the laptop sleeps or the CI job ends. If your agent is mid-checkout, that is a lost task. Hosted sessions decouple the browser's lifetime from the worker's lifetime.
State contamination. Reusing one browser profile across tasks leaks cookies, local storage, and logged-in sessions between unrelated jobs. You want per-task isolation by default and persistent profiles only when a workflow genuinely needs a saved login.
Unobservable failures. When an agent fails, you need to see what it saw. Screenshots at each step, a session recording, and a live viewer turn a mystery into a debuggable trace.
Cost blowouts. A looping agent that retries a failing step burns browser-hours silently. Step budgets and per-session time limits are not optional.
None of these are model problems. They are runtime problems, which is why the choice of runtime deserves as much attention as the choice of model.
Local Chrome vs hosted Chromium for computer use AI
The first architectural decision is where the browser runs. Here is how the two options compare on the dimensions that matter in production.
| Dimension | Local Chrome | Hosted Chromium (Remote Browser) |
|---|---|---|
| Session lifetime | Tied to the worker or laptop | Independent; survives worker restarts |
| Isolation | Manual; easy to leak profiles | Per-session isolation by default |
| Scaling | Bounded by the host machine | Scale sessions independently of compute |
| Observability | Whatever you instrument yourself | Live viewer plus recordings |
| Network settings | Whatever the host has | Configurable proxy and browser settings |
| Setup cost | Install and pin Chromium per machine | Connect over CDP with a session URL |
| Best fit | Prototyping, single-user tools | Multi-tenant agents, scheduled jobs, CI |
Local Chrome is the right starting point. It is free, fast to iterate on, and you can watch every action. The transition point is when you need more than one concurrent agent, when tasks must survive the process that started them, or when you need a record of what happened. At that point the local model becomes an operational liability.
For a deeper treatment of the migration, see Remote browsers for AI agents: the missing runtime layer.
Wiring a computer use agent to hosted Chromium
The transport for computer use AI is CDP. Playwright, Puppeteer, and Selenium all speak it, and hosted Chromium exposes a WebSocket endpoint you connect to. The pattern is the same regardless of framework: create a session, get its CDP URL, connect, drive, disconnect.
The following TypeScript example uses Playwright's connectOverCDP. It assumes you have a session endpoint from your provider; check the documentation for the exact session-creation call.
import { chromium, type Browser, type Page } from "playwright";
interface SessionInfo {
cdpUrl: string;
sessionId: string;
}
// Your provider's session-creation endpoint returns a CDP WebSocket URL.
async function createSession(): Promise<SessionInfo> {
const res = await fetch(`${process.env.BROWSER_API_URL}/sessions`, {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.BROWSER_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
// Pin the viewport so screenshot coordinates stay stable.
viewport: { width: 1280, height: 800 },
// Isolate state per task unless the workflow needs a saved login.
persistentProfile: false,
}),
});
if (!res.ok) throw new Error(`session create failed: ${res.status}`);
return (await res.json()) as SessionInfo;
}
async function runTask(task: string): Promise<void> {
const session = await createSession();
let browser: Browser | undefined;
try {
browser = await chromium.connectOverCDP(session.cdpUrl);
const context = browser.contexts()[0];
const page: Page = context.pages()[0] ?? (await context.newPage());
await page.goto("https://example.com/login", { waitUntil: "domcontentloaded" });
// The agent loop: observe, decide, act.
// Feed the model a screenshot plus the accessibility tree so it can
// target elements semantically instead of guessing pixel coordinates.
const shot = await page.screenshot({ type: "png" });
const tree = await page.accessibility.snapshot();
// ... send { shot, tree, task } to your model and receive an action ...
// ... execute the action against `page` (click, fill, press) ...
// ... repeat until the task completes or the step budget is exhausted ...
console.log("task finished", { task, bytes: shot.byteLength, hasTree: !!tree });
} finally {
// Always release the session; leaked sessions bill by the hour.
await browser?.close().catch(() => {});
}
}
runTask("Log in and download the latest invoice").catch((err) => {
console.error(err);
process.exitCode = 1;
});A few details in that snippet are load-bearing:
- Pin the viewport. Coordinate-based actions depend on it. Set it at session creation, not after the first screenshot.
- Send both pixels and structure. The accessibility tree gives the model element roles and names; the screenshot gives it layout and visual state. Together they cut misclicks substantially.
- Close the session in a `finally` block. A crashed worker that never releases a session is a silent cost leak.
- Do not reuse the CDP connection across tasks. Create a session per task, or at least per tenant, so state does not bleed.
If you are connecting from a different framework, the mechanics are identical — see the Chrome DevTools Protocol documentation for the underlying domains and events.
Production criteria for a computer use AI runtime
When you evaluate a runtime for computer use AI, score it against these criteria rather than against a feature list.
Connection stability. Does the CDP WebSocket survive network blips? Does the provider reconnect or fail fast? Long-running agents will hit transient disconnects; the runtime should handle them without losing the session.
Isolation guarantees. Are sessions isolated at the browser-context level, the process level, or the VM level? The stronger the isolation, the safer it is to run untrusted or multi-tenant workloads.
Profile control. Can you choose between ephemeral and persistent profiles per session? Persistent profiles are essential for workflows behind a login, but they should be opt-in.
Network configuration. Proxy support and configurable browser settings matter for agents that need to appear from a specific region or that hit sites with strict bot policies. Be precise about what you need; "stealth" is a marketing word, and the actual controls are proxies, headers, and browser flags.
Observability. A live viewer for debugging in real time, plus recordings for post-mortems. If you cannot see the session, you cannot fix it.
Usage controls. Per-session time limits, concurrency caps, and clear metering. See pricing for how Remote Browser meters browser-hours.
Framework compatibility. Playwright, Puppeteer, and Selenium should all connect without a shim. If a provider only supports one, you are locked in more than you think.
Practical patterns that raise task success
Beyond the runtime, a few agent-design choices move the needle for computer use AI.
Constrain the action space. Give the model a small set of verbs — click, type, press, scroll, wait, done — rather than free-form code. Smaller action spaces are easier to validate and easier to retry.
Validate before acting. Before executing a click, check that the target element still exists and is visible. A cheap DOM check catches most coordinate-drift failures before they corrupt state.
Use deterministic steps where you can. If a login is a fixed sequence, script it with Playwright selectors and hand control back to the model only for the parts that genuinely need judgment. Hybrid agents are more reliable and cheaper than pure-vision agents.
Budget steps and time. Cap the loop at a step count and the session at a wall-clock limit. When the budget is hit, fail loudly with a trace rather than looping.
Record everything. Screenshots per step, console logs, and network failures. The cost of storage is trivial next to the cost of an unreproducible bug.
For teams running many concurrent agents, these patterns compound: a runtime that isolates sessions plus an agent that validates actions gives you a system you can actually operate. If you want to see a hosted session in action before committing, run a remote browser online and watch the viewer.
When computer use AI is the wrong tool
It is worth saying plainly: pixel-level computer use is the most expensive and least reliable way to automate a website. If a site has an API, use it. If it has stable selectors, use Playwright directly. Computer use AI earns its cost when the interface is the only contract available — legacy internal tools, sites that change layout often, workflows that span multiple applications, or tasks where the set of steps is not known in advance.
A useful heuristic: if you can write the automation as a script in an afternoon, do that. If the task requires judgment about what is on screen, computer use AI is the right layer, and the runtime is what makes it dependable.
Getting started
The shortest path from a local prototype to a production computer use AI system is to keep your agent loop and swap the browser underneath it. Connect to a hosted Chromium session over CDP, pin the viewport, isolate state per task, and instrument every step. Start with the documentation to create your first session, review pricing to understand how browser-hours are metered, and read remote web browser: the practical runtime for browser automation for the broader architecture. When you need to drive a session from code and agents at the same time, remote control browser covers the control model.
The model will keep improving. The runtime you build around it is what determines whether your agents finish tasks or just start them.