BLOG
Computer Use Agent: What It Is and How to Run One
A computer use agent drives a real browser to complete tasks. Learn the architecture, runtime requirements, and how to run one in production.

A computer use agent is an AI system that completes tasks by operating a real computer interface — clicking, typing, scrolling, and reading what's on screen — rather than calling a fixed API. In practice, most computer use agents today operate a browser: they navigate pages, fill forms, extract data, and verify results the way a person would. This guide explains how a computer use agent actually works, what separates a demo from a production deployment, and how to wire one to a hosted Chromium runtime you can trust.
If you're evaluating runtimes rather than concepts, the remote browser for AI agents overview is the faster starting point. This article covers the full picture: architecture, the control loop, runtime criteria, and a working Playwright/CDP connection.
What a computer use agent actually is
The term gets used loosely, so it's worth being precise. A computer use agent has three parts:
- A perception layer that turns the screen into something the model can reason about — a screenshot, an accessibility tree, a DOM snapshot, or a combination.
- A reasoning layer (usually an LLM) that decides the next action given the current state and the goal.
- An action layer that executes that decision against a real machine — a click at coordinates, a keypress, a navigation, or a scripted DOM interaction.
What distinguishes a computer use agent from a traditional automation script is that the action sequence is not fixed in advance. The agent observes, decides, acts, and re-observes in a loop. That loop is what makes it flexible across sites that change, and it's also what makes it harder to run reliably at scale.
A computer use agent is not the same as a browser agent framework, though the two overlap heavily. Frameworks like browser-use or Stagehand provide the reasoning and tooling layer; the computer use agent still needs a machine to act on. That machine — a real Chromium instance with a stable network identity, a persistent profile, and a debuggable session — is the runtime problem this article addresses.
How the control loop works
Most production computer use agents follow a loop that looks like this:
- Observe. Capture the current page state. Screenshot-based agents send an image to a vision model. DOM-based agents serialize the accessibility tree or a trimmed HTML snapshot. Hybrid agents use both.
- Decide. The model receives the observation plus the task and returns an action — often a structured call like
click(element_id),type(text), ornavigate(url). - Act. The runtime executes the action against the live browser over the Chrome DevTools Protocol (CDP).
- Verify. The agent checks whether the action produced the expected state change before continuing. Skipping this step is the most common cause of silent failures.
- Repeat or terminate. The loop ends when the goal is met, an error threshold is hit, or a step budget is exhausted.
The critical detail is that every step depends on the browser being in a known, inspectable state. If the runtime drops the session, resets the profile, or changes the IP mid-task, the agent's mental model of the page diverges from reality and the loop breaks. That's why runtime choice matters more than most teams expect.
Why the runtime is the hard part
A computer use agent that works on your laptop is a prototype. Getting it to work across many concurrent tasks, on sites that load slowly or block suspicious traffic, is an infrastructure problem. The runtime has to provide:
- Session isolation. Each agent task needs its own browser context so cookies, storage, and logins don't leak between tasks.
- Persistent profiles. Many tasks require staying logged in across steps or across sessions. A profile that survives the session is often the difference between a task that completes and one that hits a login wall.
- Network identity control. Sites increasingly gate behavior on IP reputation and geography. Configurable proxy settings let you route sessions through the right region or residential pool.
- Live observability. When an agent gets stuck, you need to watch the session in real time and inspect the DOM, network, and console — not read a log after the fact.
- CDP access. The agent's action layer speaks CDP. Any runtime that doesn't expose a CDP endpoint forces you into a proprietary API and limits which frameworks you can use.
Local Chrome fails on most of these at scale. You can run headless Chromium in a container, but you're then responsible for profile persistence, proxy configuration, session cleanup, crash recovery, and concurrency limits — work that has nothing to do with the agent itself. A hosted runtime moves that burden off your team. The remote browser online guide covers the operational trade-offs in more depth.
Runtime options compared
| Approach | Session isolation | Persistent profiles | CDP access | Live debugging | Ops burden |
|---|---|---|---|---|---|
| Local Chrome on a dev machine | Manual | Manual | Yes (local) | Yes | Low at 1, high at N |
| Self-hosted headless Chromium in containers | You build it | You build it | Yes | You build it | High |
| Hosted Chromium runtime (Remote Browser) | Built in | Built in | Yes | Built in viewer | Low |
| Vendor-specific agent API | Built in | Varies | Often no | Varies | Low but locked in |
The trade-off is straightforward. Self-hosting gives you maximum control and no per-session cost, but you own every failure mode. A hosted runtime trades some control for operational reliability and CDP compatibility. A vendor-specific API is easiest to start with but tends to hide the browser entirely, which makes debugging a stuck agent much harder.
For a computer use agent specifically, CDP access is the deciding factor. It's what lets you use Playwright, Puppeteer, Selenium, or a framework like browser-use against the same runtime, and it's what lets you drop down to raw protocol calls when the high-level API isn't enough.
Connecting a computer use agent over CDP
The connection pattern is the same whether you're running a reasoning loop or a scripted test. You request a session from the runtime, get back a CDP WebSocket endpoint, and connect Playwright to it. Here's a TypeScript example:
import { chromium, Browser, Page } from "playwright";
interface SessionResponse {
cdpUrl: string;
sessionId: string;
}
async function openAgentSession(): Promise<{ browser: Browser; page: Page }> {
// Request a hosted Chromium session from the runtime API.
const res = await fetch("https://api.remote-browser.dev/v1/sessions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${process.env.REMOTE_BROWSER_API_KEY}`,
},
body: JSON.stringify({
// Optional: pin a persistent profile so logins survive the session.
profileId: process.env.AGENT_PROFILE_ID,
// Optional: route through a configured proxy for region or reputation.
proxyId: process.env.AGENT_PROXY_ID,
}),
});
if (!res.ok) {
throw new Error(`Session create failed: ${res.status} ${await res.text()}`);
}
const session: SessionResponse = await res.json();
// Connect over CDP. This is the same call you'd use against local Chrome.
const browser = await chromium.connectOverCDP(session.cdpUrl);
const context = browser.contexts()[0] ?? (await browser.newContext());
const page = context.pages()[0] ?? (await context.newPage());
return { browser, page };
}
async function runAgentStep(page: Page, goal: string) {
// Observe: capture state for the reasoning layer.
const snapshot = await page.accessibility.snapshot();
const screenshot = await page.screenshot({ type: "png" });
// Decide: your model call goes here, returning a structured action.
const action = await decideNextAction({ goal, snapshot, screenshot });
// Act: execute against the live browser.
switch (action.type) {
case "click":
await page.click(action.selector);
break;
case "type":
await page.fill(action.selector, action.text);
break;
case "navigate":
await page.goto(action.url, { waitUntil: "domcontentloaded" });
break;
}
// Verify: confirm the state changed before the next loop iteration.
await page.waitForLoadState("networkidle").catch(() => {});
}
async function main() {
const { browser, page } = await openAgentSession();
try {
await page.goto("https://example.com");
await runAgentStep(page, "Find the contact form and fill in the name field.");
} finally {
await browser.close();
}
}
main().catch((err) => {
console.error(err);
process.exit(1);
});Two things to note. First, connectOverCDP is the standard Playwright entry point for remote browsers — the Playwright CDP documentation is the authoritative reference for its behavior and limitations. Second, the finally block matters: sessions that aren't closed cleanly can leave profiles locked and consume runtime capacity.
Production criteria for a computer use agent
Once the loop works, the questions shift to reliability. These are the criteria worth checking before you scale:
Session lifecycle. Can you create, inspect, and terminate sessions programmatically? Does the runtime clean up orphaned sessions automatically? A leaked session is a silent cost and a potential data exposure.
Profile management. Can you create named profiles, reuse them across sessions, and reset them when needed? For agents that log in, profile persistence is non-negotiable. For agents that must stay anonymous, profile isolation is equally important.
Proxy and network settings. Can you configure per-session proxy routing? Is the configuration exposed through the API rather than a dashboard-only setting? Agents that hit geo-restricted content need this at the session level.
Observability. Can you watch a live session, read console output, and inspect network requests while the agent runs? The remote control browser guide covers what live debugging looks like in practice.
Concurrency and quotas. How many sessions can you run at once, and how is usage metered? Check pricing for current limits — don't assume a number from a blog post, including this one.
Framework compatibility. Does the runtime expose a standard CDP endpoint, or a proprietary protocol? Standard CDP means you can swap frameworks without rewriting your agent.
Failure recovery. What happens when a session crashes mid-task? Can you reconnect to the same profile and resume, or do you restart from scratch? The answer determines how much of your agent logic needs to be idempotent.
Common failure modes and how to handle them
Even a well-built computer use agent hits predictable problems. A few worth designing for:
- Stale element references. The agent clicks a selector that no longer exists because the page re-rendered. Mitigation: re-observe after every action and use stable selectors or accessibility roles where possible.
- Login walls and CAPTCHAs. The agent hits an authentication gate it can't pass. Mitigation: persistent profiles with pre-authenticated sessions, and a human-in-the-loop escalation path for CAPTCHAs.
- Network identity blocks. The site serves a challenge page or blocks the request entirely. Mitigation: configurable proxy routing and careful attention to request patterns.
- Runaway loops. The agent repeats the same failing action indefinitely. Mitigation: step budgets, repeated-action detection, and a hard timeout per task.
- Session drift. The browser state diverges from the agent's model of it. Mitigation: verification steps after each action, and a runtime that doesn't silently reset state.
None of these are solved by a better model alone. They're runtime and orchestration problems, which is why the infrastructure layer deserves as much attention as the reasoning layer.
Where hosted Chromium fits
A hosted Chromium runtime like Remote Browser gives a computer use agent a stable, isolated, CDP-accessible browser without your team managing the underlying infrastructure. Sessions are created via API, connected over CDP, watched live, and terminated cleanly. Persistent profiles, configurable proxy settings, and session isolation are built into the runtime rather than bolted on.
The practical benefit is that your agent code stays focused on the loop — observe, decide, act, verify — while the runtime handles the browser lifecycle. You can develop against local Chrome and switch to a hosted endpoint by changing one connection string, because both speak CDP.
If you're ready to wire it up, the documentation covers session creation, profile management, and CDP connection details. Start with a single session, verify the loop works end to end, then scale concurrency once you've measured where your agent actually fails. The remote web browser guide is a good next read for teams planning multi-session deployments.
Summary
A computer use agent is an AI system that operates a real browser through an observe-decide-act-verify loop. The reasoning layer gets most of the attention, but the runtime determines whether the agent works in production. Look for session isolation, persistent profiles, configurable network settings, live observability, and standard CDP access. Connect over CDP with Playwright or Puppeteer, design for the common failure modes, and treat the browser runtime as infrastructure rather than an afterthought.