AI Agent for Website: Runtime Choices That Hold Up

An AI agent for website work is only as reliable as the browser it drives. The model decides what to do; the runtime decides whether the page actually loads, whether the session survives a redirect, and whether you can see what went wrong at 2 a.m. Most "agent fails on the web" stories are runtime stories. This guide covers what an AI agent for website automation actually needs from a browser, where local Chrome breaks down, and how to wire a hosted Chromium session over CDP with Playwright. If you want the conceptual background first, read Remote browsers for AI agents; this post is the implementation layer.

What an AI agent for website automation actually does

Strip away the marketing and an agent loop is simple. The model receives an observation — usually a screenshot, an accessibility tree, or extracted DOM text — and emits an action: click, type, scroll, navigate, wait. The runtime executes that action against a live page and returns the next observation. Repeat until the task is done or the budget runs out.

That loop puts unusual demands on the browser:

• Statefulness. A login flow spans multiple pages and minutes. The browser must keep cookies, localStorage, and session tokens intact across steps.
• Observability. When the agent picks the wrong button, you need a recording or a live view, not a stack trace.
• Isolation. Two agents running concurrently must not share cookies or tabs.
• Determinism where possible. Stable viewport sizes, consistent user agents, and predictable network behavior reduce flaky decisions.
• Recoverability. Sessions crash. The runtime should let you reconnect to the same profile rather than starting from zero.

Traditional web automation tools — Selenium, Puppeteer, Playwright — were built for scripted tests where the author knows the page. An agent does not know the page. It discovers it. That difference is why the runtime layer matters more for agents than for test suites.

Why local Chrome stops working for agents

Running Chrome on the same machine as your agent works fine for a demo. It falls apart in production for predictable reasons.

Resource contention. Each Chromium instance consumes a significant amount of RAM. Ten concurrent agents on one box will thrash. Twenty will fail.

Session leakage. Local Chrome profiles persist by default. Agent A's logged-in bank session becomes Agent B's problem unless you carefully isolate user-data directories — and even then, crashes leave lock files behind.

No shared visibility. If your agents run on a worker fleet, nobody can watch a session live. Debugging means reading logs after the fact.

Environment drift. Chrome auto-updates. Your pinned Playwright version expects a specific Chromium build. The mismatch surfaces as cryptic protocol errors mid-task.

Network identity. Datacenter IPs and default browser settings get flagged by bot detection on many sites. You need configurable proxy and browser settings, which is awkward to manage per-machine.

A hosted runtime solves these by moving Chromium off your workers. Your agent process becomes a thin client that opens a session over the network, drives it, and closes it. The heavy lifting — process management, isolation, profile storage, proxying — belongs to the provider.

Hosted Chromium vs. local Chrome for agents

The trade-off is real: hosted sessions add network latency per action and a per-hour cost. For a single agent doing occasional lookups, local Chrome is fine. For anything running continuously, in parallel, or against sites that require stable identity, the hosted model wins on total cost once you count engineering time.

The runtime contract: what to require from a browser API

Before picking a provider, write down what your agent needs. A useful checklist:

1. CDP access. The runtime must expose a Chrome DevTools Protocol endpoint so Playwright, Puppeteer, or Selenium can attach. Anything proprietary locks you in.
2. Session lifecycle API. Create, connect, inspect, and terminate sessions programmatically. You need to clean up orphans.
3. Persistent profiles. Named profiles that survive session end, so agents can stay logged in without re-authenticating every run.
4. Live viewer. A URL you can open to watch the session in real time — essential for debugging agent decisions.
5. Configurable browser settings. Viewport, user agent, locale, timezone, and proxy settings should be parameters, not hardcoded defaults.
6. Usage controls. Per-session timeouts and concurrency limits so a runaway agent cannot burn your budget.
7. Recordings or logs. Some way to replay what happened after the fact.

Remote Browser exposes each of these: hosted Chromium sessions, a CDP endpoint, Playwright/Puppeteer/Selenium compatibility, a live viewer, persistent profiles, configurable proxy and browser settings, session isolation, and usage controls. Current limits and pricing live on the pricing page — check there rather than assuming numbers.

Wiring an agent to a hosted session with Playwright and CDP

The mechanics are the same as connecting Playwright to any remote Chrome. You request a session, get a WebSocket debugger URL, and call connectOverCDP. The Playwright CDP documentation covers the API surface; the code below shows the production shape.

import { chromium, Browser, Page } from "playwright";

interface SessionInfo {
  id: string;
  cdpUrl: string; // wss://... from the session API
}

async function openAgentSession(profileId: string): Promise<{ browser: Browser; page: Page; sessionId: string }> {
  // 1. Ask the runtime for a session bound to a persistent profile.
  const res = await fetch("https://api.remote-browser.dev/v1/sessions", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.REMOTE_BROWSER_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      profileId,
      viewport: { width: 1280, height: 800 },
      timeoutSeconds: 900,
    }),
  });
  if (!res.ok) throw new Error(`session create failed: ${res.status}`);
  const session: SessionInfo = await res.json();

  // 2. Attach over CDP. No local Chromium download required.
  const browser = await chromium.connectOverCDP(session.cdpUrl);

  // 3. Reuse the existing context so cookies from the profile are present.
  const context = browser.contexts()[0] ?? await browser.newContext();
  const page = context.pages()[0] ?? await context.newPage();

  return { browser, page, sessionId: session.id };
}

async function runAgentStep(page: Page, action: { type: string; selector?: string; text?: string }) {
  switch (action.type) {
    case "click":
      await page.locator(action.selector!).click({ timeout: 10_000 });
      break;
    case "type":
      await page.locator(action.selector!).fill(action.text ?? "");
      break;
    case "navigate":
      await page.goto(action.text!, { waitUntil: "domcontentloaded" });
      break;
    default:
      throw new Error(`unknown action: ${action.type}`);
  }
  // Return a compact observation for the model.
  return {
    url: page.url(),
    title: await page.title(),
    text: (await page.locator("body").innerText()).slice(0, 4000),
  };
}

Three details matter here. First, connectOverCDP returns a Browser whose contexts already exist — you must reuse browser.contexts()[0] rather than creating a fresh context, or you lose the profile's cookies. Second, every action needs a timeout; agents will happily wait forever on a selector that never appears. Third, the observation you return to the model should be bounded. Sending a full page's innerText blows up token costs and dilutes the signal.

For a broader walkthrough of the connection model, see Remote web browser: the practical runtime.

Profiles, proxies, and identity

Persistent profiles are the single biggest reliability lever for agents that log in. Instead of re-authenticating on every run — which triggers MFA, rate limits, and CAPTCHAs — the agent reuses a profile that already holds the session cookie. The runtime stores the profile server-side; your agent just references it by ID.

Two cautions:

• Treat profiles as secrets. A profile with a logged-in session is equivalent to a credential. Scope access per agent and rotate when an agent is decommissioned.
• Do not share profiles across unrelated tasks. Cookie state from one workflow can confuse another. One profile per logical identity is the safer default.

Proxy configuration is similarly a per-session parameter. If a site blocks datacenter IPs, route through a residential proxy; if it does not, skip the cost. Remote Browser lets you set proxy and browser settings per session rather than baking them into the image. We deliberately avoid claiming specific fingerprinting behavior — what matters is that the settings are configurable and consistent across runs, which is what most detection systems actually key on.

Debugging agent failures in production

When an agent fails, the question is always "what did it see?" A live viewer answers that in seconds. Open the session URL, watch the page, and compare it to the action log. Common failure patterns:

• Selector drift. The site changed a class name. The agent's click landed on nothing. Fix: prefer role-based or text-based locators over CSS paths.
• Timing. The agent acted before a client-side render finished. Fix: wait on a specific element or network idle, not a fixed sleep.
• Modal interception. A cookie banner or newsletter popup blocked the target. Fix: add a pre-step that dismisses known overlays.
• Auth expiry. The profile's session expired mid-run. Fix: detect the login page and re-authenticate, or refresh the profile on a schedule.

Recordings and session logs turn these from guesswork into a checklist. If you need to drive a session interactively during an incident, remote control browser covers the manual-takeover pattern.

Cost and concurrency planning

Hosted browsers are metered per browser-hour. That means your cost is roughly concurrent_sessions × average_session_minutes / 60 × rate. Two levers dominate:

1. Session duration. Close sessions as soon as the task completes. An idle session still bills.
2. Concurrency. Cap it. A queue with a fixed worker pool is cheaper and more predictable than unbounded fan-out.

Set per-session timeouts at the API level so a stuck agent cannot run for hours. Monitor p95 session duration; if it creeps up, you have a reliability problem that will show up on the invoice before it shows up in your error rates. For current rates and plan details, see /pricing.

When a hosted runtime is the wrong choice

Be honest about the boundary. Hosted Chromium is overkill if:

• You run fewer than a handful of sessions per day and latency per action matters more than isolation.
• Your targets are internal tools on a private network the provider cannot reach.
• You need browser extensions that the runtime does not support.

In those cases, local Playwright with careful profile management is the pragmatic answer. The hosted model pays off when you have concurrency, persistent logins, or a need to debug sessions you cannot see.

Getting started

The shortest path from zero to a working agent session:

1. Read the documentation for the session API and CDP endpoint format.
2. Create a profile, run one session, and confirm you can attach with connectOverCDP.
3. Add a live viewer check to your debugging workflow before you scale concurrency.
4. Set timeouts and concurrency caps, then load-test with your real task mix.

An AI agent for website automation is a systems problem disguised as a model problem. Get the runtime right — isolated sessions, persistent profiles, observable execution, and a standard protocol — and the agent loop becomes something you can actually operate.