Website Automation with Hosted Chromium for AI Agents

Website automation is the practice of driving a real browser — loading pages, clicking, typing, reading the DOM — programmatically instead of by hand. For AI agents, the hard part is rarely the click. It is the runtime underneath: a browser that stays alive, keeps its state, isolates sessions, and exposes a stable control surface. This guide covers how website automation actually works in production, where local Chrome setups break down, and how to evaluate a hosted runtime for agents built on Playwright, Puppeteer, Selenium, or a browser-use style loop.

What website automation actually requires

Strip away the tooling and every website automation stack has four layers:

1. A browser process. Real Chromium, not a headless HTTP client. Modern sites render with JavaScript, gate content behind login flows, and behave differently under bot detection.
2. A control protocol. The Chrome DevTools Protocol (CDP) is the wire format that Playwright, Puppeteer, and most agent frameworks speak. It exposes navigation, input, network, and DOM inspection.
3. State. Cookies, localStorage, and logged-in sessions. Without persistence, every run starts from zero and re-authenticates.
4. Isolation and lifecycle. Each task needs its own browser context so one agent's cookies never leak into another's, and sessions need to start, run, and stop predictably.

Locally, all four layers live on your machine. That works for a script you run once. It stops working when you have ten agents running concurrently, a CI pipeline that needs a browser, or a task that must survive a laptop lid closing.

Where local Chrome breaks down

The failure modes are consistent and well documented across teams running browser automation at scale:

• Resource contention. Each Chromium instance consumes a large amount of RAM. Running twenty in parallel on one box degrades everything, including the host application.
• Version drift. Playwright and Puppeteer pin browser builds. When the local Chrome auto-updates, selectors and behaviors shift underneath you.
• No persistence across workers. A cloud function or container that handles one task cannot hold a logged-in profile between invocations unless you build storage for it.
• Debugging blind spots. When an agent fails at step 14 of a 20-step flow, you need to see what the page looked like. Local headless runs give you a stack trace and nothing else.
• Network identity. Datacenter IPs and default browser settings get flagged. Teams end up building proxy management and browser configuration from scratch.

None of these are agent problems. They are infrastructure problems, and they are the reason hosted browser runtimes exist.

What a hosted runtime provides

A hosted runtime moves the browser off your machine and exposes it over a network endpoint. Remote Browser, for example, gives you hosted Chromium sessions reachable over CDP, compatible with Playwright, Puppeteer, and Selenium clients. The pieces that matter in production:

• Session isolation. Each session gets its own browser context. Cookies and storage do not bleed between tasks.
• Persistent profiles. You can save a logged-in state and reuse it, so agents skip the login flow on subsequent runs. This is the difference between a demo and a workflow that runs hourly.
• Live viewer. A real-time view of the session while it runs. When an agent stalls, you watch the page instead of guessing.
• Configurable browser settings. Proxy configuration, user agent, viewport, and related options are set per session rather than hardcoded into a launch script.
• Usage controls. Sessions are metered, so you can reason about cost per task. See /pricing for current rates and limits.

The control surface is the same CDP you already use locally. That is the point: the migration is a connection-string change, not a rewrite.

Connecting Playwright to a hosted browser over CDP

Playwright's chromium.connectOverCDP() attaches to a running browser instead of launching one. The Playwright CDP documentation covers the API; the production detail is what you do after connecting.

import { chromium, Browser, BrowserContext, Page } from "playwright";

interface SessionHandle {
  cdpUrl: string;
  sessionId: string;
}

async function runAutomation(session: SessionHandle): Promise<void> {
  // Attach to the hosted Chromium instance over CDP.
  const browser: Browser = await chromium.connectOverCDP(session.cdpUrl);

  // Reuse the existing context so a persistent profile's cookies apply.
  const context: BrowserContext = browser.contexts()[0];
  const page: Page = context.pages()[0] ?? (await context.newPage());

  try {
    await page.goto("https://example.com/account", {
      waitUntil: "domcontentloaded",
      timeout: 30_000,
    });

    // Wait on a real signal, not a fixed sleep.
    await page.waitForSelector("[data-testid=account-balance]", {
      state: "visible",
      timeout: 15_000,
    });

    const balance = await page.textContent("[data-testid=account-balance]");
    console.log({ sessionId: session.sessionId, balance });
  } finally {
    // Disconnect the client; the hosted session lifecycle is managed separately.
    await browser.close();
  }
}

Three details separate this from a tutorial snippet:

• `browser.contexts()[0]` assumes the session was created with a persistent profile. If you create a fresh session per task, call context.newPage() on a new context instead.
• `browser.close()` disconnects your client. Whether the underlying session stops depends on the runtime's lifecycle API — check your provider's session-stop semantics rather than assuming.
• Timeouts and wait conditions are where agents fail. Prefer waitForSelector and waitForResponse over waitForTimeout. A fixed sleep is a race condition with extra steps.

If you are wiring an agent loop rather than a deterministic script, the same connection pattern applies — the agent decides what to click, but the CDP session, profile, and isolation come from the runtime. The /blog/remote-browser-for-ai-agents post covers that architecture in more depth.

Comparing runtime options

The table below compares the common ways teams run website automation. It is a decision aid, not a ranking.

The self-hosted grid row is where most "we'll just run it in Docker" plans land. It works, but you inherit browser version management, proxy configuration, session cleanup, and the on-call rotation for all of it. For a team whose product is an AI agent, that is undifferentiated work.

Production criteria for choosing a runtime

When you evaluate a hosted browser for website automation, test these specific things rather than reading feature lists:

CDP fidelity. Can you connect with connectOverCDP and use the full Playwright API, including network interception and page.evaluate? Some runtimes expose a reduced command set. Ask for a compatibility matrix.

Profile lifecycle. Can you create a profile, log in once, and reuse it across sessions? Can you delete it? Is profile storage scoped per account? Persistent profiles are the single biggest reliability lever for authenticated workflows.

Session isolation guarantees. Confirm that concurrent sessions do not share cookies or storage. Ask how contexts are separated — process-level isolation is stronger than context-level.

Live observability. A live viewer is table stakes. The better question is whether sessions are recorded and whether you can replay a failed run. See /blog/remote-control-browser for how live control and debugging interact.

Network configuration. Proxy support, geographic targeting, and configurable browser settings matter for sites that treat datacenter traffic differently. Be precise about what is configurable versus what is claimed.

Metering transparency. Understand what a "browser-hour" covers: wall-clock time from session start to stop, or active compute. Idle sessions that bill by wall clock add up fast. /pricing documents the current model.

Client compatibility. If your team uses Selenium rather than Playwright, confirm the runtime exposes a CDP endpoint Selenium can attach to. Most do, but the connection syntax differs.

Common failure modes and fixes

Even with a solid runtime, website automation fails in predictable ways. These are the ones worth designing around:

• Selectors that break on deploy. Prefer data-testid attributes and role-based locators (getByRole) over CSS paths. If you control the target site, add stable test hooks.
• Login flows that trigger MFA. Persistent profiles reduce re-authentication, but MFA challenges still need a human or a pre-authorized session. Plan for a manual bootstrap step.
• Rate limiting and bot detection. Configurable browser settings and proxy choice help, but no runtime guarantees access to every site. Design retries with backoff and treat blocks as a first-class state.
• Agent loops that never terminate. Cap steps per task and set a wall-clock timeout on the session. An agent that clicks forever is a billing incident.
• Stale page state. After navigation, wait for a specific element or network response. networkidle is unreliable on pages with polling or analytics beacons.

For teams moving from local scripts to hosted sessions, /blog/remote-browser-online walks through the migration checklist, and /documentation has the API reference for session creation and CDP endpoints.

When hosted Chromium is the right call

Hosted website automation pays for itself when any of these are true:

• You run more than a handful of concurrent browser tasks.
• Your workflows need authenticated sessions that persist across runs.
• You need to debug agent behavior visually, not from logs.
• Your team does not want to own browser version management and proxy infrastructure.
• You are running browser-use or similar agent frameworks that expect a CDP endpoint.

It is overkill for a single script that runs nightly on one machine. The honest framing: a hosted runtime is infrastructure, and you adopt it when browser management stops being incidental to your product.

Getting started

The shortest path from local to hosted is to keep your Playwright code and change the connection. Create a session, take the CDP URL, pass it to connectOverCDP, and run your existing script unchanged. Then add a persistent profile and a live viewer, and you have the two features that most improve reliability for agent workloads.

Start with the /documentation for session creation and connection details, review /pricing to model cost per task, and read /blog/remote-web-browser if you are still deciding between a hosted runtime and a self-managed browser fleet. The goal is not to add a vendor — it is to stop spending engineering time on Chromium lifecycle management so your team can work on the automation itself.