Web Automation with Hosted Browsers: A Production Guide

Web automation is the practice of driving a real browser programmatically to complete tasks a human would otherwise do by hand: filling forms, extracting data, running checkout flows, or verifying that a deployment did not break a critical page. The hard part is rarely the script. It is the runtime underneath it — the browser process, the network identity, the session state, and the ability to debug a failure after it happens. This guide covers how to move web automation from a laptop to a hosted Chromium runtime, with concrete details on CDP connections, persistent profiles, and the production criteria that separate a demo from a system you can leave running.

If you want the conceptual background first, the remote browser for AI agents post covers why the runtime layer exists. This article is about the mechanics of web automation.

What web automation actually requires in production

A local Playwright or Puppeteer script works because your machine supplies everything implicitly: a Chrome binary, a display server, a residential IP, a logged-in profile, and a human sitting nearby to watch it fail. Production removes all of that. When you run web automation on a server or inside a container, you have to supply each piece deliberately.

The requirements break down into five layers:

• Browser binary and version. You need a pinned Chromium build that matches your automation library's expectations. Version drift between the browser and the driver is a common source of flaky selectors and missing APIs.
• Execution environment. Headless Chromium needs the right shared libraries, fonts, and sandbox configuration. Container images that "work on my machine" often fail on a minimal base image.
• Network identity. Many sites behave differently for datacenter IPs. Whether you need residential egress depends on the target, not on the framework.
• Session state. Logins, cookies, and local storage have to persist across runs if your workflow depends on an authenticated session.
• Observability. When a run fails at step 14 of 30, you need a screenshot, a DOM snapshot, or a live view — not a stack trace that says "timeout."

A hosted runtime packages the first four and gives you the fifth. That is the entire value proposition, and it is worth evaluating any provider against exactly those five items.

How hosted Chromium changes the web automation model

With a hosted runtime, your code no longer launches a browser. It connects to one that already exists. The connection is the Chrome DevTools Protocol (CDP), the same protocol Chrome uses internally to expose debugging and control surfaces. Playwright, Puppeteer, and Selenium can all speak it, which means your existing web automation code usually needs only a connection change, not a rewrite.

The lifecycle looks like this:

1. Your code requests a session from the runtime's API.
2. The runtime provisions a Chromium instance, optionally with a persistent profile and proxy settings.
3. The API returns a CDP WebSocket endpoint.
4. Your Playwright or Puppeteer client connects over that endpoint.
5. You drive the page as normal.
6. You close the session, and the runtime tears down the browser.

The important shift is that the browser is now a managed resource with a lifecycle, not a child process. That has consequences for how you write web automation code: you handle connection failures, you respect session timeouts, and you treat the browser as something that can be observed and replayed rather than something you own.

Connecting with Playwright over CDP

The connection itself is a few lines. The example below uses TypeScript and Playwright's connectOverCDP, which is the standard path for attaching to an existing Chromium instance. The endpoint comes from the runtime's session API; the shape is a ws:// or wss:// URL.

import { chromium, type Browser, type BrowserContext } from "playwright";

interface SessionInfo {
  cdpUrl: string;
  sessionId: string;
}

async function openSession(): Promise<SessionInfo> {
  // Replace with your runtime's session endpoint.
  const res = await fetch("https://api.remote-browser.dev/v1/sessions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.RB_API_KEY}`,
    },
    body: JSON.stringify({
      profileId: "checkout-smoke-test",
      // Configurable browser settings; exact options depend on your plan.
      proxy: { country: "us" },
    }),
  });

  if (!res.ok) {
    throw new Error(`Session create failed: ${res.status} ${await res.text()}`);
  }
  return (await res.json()) as SessionInfo;
}

async function runCheckoutFlow(): Promise<void> {
  const session = await openSession();
  let browser: Browser | undefined;

  try {
    browser = await chromium.connectOverCDP(session.cdpUrl, {
      timeout: 30_000,
    });

    // Reuse the default context so the persistent profile is in effect.
    const context: BrowserContext = browser.contexts()[0];
    const page = context.pages()[0] ?? (await context.newPage());

    await page.goto("https://example.com/cart", {
      waitUntil: "domcontentloaded",
    });
    await page.getByRole("button", { name: "Checkout" }).click();
    await page.getByLabel("Email").fill("qa@example.com");
    await page.getByRole("button", { name: "Place order" }).click();

    await page.waitForURL(/\/orders\/\d+/, { timeout: 20_000 });
    console.log("Order confirmed:", page.url());
  } finally {
    // Always close the browser connection; the runtime reclaims the session.
    await browser?.close();
  }
}

runCheckoutFlow().catch((err) => {
  console.error(err);
  process.exit(1);
});

Two details matter here. First, connectOverCDP returns a Browser whose contexts are already created by the remote instance — you generally should not call browser.newContext() unless you specifically want an isolated context inside the session. Second, closing the browser connection is how you signal the runtime that the session is done. If you skip it, you pay for idle browser time until the session times out.

The Playwright documentation on connecting over CDP is the authoritative reference for the connection options, including timeouts and headers.

Persistent profiles and the profile list problem

The single most common production failure in web automation is losing session state. A workflow logs in, runs for a week, and then one day every run starts at the login page because the container was rescheduled and the profile directory vanished.

Hosted runtimes solve this with persistent profiles: named storage that survives across sessions. You reference a profile by ID when you create a session, and the runtime mounts the saved cookies, local storage, and cache into the new browser. The next run picks up where the last one left off.

The operational challenge is the profile list. Once you have more than a handful of profiles — one per customer account, one per region, one per test tenant — you need to treat them as data:

• Name them by purpose, not by timestamp. acme-us-prod is debuggable; profile-2026-10-28-3 is not.
• Track ownership. Which service or team is allowed to use which profile? A profile shared across two workflows that both mutate login state will produce intermittent failures that are painful to trace.
• Plan for rotation. Credentials expire, sessions get invalidated, and a profile can become poisoned by a bad run. You need a way to reset a profile without deleting the identity it represents.
• Audit access. If a profile holds a real customer login, its usage should be logged the same way you log access to a database.

A profile list is not glamorous, but it is the difference between a web automation system that runs for months and one that needs a human every other day. The documentation covers how profiles are created and attached to sessions in this runtime.

Comparing local and hosted web automation

The table below is a decision aid, not a scorecard. Most teams end up hybrid: local for fast iteration, hosted for anything that runs on a schedule or in CI.

The scaling row deserves a caveat. Hosted runtimes do not give you unbounded parallelism; every provider has concurrency limits tied to your plan. Check the current numbers on the pricing page rather than assuming a specific ceiling.

Production criteria for choosing a runtime

When you evaluate a hosted runtime for web automation, test against your actual workload rather than a marketing page. The criteria that matter:

CDP compatibility. Does the runtime expose a standard CDP endpoint that Playwright, Puppeteer, and Selenium can connect to without a proprietary SDK? If the answer is "use our wrapper," you are locked in more than you think.

Session isolation. Each session should have its own browser process, its own storage, and its own network identity. Shared state between sessions is a correctness bug waiting to happen.

Configurable browser settings. You want control over viewport, locale, timezone, user agent, and proxy. Be precise about what "stealth" means for your provider — ask which settings are configurable and which are fixed, rather than assuming advanced fingerprint spoofing is included.

Live observability. A live viewer that lets you watch a session in real time, plus the ability to capture screenshots and DOM snapshots, cuts debugging time dramatically. The remote control browser post goes deeper on this.

Usage controls. You need per-session timeouts, a way to kill runaway sessions, and visibility into browser-hour consumption. Without these, a single infinite loop can burn through a budget overnight.

Data handling. If your automation touches customer data, ask about retention, logging, and whether session recordings are stored. Enterprise buyers will want a DPA and a clear statement on zero data retention.

Common failure modes and how to avoid them

Even with a solid runtime, web automation fails in predictable ways. A short list of the ones worth designing around:

• Selector drift. Sites change markup. Prefer role-based and label-based locators over CSS paths, and add assertions that fail loudly when a page structure changes.
• Race conditions on navigation. waitForURL and waitForLoadState are more reliable than fixed sleeps. If you find yourself adding waitForTimeout, you are usually missing a signal.
• Session leaks. A thrown exception before browser.close() leaves the session running. Wrap session creation in try/finally as shown above, and set a server-side timeout as a backstop.
• Profile contention. Two concurrent sessions using the same profile will corrupt state. Enforce a lock or use separate profiles per concurrent worker.
• IP reputation. If a target site blocks your requests, the fix is usually network identity, not a different framework. Test with a proxy before rewriting your script.

Where this fits in an agent stack

Web automation is increasingly driven by AI agents rather than hand-written scripts, but the runtime requirements are identical. An agent that decides which button to click still needs a browser that persists logins, exposes CDP, and can be watched when it goes wrong. The difference is that agents fail in less predictable ways, which makes observability and session isolation more important, not less.

If you are building that kind of system, start with a runtime that treats the browser as infrastructure: a session API, a CDP endpoint, persistent profiles, and a viewer. You can read more about running Chromium without local setup in the remote browser online guide, and the documentation has the API reference for sessions and profiles.

Getting started

The shortest path to production web automation is to stop managing browsers and start managing sessions. Concretely:

1. Pick one workflow — a login-and-extract job or a checkout smoke test — and port it to connectOverCDP.
2. Create a named persistent profile for it and confirm state survives across two runs.
3. Add a try/finally that closes the browser, and set a session timeout on the server side.
4. Run it on a schedule for a week and watch the failure rate before you scale out.

That sequence will surface the real constraints — profile contention, selector drift, proxy needs — faster than any architecture diagram. When you are ready to size concurrency and budget, check /pricing for current browser-hour rates and plan limits.