Web Research Tool: Hosted Chromium for AI Agents

A web research tool for AI agents is not a search API with a nicer wrapper. It is a browser runtime that lets an agent open real pages, follow links, fill forms, read rendered content, and return structured findings. The hard part is not the model. It is keeping a real Chromium session alive, observable, and reproducible while the agent navigates. This guide covers what a web research tool actually needs, how hosted Chromium fits, and how to wire it with Playwright over CDP.

If you are still deciding between a local browser and a remote one, start with Remote Browser for AI agents. This article assumes you have picked the runtime and now need to make research workloads work.

What a web research tool actually does

Research is a multi-step browser workload, not a single fetch. A typical agent task looks like this:

1. Open a search engine or a known source URL.
2. Read the rendered result list, not the raw HTML.
3. Open two or three candidate pages in tabs.
4. Extract text, tables, and metadata.
5. Follow a link to a primary source.
6. Record what was found, with the URL and timestamp.

Each step depends on the previous one. If the browser dies at step four, the agent loses context. If the page renders differently because of a missing cookie or a blocked script, the extraction is wrong. If you cannot replay the session, you cannot debug it.

That is why a research tool is really a runtime problem. The model decides *what* to read; the browser decides *whether it can read it at all*.

Why a plain HTTP client is not enough

Many pages now render client-side. Search results, dashboards, documentation portals, and pricing pages often load content after JavaScript runs. An HTTP client sees the shell, not the content. A headless browser sees the final DOM, which is what the agent needs.

There is also the question of state. Research frequently requires logging in, accepting a cookie banner, or setting a locale. A browser session carries that state across pages. A stateless fetch does not.

The runtime requirements for research workloads

Before comparing tools, define what "good" means for research. These are the criteria that separate a demo from a production pipeline.

• Real rendering. Chromium with JavaScript enabled, not a text extractor.
• Session persistence. Cookies, local storage, and logins survive across steps and across worker restarts.
• Observability. A live view or recording so you can see what the agent saw.
• Isolation. One task's cookies and network settings do not leak into another.
• Programmatic control. CDP access so Playwright, Puppeteer, or Selenium can drive it.
• Usage controls. You can cap session time and concurrency so a runaway loop does not burn budget.

Remote Browser provides hosted Chromium sessions with CDP access, a live viewer, persistent profiles, configurable browser settings, session isolation, and usage controls. Pricing and limits are on /pricing; check there for current numbers rather than relying on secondhand figures.

How hosted Chromium fits a research pipeline

The architecture is simple. Your agent runs wherever you run code. It connects to a remote Chromium instance over a WebSocket endpoint using the Chrome DevTools Protocol. From there, Playwright or Puppeteer drives the page exactly as it would a local browser.

The difference is where the browser lives. Instead of installing Chromium on every worker, you request a session and connect. That matters for research because:

• Workers are disposable. A research job can run on a short-lived container and still keep its browser state in a persistent profile.
• Sessions are inspectable. The live viewer lets a human watch the agent mid-task, which is invaluable when extraction goes wrong.
• Isolation is the default. Each session gets its own browser context, so parallel research tasks do not share cookies.

This is the same pattern described in Remote web browser: the practical runtime for browser automation, applied specifically to research.

CDP is the contract

The Chrome DevTools Protocol is the interface between your automation code and the browser. Playwright's connectOverCDP method attaches to a running browser rather than launching one. That is the whole trick. The official Playwright CDP documentation covers the method signature; the practical part is that your remote endpoint must expose a CDP WebSocket URL.

A minimal Playwright + CDP research loop

The following TypeScript example connects to a hosted Chromium session, runs a small research loop, and extracts structured results. It assumes you have a CDP endpoint from your runtime provider.

import { chromium, type Browser, type Page } from "playwright";

type Finding = {
  url: string;
  title: string;
  excerpt: string;
};

async function connect(cdpUrl: string): Promise<Browser> {
  // connectOverCDP attaches to an already-running browser.
  return chromium.connectOverCDP(cdpUrl);
}

async function research(
  page: Page,
  query: string,
  maxSources = 3
): Promise<Finding[]> {
  const findings: Finding[] = [];

  await page.goto(`https://duckduckgo.com/?q=${encodeURIComponent(query)}`, {
    waitUntil: "domcontentloaded",
  });

  // Wait for rendered results, not the initial HTML shell.
  await page.waitForSelector('a[data-testid="result-title-a"]');

  const links = await page
    .locator('a[data-testid="result-title-a"]')
    .evaluateAll((nodes) =>
      nodes
        .map((n) => (n as HTMLAnchorElement).href)
        .filter((href) => href.startsWith("http"))
        .slice(0, 3)
    );

  for (const url of links.slice(0, maxSources)) {
    const tab = await page.context().newPage();
    try {
      await tab.goto(url, { waitUntil: "domcontentloaded", timeout: 30_000 });
      const title = await tab.title();
      const excerpt = await tab
        .locator("main, article, body")
        .first()
        .innerText()
        .then((t) => t.replace(/\s+/g, " ").slice(0, 400));
      findings.push({ url, title, excerpt });
    } finally {
      await tab.close();
    }
  }

  return findings;
}

async function main() {
  const cdpUrl = process.env.REMOTE_BROWSER_CDP_URL;
  if (!cdpUrl) throw new Error("Set REMOTE_BROWSER_CDP_URL");

  const browser = await connect(cdpUrl);
  const context = browser.contexts()[0] ?? (await browser.newContext());
  const page = await context.newPage();

  const results = await research(page, "hosted chromium for ai agents");
  console.log(JSON.stringify(results, null, 2));

  // Do not close the browser here; the runtime owns the session lifecycle.
  await page.close();
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

A few details matter in production:

• `waitUntil: "domcontentloaded"` plus an explicit selector is more reliable than networkidle, which can hang on pages with long-polling or analytics beacons.
• Open each source in a new page so a failed navigation does not destroy the search results page.
• Do not call `browser.close()` on a hosted session unless you intend to end it. The runtime manages teardown.

Comparing runtime options for research

There is no single best runtime. The right choice depends on how much control you need and how much infrastructure you want to own.

The middle rows are where most teams get stuck. Self-hosting a Chromium fleet means managing browser versions, zombie processes, memory leaks, and network egress. A hosted runtime moves that work to the provider while keeping CDP access, so your Playwright code does not change.

If you want to compare a hosted runtime against running Chromium yourself, Remote browser online: run real Chromium without managing Chrome walks through the trade-offs.

Where browser-use and similar frameworks fit

Frameworks like browser-use and Stagehand sit *above* the runtime. They decide which element to click or which page to read. The runtime decides whether the browser is alive and reachable. You can pair either framework with a hosted Chromium session; the framework does not care where the browser runs as long as it can connect over CDP.

This separation is useful. You can swap the agent framework without rebuilding your browser infrastructure, and you can swap the runtime without rewriting your agent logic.

Production criteria for a research tool

Once the prototype works, the questions change. Here is what to evaluate before you run research at scale.

Session lifecycle

Research tasks can run for minutes. You need to know how long a session can live, what happens when it times out, and whether state persists across reconnects. Persistent profiles are the mechanism that lets a logged-in session survive a worker restart.

Observability

A live viewer is not a luxury. When an agent extracts the wrong paragraph, you need to see the page it actually saw. Recordings help with post-hoc debugging and with auditing what a research task touched.

Isolation and data handling

Research often touches authenticated pages. Session isolation ensures one task's cookies do not bleed into another. If you handle sensitive data, check the provider's data retention policy and whether sessions are destroyed on completion.

Usage controls

A research loop that follows links recursively can run far longer than intended. Concurrency caps and session time limits are the guardrails. Refer to /pricing for how usage is metered.

Network configuration

Some sources behave differently by region or block datacenter IPs. Configurable browser settings and proxy options matter here. Be precise about what you need: a provider may offer configurable proxy settings without guaranteeing any particular success rate against anti-bot systems.

Common failure modes and how to handle them

Most research failures fall into a few categories. Knowing them saves debugging time.

• Selector drift. Search engines and sites change markup. Prefer semantic selectors and add fallbacks. Log the DOM snapshot when extraction fails.
• Navigation timeouts. Some pages never reach networkidle. Use domcontentloaded plus a specific selector.
• Cookie banners blocking content. Handle the banner explicitly, or persist a profile where it was already dismissed.
• Session expiry mid-task. Check for login redirects and re-authenticate through the persistent profile rather than starting over.
• Runaway recursion. Cap depth and total pages per task in your agent code, not just in the runtime.

None of these are solved by a better model. They are solved by treating the browser as infrastructure with explicit failure handling.

When to move from local to hosted

Start local. It is faster to iterate and easier to inspect. Move to a hosted runtime when one of these becomes true:

• You need to run many research tasks in parallel and local Chromium instances exhaust memory.
• Your workers are ephemeral and cannot hold browser state.
• You need a live viewer or recordings for debugging and audit.
• You are spending more time maintaining Chromium than building the agent.

The migration itself is small if you already use Playwright. Replace chromium.launch() with chromium.connectOverCDP(endpoint). The rest of your code stays the same. The documentation covers connection setup, profiles, and session controls.

Bottom line

A web research tool is a browser runtime plus an agent loop. The agent decides what to read; the runtime decides whether reading is possible, repeatable, and observable. Hosted Chromium over CDP gives you real rendering, persistent state, and a live view without owning a browser fleet. Start with a local prototype, define your production criteria early, and move to a hosted runtime when the operational burden outweighs the control you gain. For the broader runtime picture, see Remote control browser: when code and agents need to drive the web.