BLOG
Agent Benchmark: How to Measure Browser Agents Properly
An agent benchmark only means something when the runtime is controlled. Learn what to measure, what to hold constant, and how to run repeatable tests.
An agent benchmark is a repeatable test that scores how well an AI agent completes web tasks — and the score is only as trustworthy as the environment it ran in. Most public agent benchmark numbers are produced on uncontrolled infrastructure: a laptop with a warm cache, a shared CI runner, a browser that may or may not have loaded the same assets as the run before it. If you want a number you can defend to your team, you need to separate the agent's reasoning from the runtime it drove. This guide covers what to measure, what to hold constant, and how to wire a benchmark harness to a hosted Chromium session over CDP so results are reproducible.
What an agent benchmark actually measures
A benchmark score is a composite. It bundles at least four things:
- Task definition — the goal, the starting URL, and the success criteria.
- Agent policy — the model, the prompt, the tool schema, and the retry logic.
- Runtime — the browser build, viewport, network conditions, proxy, and session state.
- Harness — how the task is launched, how success is detected, and how failures are logged.
When someone reports "our agent scores X," they are usually reporting the interaction of all four. That is fine for a demo and useless for engineering. A useful agent benchmark isolates the variable you are changing. If you are evaluating a new model, the runtime must be identical across runs. If you are evaluating a runtime, the agent policy must be frozen.
The practical consequence: you cannot compare your numbers to a published leaderboard unless you match the harness. Treat external scores as directional signals about task difficulty, not as a target to beat.
The runtime is the hidden variable
Most benchmark variance comes from the browser, not the model. The same agent on the same task can pass or fail depending on:
- Browser build and version. A Chromium update can change layout, lazy-loading behavior, or which elements are clickable.
- Viewport and device emulation. A 1280×720 desktop viewport and a 390×844 mobile viewport expose different DOM and different hit targets.
- Network conditions. Cold DNS, throttled bandwidth, and flaky third-party scripts change timing-dependent behavior.
- Session state. Cookies, localStorage, and service workers from a previous run can make a task trivially easy or impossible.
- IP reputation. Some sites serve different content, challenge pages, or CAPTCHAs depending on the source IP.
- Resource contention. Ten parallel browsers on a 2-core runner behave nothing like one browser on a dedicated host.
A benchmark that does not pin these is measuring luck. Hosted Chromium sessions help because each session starts from a known image and can be isolated per task. You can read more about the runtime layer in Remote browsers for AI agents.
What to hold constant
Before you run anything, write down the constants. A minimal benchmark spec looks like this:
| Dimension | What to pin | Why it matters |
|---|---|---|
| Browser | Chromium version and channel | Layout and API behavior change between builds |
| Viewport | Width, height, device scale factor | Affects DOM visibility and click targets |
| Locale / timezone | en-US, UTC | Date and currency rendering differ |
| Network | Fixed latency/bandwidth profile | Timing-dependent tasks are sensitive to it |
| Session | Fresh profile per task, or seeded profile | Prevents cross-task contamination |
| Proxy | Fixed egress or none | IP-dependent content and challenges |
| Concurrency | Fixed number of parallel sessions | CPU contention changes timing |
| Harness | Same launcher, same success check | Removes scoring drift |
| Agent | Frozen model, prompt, tools, temperature | The variable under test |
If you change two rows at once, you have not run a benchmark — you have run an experiment with confounded variables. Change one row per comparison.
Designing tasks that discriminate
A good agent benchmark task has three properties: a clear success condition, a bounded action space, and a failure mode that is diagnosable.
Clear success condition. "Book a flight" is not a success condition. "Reach the confirmation page and extract the booking reference matching regex ^[A-Z0-9]{6}$" is. Machine-checkable success criteria let you score thousands of runs without human review.
Bounded action space. Tasks that require unbounded exploration produce high variance. Prefer tasks where the correct path is discoverable within a reasonable number of steps, and cap the step budget explicitly.
Diagnosable failure. When a run fails, you need to know why. Capture a screenshot, the DOM snapshot, the console log, and the action trace at the point of failure. A benchmark that only reports pass/fail is a benchmark you cannot improve against.
A useful task taxonomy:
- Navigation — reach a specific page through a multi-step flow (search → filter → detail).
- Extraction — pull structured data from a page that requires interaction to reveal.
- Form fill — complete a multi-field form with validation and error recovery.
- State change — perform an action that mutates server state and verify the result.
- Recovery — handle a deliberate obstacle (modal, cookie banner, slow load) without failing.
Weight these by what your production traffic actually looks like. A benchmark that is 80% navigation and 0% recovery will not predict production reliability.
Wiring a benchmark harness to hosted Chromium
The cleanest way to run an agent benchmark is to launch each task in an isolated hosted session and drive it over CDP. Playwright's `connectOverCDP` attaches to a remote browser without launching a local one, which means the browser build, viewport, and network profile are controlled by the runtime, not the test machine.
import { chromium, Browser, Page } from "playwright";
interface BenchmarkTask {
id: string;
startUrl: string;
goal: string;
maxSteps: number;
check: (page: Page) => Promise<boolean>;
}
interface RunResult {
taskId: string;
passed: boolean;
steps: number;
durationMs: number;
failureReason?: string;
}
async function runTask(
cdpEndpoint: string,
task: BenchmarkTask,
agent: (page: Page, goal: string, maxSteps: number) => Promise<number>
): Promise<RunResult> {
const started = Date.now();
let browser: Browser | undefined;
try {
browser = await chromium.connectOverCDP(cdpEndpoint);
const context = browser.contexts()[0] ?? (await browser.newContext());
const page = context.pages()[0] ?? (await context.newPage());
await page.setViewportSize({ width: 1280, height: 720 });
await page.goto(task.startUrl, { waitUntil: "domcontentloaded", timeout: 30_000 });
const steps = await agent(page, task.goal, task.maxSteps);
const passed = await task.check(page);
return {
taskId: task.id,
passed,
steps,
durationMs: Date.now() - started,
};
} catch (err) {
return {
taskId: task.id,
passed: false,
steps: 0,
durationMs: Date.now() - started,
failureReason: err instanceof Error ? err.message : String(err),
};
} finally {
// Close the CDP connection; the hosted session is released by the runtime.
await browser?.close();
}
}Two implementation notes. First, connectOverCDP gives you a Browser handle, but the session lifecycle belongs to the runtime — closing the connection does not necessarily terminate the remote browser. Check your provider's session controls so you do not leak billable sessions. Second, always set the viewport explicitly after connecting. The default viewport of a remote session may not match your benchmark spec, and a mismatch silently changes which elements are visible.
For the connection details — endpoint format, auth headers, and session creation — see the documentation.
Scoring: beyond pass rate
Pass rate is the headline number, but it hides the distribution. Report at least these:
- Pass rate — fraction of tasks that met the success condition.
- Steps to success — median and p95. A high step count on passing runs signals inefficiency that will cost you in production.
- Time to success — wall-clock, excluding session startup if you are measuring agent speed.
- Failure taxonomy — group failures by cause (element not found, timeout, wrong page, validation error, CAPTCHA). The distribution tells you what to fix.
- Cost per successful task — browser-hours consumed divided by passes. This is the number that matters for unit economics.
A benchmark that reports only pass rate will push you toward agents that pass easy tasks and fail expensively on hard ones. The failure taxonomy is where the engineering value is.
Common mistakes
Running on a warm cache. The first run of a task is not the same as the tenth. Use a fresh profile per task, or explicitly seed and document the profile state.
Ignoring session startup cost. If your benchmark measures end-to-end time including session creation, you are partly measuring the runtime's provisioning speed. That is a legitimate metric, but label it separately from agent latency.
Scoring with an LLM judge on the same model as the agent. This introduces correlated bias. Prefer deterministic checks — URL patterns, DOM assertions, extracted values — and reserve LLM judges for tasks where no deterministic check exists.
Too few runs. Web tasks are stochastic. A 20-task suite run once gives you a point estimate with wide error bars. Run each task multiple times and report confidence intervals, or at minimum report the run count.
Comparing across runtimes without pinning the agent. If you switch from a local browser to a hosted runtime and the score changes, you have learned something about the runtime — but only if the agent policy was frozen. Otherwise you have learned nothing.
When a hosted runtime is the right benchmark substrate
A hosted Chromium runtime is the right substrate for an agent benchmark when you need any of the following:
- Reproducibility across machines. The browser image is the same whether the harness runs on a laptop, CI, or a cloud worker.
- Isolation per task. Fresh sessions prevent cross-task contamination without manual profile cleanup.
- Controlled network egress. Fixed proxy configuration removes IP-dependent variance.
- Parallelism without contention. Each session gets its own resources, so concurrency does not silently change timing.
- Observability. Live viewer and session recordings let you inspect failures after the fact instead of guessing.
The trade-off is cost and a dependency on the provider's browser build. If your benchmark is specifically about a browser version you ship to users, you may need to pin that version in the runtime or run locally. For most agent evaluation, the hosted path is the more honest measurement. See Remote browser online for how sessions are provisioned, and Remote control browser for the control-plane side.
A minimal benchmark protocol
If you take nothing else from this article, take this protocol:
- Write the spec. Pin every row in the constants table above.
- Build 30–50 tasks across navigation, extraction, form fill, state change, and recovery.
- Freeze the agent. Model, prompt, tools, temperature, step budget.
- Run each task N times (N ≥ 3) on isolated sessions.
- Capture artifacts on every failure: screenshot, DOM, console, action trace.
- Report pass rate, steps, time, failure taxonomy, and cost per pass.
- Change one variable. Re-run. Compare distributions, not point estimates.
This protocol will not produce a number that competes with a public leaderboard. It will produce a number you can act on, which is the only kind that matters. For current session limits and usage controls, check pricing.
The bottom line
An agent benchmark is a measurement instrument, and like any instrument it needs calibration. The agent is one variable; the runtime is another; the harness is a third. Most published scores conflate all three. If you want to know whether your agent is actually getting better, control the runtime, freeze the policy, and report the distribution — not just the pass rate. Hosted Chromium sessions make that control practical, because the browser becomes a parameter you set rather than a machine you hope is configured correctly.