What is measured
Benchmarks should separate browser session startup, CDP connection readiness, first navigation, task completion, and teardown. Keeping those measurements separate helps teams understand whether delay comes from the browser runtime, the target site, the agent model, or the automation script.
- Session startup: create request to browser ready.
- CDP connection: WebSocket URL available to client connected.
- Workflow time: first navigation to validated task output.
How to reproduce
Run the same script from a stable environment, record the region, plan, resolution, profile state, proxy settings, browser version, target URL, date, sample size, and percentile method. Report p50, p75, p95, and failures instead of only best-case timings.
- Use at least 30 runs for quick comparisons.
- Keep target pages and agent prompts fixed between providers.
- Publish limitations when a target site changes during a test.
How claims should be cited
Any public speed claim should link to this methodology or to a specific dated benchmark page. Claims should say what changed, what was measured, the baseline, the sample size, and whether the result is lab-only or backed by production telemetry.
- Prefer dated benchmark tables over broad marketing statements.
- Avoid comparing providers without naming test conditions.
- Update or remove claims when infrastructure changes.
