Best Infra Setup for Running a Bunch of Browser Automations

# Best Infra Setup for Running a Bunch of Browser Automations

If you're running a bunch of browser automations in production—whether they're Playwright test suites, Selenium grids, or AI agents that need to click through the web—the infrastructure you choose determines everything: reliability, cost, and how much of your engineering time gets eaten by flaky sessions. The best infra setup for running a bunch of browser automations is not a local Chrome instance or a single EC2 box with a cron job. It's a hosted Chromium runtime that gives you isolated sessions, persistent profiles, and CDP access without forcing you to babysit the browser process.

This post walks through the concrete infrastructure decisions you need to make when scaling browser automation: where the browser runs, how sessions stay alive, how you connect to it, and what to watch out for when you move from a single script to a fleet of concurrent automations.

Why Local Browser Setup Breaks at Scale

The default approach—install Playwright or Selenium locally, write a script, run it—works fine for a handful of jobs. It falls apart when you need to run many automations concurrently or keep them running for hours.

Here's what happens when you try to scale a local setup:

• Resource contention: Each Chromium instance eats 300–800 MB of RAM. Ten concurrent sessions on a single machine will exhaust memory and cause crashes.
• Session loss: If your script dies, the browser dies with it. There's no way to reattach to a session that was running in a worker that got recycled.
• IP blocking: Running many automations from the same IP address triggers bot detection on target sites. You need to manage IP diversity, which is painful to handle locally.
• No live debugging: When a test fails at 2 AM, you want to see what the browser was looking at. Local setups don't give you a replayable session.

The solution is to treat the browser as a service: a hosted Chromium instance that you connect to over the network, rather than a process you spawn on your own machine.

The Core Components of a Production Browser Automation Stack

A production-grade setup for running many browser automations has five components:

1. Browser runtime: Hosted Chromium instances that are isolated from each other.
2. Connection layer: CDP (Chrome DevTools Protocol) or WebDriver endpoints that let your code attach to a running browser.
3. Session persistence: The ability to keep a browser session alive across multiple requests, workers, or even days.
4. Profile management: Persistent cookies, localStorage, and browser fingerprints that survive restarts.
5. Scaling and orchestration: The ability to spin up and tear down browsers on demand, without manual provisioning.

Let's look at each one in detail.

Browser Runtime: Hosted Chromium vs. Self-Managed

The first decision is where the browser actually runs. You have two options: manage your own browser fleet or use a hosted browser service.

Self-managed infrastructure gives you control, but it costs engineering time. Every Chromium update, every security patch, every flaky session that needs debugging is on you. Hosted Chromium removes that overhead, which is why it's the better choice when you're running a bunch of automations and want to focus on the automation logic, not the browser plumbing.

How to Connect: CDP, Playwright, and Selenium

Once the browser is hosted, you need a way to connect to it. The most flexible approach is CDP—the protocol that Chrome DevTools uses. CDP gives you low-level control over the browser: you can navigate, click, extract DOM, capture screenshots, and even emulate network conditions.

Playwright and Puppeteer both support connecting to an existing browser over CDP. Here's a TypeScript example using Playwright to connect to a hosted Chromium session:

import { chromium } from 'playwright';

// Connect to a hosted Chromium session via CDP
const browser = await chromium.connectOverCDP('wss://remote-browser.dev/cdp/v1/session_abc123');

// Get the default context and page
const context = browser.contexts()[0];
const page = context.pages()[0];

// Navigate and interact
await page.goto('https://example.com');
await page.fill('#search', 'browser automation');
await page.click('button[type="submit"]');

// Wait for results
await page.waitForSelector('.results');
const results = await page.$$eval('.result-title', els => els.map(e => e.textContent));

console.log(results);

// The session stays alive even after this script exits
await browser.close();

The key difference from a local setup: connectOverCDP attaches to an already-running browser. If your script crashes, the browser session survives. You can reconnect from a different worker, inspect the current state, and continue where you left off.

For Selenium users, the equivalent is a WebDriver endpoint that points to a hosted browser. The connection string changes, but the pattern is the same: your test code talks to a remote browser, not a local one.

Keeping Browser Sessions Alive Across Cloud Workers

One of the hardest problems in browser automation is session persistence. In a typical cloud deployment, workers are ephemeral—they get created, do a job, and get destroyed. If your browser session lives inside a worker, it dies with the worker.

The fix is to decouple the browser session from the worker. The browser runs in a hosted environment, and your worker connects to it over CDP. This gives you three important capabilities:

1. Reconnection: If a worker dies, a new worker can connect to the same browser session and pick up where the old one left off.
2. Long-running tasks: A browser can stay open for hours or days, even if no worker is actively connected to it.
3. State preservation: Cookies, localStorage, and session data persist in the browser profile, so you don't lose login state between runs.

This is especially important for AI agents that need to maintain context across multiple steps. An agent that's booking a flight, filling out a form, or scraping a paginated site needs the browser to stay alive between LLM calls. With a hosted session, the agent can make a call, wait for the LLM response, then reconnect to the same browser and continue.

Persistent Profiles: The Key to Realistic Automation

When you run many browser automations, you'll quickly discover that sites treat fresh browser profiles with suspicion. A brand-new Chromium instance with no cookies, no history, and a datacenter IP looks like a bot.

Persistent profiles solve this. Instead of starting from scratch every time, you can:

• Reuse login sessions: Store cookies and localStorage so you don't have to log in repeatedly.
• Maintain consistent fingerprints: Keep the same user agent, screen resolution, and other browser attributes across sessions.
• Isolate profiles per task: Use different profiles for different tasks, so a ban on one profile doesn't affect others.

Remote Browser supports persistent profiles that you can attach to any session. You create a profile once, and every session that uses that profile starts with the same state. This is a game-changer for scraping, testing authenticated flows, and running AI agents that need to maintain a consistent identity.

Configurable Browser Settings for Anti-Bot Resistance

When you're running a bunch of automations, you will hit bot detection. Sites use a combination of IP reputation, browser fingerprinting, and behavioral analysis to block automated traffic.

A hosted browser runtime should give you control over the settings that matter for anti-bot resistance:

• IP configuration: Route traffic through residential or mobile IPs to avoid datacenter IP blocks.
• Geolocation: Set a specific location for the browser, which is useful for testing geo-restricted content.
• User agent and viewport: Customize these to match a real device profile.
• WebRTC and timezone: Align these with your IP location to avoid mismatches that trigger detection.

The key word here is "configurable." You don't want a service that hardcodes stealth settings—you want the ability to tune the browser to match the specific site you're targeting. Remote Browser exposes these settings per session, so you can run different automations with different configurations without spinning up separate infrastructure.

Scaling Patterns for Concurrent Automations

Once you have a hosted browser runtime, the next question is how to scale. Here are three patterns that work well:

1. One Session Per Task

The simplest pattern: each automation task gets its own browser session. The task connects, does its work, and disconnects. The session can be terminated immediately or kept alive for a grace period.

This works well for test suites, where each test should run in isolation. It also works for scraping jobs that are independent of each other.

2. Shared Session with Sequential Tasks

For tasks that need to maintain state—like a multi-step checkout or a login-then-scrape workflow—you can share a single session across multiple workers. Worker A logs in, Worker B uses the authenticated session to scrape data, Worker C verifies the results.

This pattern reduces the number of browser instances you need, which lowers cost. It also avoids the overhead of repeated logins.

3. Session Pool with Dynamic Allocation

For high-throughput workloads, you can maintain a pool of pre-warmed sessions. Each session has a persistent profile and is ready to accept work. When a task comes in, you allocate it to an available session. When the task finishes, the session returns to the pool.

This is the most complex pattern, but it gives you the lowest latency and the highest throughput. It's the right choice when you're running hundreds or thousands of automations per hour.

What to Look for in a Browser Automation Infrastructure

When you're evaluating infrastructure for running a bunch of browser automations, here's a checklist:

• CDP support: Can you connect with Playwright, Puppeteer, or raw CDP? This determines your flexibility.
• Session persistence: Can you reconnect to a session after a worker crash? Can you keep a session alive for days?
• Profile management: Can you create and reuse persistent profiles? Do profiles include cookies, localStorage, and browser settings?
• Live debugging: Can you watch a session in real time? This is essential for debugging AI agents and complex workflows.
• IP management: Can you route sessions through different IPs? This is critical for avoiding blocks.
• Usage controls: Can you set limits on session duration, concurrent sessions, or spending? This prevents runaway costs.
• API simplicity: Can you start a session with one API call? The simpler the API, the less code you have to maintain.

The Bottom Line

The best infra setup for running a bunch of browser automations is a hosted Chromium runtime that you connect to over CDP. It gives you session persistence, profile management, and scaling without the operational overhead of managing your own browser fleet.

If you're already using Playwright or Selenium, the migration path is straightforward: change your connection string from a local browser to a remote CDP endpoint. Your existing code works, but now it runs on infrastructure that's designed for production scale.

For AI agents, the hosted runtime is even more critical. Agents need to maintain context across LLM calls, and that requires a browser session that stays alive between requests. A hosted session gives you that persistence, plus the ability to inspect what the agent is doing in real time.

Start with a single session to validate the approach, then scale up as you gain confidence. The infrastructure is the boring part—get it right, and you can focus on the automations that actually create value.

---

*Related reading: Remote Browser for AI Agents, Remote Browser Online, Remote Web Browser. For API details, see the documentation. For current pricing, see /pricing.*