Browser Control In-App Browser Skill Skill.Md: A Practical Guide

# Browser Control In-App Browser Skill Skill.Md: A Practical Guide

Browser control in-app browser skill skill.md is a pattern that gives AI agents the ability to navigate and interact with web pages through a structured skill definition. This guide explains how to implement browser control for AI agents using a hosted Chromium runtime, why that matters, and how to wire it up with Playwright and CDP. You'll learn how to move from fragile local browser setups to a production-ready remote browser architecture.

The Problem with Local Browser Control

Most in-app browser skill definitions assume your agent runs on the same machine as the browser. That works for a demo. In production, it fails for several reasons:

• Session persistence: Local browser profiles die with the process. If your agent needs to log in, maintain a cart, or keep a session alive across multiple cloud workers, a local browser can't help.
• Scaling: Running 50 headless Chrome instances on one machine exhausts memory and CPU. You end up managing infrastructure instead of building your agent.
• Network constraints: Local browsers use your machine's IP. If you're scraping or automating tasks that require different IPs or proxy configurations, you're stuck.
• Debugging: When a local browser fails, you have logs. When a remote browser fails, you need a live view and session replay to understand what happened.

The solution is a remote browser runtime. Instead of controlling a browser on your local machine, you connect to a hosted Chromium instance over the network. This is what Remote Browser provides, and it's the pattern we'll implement in this guide.

What Is a Browser Control Skill?

A browser control skill is a structured instruction set—often a skill.md file—that tells an AI agent how to interact with a browser. It typically includes:

• Tool definitions: Functions like navigate(url), click(selector), type(text), screenshot().
• State management: How to handle sessions, cookies, and local storage.
• Error handling: What to do when an element isn't found or a page times out.
• Configuration: Browser settings, proxy details, and viewport options.

The problem is that most skill definitions are written for a local browser. They assume chromium.launch() works and that the browser is available on localhost. In a production environment, you need to adapt these skills to connect to a remote browser.

Connecting to a Remote Browser: The Core Pattern

The key insight is that a remote browser exposes a WebSocket endpoint that speaks the Chrome DevTools Protocol (CDP). Playwright and Puppeteer can connect to this endpoint directly. Here's the pattern:

1. Create a browser session via the Remote Browser API.
2. Get the WebSocket URL for the session.
3. Connect using Playwright's connectOverCDP method.
4. Run your automation as if the browser were local.

Here's a TypeScript example using Playwright:

import { chromium } from 'playwright';

async function connectToRemoteBrowser(wsEndpoint: string) {
  // Connect to the hosted Chromium instance via CDP
  const browser = await chromium.connectOverCDP(wsEndpoint);
  
  // Get the default context (this is where your persistent profile lives)
  const context = browser.contexts()[0];
  const page = await context.newPage();
  
  // Navigate and interact
  await page.goto('https://example.com');
  await page.click('button#submit');
  await page.fill('input[name="email"]', 'agent@example.com');
  
  // Take a screenshot for debugging
  await page.screenshot({ path: 'debug.png' });
  
  // Don't close the browser—the session stays alive on the server
  // await browser.close();
}

The critical difference from local automation: you don't launch a browser. You connect to an existing one. This means your agent can:

• Reconnect to the same session later.
• Share a session across multiple workers.
• Persist cookies and local storage between runs.

How to Keep Browser Sessions Alive Across Multiple Cloud Workers

One of the most common questions we hear is: "How do I keep a browser session alive across multiple cloud workers?" The answer is to decouple the browser from the worker.

In a traditional setup, each worker launches its own browser. When the worker dies, the browser dies. With a remote browser, the browser lives on a server. Workers connect to it, do their work, and disconnect. The browser stays alive.

Here's how to implement this:

1. Create a session with a persistent profile ID.
2. Store the session ID in your database or cache.
3. Have each worker connect to the same session ID.
4. Use a lock to prevent concurrent access if needed.

The Remote Browser API supports persistent profiles. You create a profile once, then reuse it across sessions. This gives you:

• Stable cookies: Logins persist.
• Consistent state: Local storage and IndexedDB survive.
• Session continuity: Your agent can pick up where it left off.

Playwright Connect to Remote Browser: Step-by-Step

Let's walk through a concrete implementation. We'll use the Remote Browser API to create a session and connect with Playwright.

Step 1: Create a Browser Session

First, you need to create a browser session. The Remote Browser API exposes a REST endpoint for this. You'll get back a JSON object with a wsEndpoint field.

curl -X POST https://api.remote-browser.dev/v1/sessions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "profileId": "my-persistent-profile",
    "proxy": { "enabled": true }
  }'

The response includes the WebSocket endpoint you'll use to connect.

Step 2: Connect with Playwright

Use the connectOverCDP method to attach to the remote browser. This is the same method you'd use to connect to a local Chrome instance running with --remote-debugging-port.

import { chromium } from 'playwright';

const wsEndpoint = 'wss://api.remote-browser.dev/v1/sessions/abc123';
const browser = await chromium.connectOverCDP(wsEndpoint);

Step 3: Run Your Automation

Once connected, you use the standard Playwright API. The only difference is that you don't manage the browser lifecycle. The remote runtime handles that.

Step 4: Debug with the Live Viewer

One advantage of a remote browser is the live viewer. You can watch your agent's actions in real time. This is invaluable for debugging. If your agent fails, you can see exactly what happened.

Browser Automation API: What to Look For

When evaluating a browser automation API, consider these production criteria:

Remote Browser provides all of these. The API is designed for AI agents, not just test harnesses.

Chrome Remote Desktop Typing Fix: A Common Issue

A frequent problem in browser automation is typing. When you use page.type() or page.fill(), the browser simulates keystrokes. This can fail if the page has custom JavaScript that intercepts input events.

The "chrome remote desktop typing fix" is a known workaround. It involves sending input events at the CDP level rather than relying on Playwright's high-level API. Here's how to do it:

import { chromium } from 'playwright';

const browser = await chromium.connectOverCDP(wsEndpoint);
const page = await browser.contexts()[0].pages()[0];

// Use CDP directly to dispatch input events
const client = await page.context().newCDPSession(page);
await client.send('Input.insertText', { text: 'your text here' });

This bypasses the JavaScript event handlers that might interfere with normal typing. It's a useful technique when you're dealing with complex web apps.

What Is an Agent Browser?

An agent browser is a browser runtime designed for AI agents. Unlike a standard browser, it's built to be:

• Programmatically controlled: Via APIs and CDP.
• Persistent: Sessions survive across connections.
• Observable: You can watch what the agent sees.
• Isolated: Each agent gets its own environment.

This is different from a "browser in the cloud" that just runs Chrome on a server. An agent browser is a managed service that handles the operational complexity: scaling, session management, proxy configuration, and debugging tools.

Browser Benchmark: Measuring Performance

When you're choosing a remote browser, benchmarks matter. But be careful about what you measure. A browser benchmark should test:

• Session startup time: How long from API call to ready-to-use browser?
• Navigation speed: How fast does the browser load pages?
• Stability: How often do sessions crash or hang?
• Concurrency: How many sessions can run in parallel without degradation?

We don't publish synthetic benchmarks because they rarely reflect real-world performance. Instead, we recommend you test with your actual workload. Create a session, run your agent, and measure the results.

Remote Browser Configuration Tool: Getting It Right

Configuration is where most automation projects fail. Here are the settings that matter:

• Viewport size: Some sites render differently based on viewport. Set this explicitly.
• User agent: Some sites block default headless user agents. Configure a realistic one.
• Locale and timezone: These affect how sites display content and handle dates.
• Proxy: If you're accessing geo-restricted content, you need the right proxy.
• Permissions: Grant or deny camera, microphone, and notification permissions.

Remote Browser exposes these as configurable browser settings in the API. You set them when creating a session, and they apply to all connections to that session.

Web Automation API: The Missing Piece

Most AI agents need more than just a browser. They need a web automation API that handles the boring parts:

• Session management: Creating, reusing, and destroying sessions.
• State persistence: Saving and loading profiles.
• Error recovery: Restarting crashed sessions.
• Logging: Recording actions and screenshots for debugging.

This is what Remote Browser provides. It's not just a browser; it's a runtime for web automation.

Putting It All Together: A Production-Ready Skill

Here's a complete skill.md example that uses Remote Browser for browser control:

# Browser Control Skill

## Purpose
Control a remote Chromium browser for web automation tasks.

## Tools

### navigate(url: string)
Navigates the current page to the specified URL.
- Uses: `page.goto(url, { waitUntil: 'networkidle' })`

### click(selector: string)
Clicks the first element matching the CSS selector.
- Uses: `page.click(selector)`

### type(selector: string, text: string)
Types text into the element matching the selector.
- Uses: `page.fill(selector, text)`

### screenshot(path?: string)
Takes a screenshot of the current page.
- Uses: `page.screenshot({ path })`

## Configuration

### Connection
Connect to the remote browser using the WebSocket endpoint:

const browser = await chromium.connectOverCDP(wsEndpoint);

### Session Persistence
Use a persistent profile ID to maintain state across runs.

### Error Handling
If a navigation times out, retry once. If it fails again, take a screenshot and report the error.

## Example Workflow

1. Navigate to `https://example.com`
2. Click `button#login`
3. Type `user@example.com` into `input#email`
4. Type `password123` into `input#password`
5. Click `button#submit`
6. Screenshot to `login-result.png`

This skill definition works with any agent that can execute TypeScript and has access to the Remote Browser API.

Conclusion

Browser control for AI agents doesn't have to mean managing local Chrome instances. By connecting to a hosted Chromium runtime, you get persistence, scalability, and observability without the infrastructure burden.

The pattern is simple: create a session, connect via CDP, run your automation, and reuse the session when you need it. Whether you're building a web agent, a scraping pipeline, or a QA harness, this approach works.

For more details on the API and configuration options, check the documentation. If you're ready to start, see the pricing page for current rates and limits.

Further Reading

• Remote Browser for AI Agents: The Missing Runtime Layer
• Remote Browser Online: Run Real Chromium Without Managing Chrome
• Remote Web Browser: The Practical Runtime for Browser Automation
• Remote Control Browser: When Code and Agents Need to Drive the Web

For a deeper dive into CDP, see the Chrome DevTools Protocol documentation.