Browser Automation

The Browser Automation extension gives the Spectral agent full control over a real Chromium browser via Playwright. With 28 tools, the agent can navigate to URLs, click elements, fill forms, take screenshots, inspect network traffic, stream the browser, record videos, and execute multi-step scenario walkthroughs — all directly from the agent conversation.

What It Can Do

Capability	Tools
Navigation	`browser_navigate`, `browser_navigate_back`, `browser_close`, `browser_resize`
Attach to your browser	`browser_attach`, `browser_detach` (CDP)
Tabs	`browser_tabs` (list, create, close, select)
Input	`browser_click`, `browser_type`, `browser_press_key`, `browser_hover`, `browser_drag`, `browser_select_option`, `browser_fill_form`, `browser_file_upload`, `browser_drop`
Evaluate	`browser_evaluate` (JavaScript evaluation), `browser_run_code_unsafe` (Playwright-level code)
Screenshots & Snapshots	`browser_take_screenshot`, `browser_snapshot` (accessibility tree)
Network	`browser_network_requests`, `browser_network_request`
Console	`browser_console_messages`
Dialogs	`browser_handle_dialog`
Waiting	`browser_wait_for` (wait for text, text disappearance, or time)
Streaming	`browser_stream_start`, `browser_stream_stop`, `browser_stream_status`
Recording	`browser_record_start`, `browser_record_stop`, `browser_record_status`
Scenarios	`browser_run_scenario`

Video Recording

One of the most powerful features: the agent can record everything that happens in the browser as a .webm video.

// The agent calls these during a session:
browser_record_start({ filename: "login-demo" })
// ... agent navigates, clicks, types ...
browser_record_stop()
// → returns the video file path, duration, and ffmpeg conversion commands

Recordings persist across navigation, clicks, and typing — everything the agent does in the browser is captured. Use this for:

Demo creation — record walkthroughs of features
Bug reproduction — capture exact steps that trigger an issue
QA evidence — document end-to-end test runs
User onboarding — generate tutorial videos automatically

Attach to your own browser

Instead of the built-in Chromium, the agent can drive your own browser (Arc, Chrome, Chromium, Edge) over the Chrome DevTools Protocol. This means your existing tabs, cookies, and logins are all available to the agent.

Launch your browser with remote debugging:

# Arc (port 9222)
open -a Arc --args --remote-debugging-port=9222

# Google Chrome
open -a "Google Chrome" --args --remote-debugging-port=9222

Tell the agent to use it, e.g. "I launched Arc with --remote-debugging-port=9222, use my browser". The agent calls browser_attach (defaults to http://127.0.0.1:9222; a bare port like "9222" works too) and starts driving your real browser.
Detach when done — ask the agent to detach, or it calls browser_detach for you. Your browser keeps running; subsequent browser calls fall back to the built-in Chromium.

Limitations in attach mode:

No video recording — browser_record_start and browser_run_scenario with record: true are unavailable (recording requires a browser context the agent owns).
The agent never closes your browser — browser_quit in attach mode only detaches; your tabs, cookies, and windows are untouched.
If you close your browser mid-session, the agent detects the lost CDP connection, resets to the built-in browser, and tells you to call browser_attach again.

Browser Streaming (WebRTC)

The agent can stream its browser view via WebRTC using canvas capture at 15 FPS. Viewers connect through the backend signaling server.

browser_stream_start({ url: "https://example.com" })
// → returns a sessionId for viewers to connect
browser_stream_stop()

Scenario Execution

Markdown-based scenario files let you define multi-step browser tests that the agent executes in a real browser:

# Login Flow
1. **Navigate** to https://myapp.com/login
2. **Type** "[email protected]" into input[name=email]
3. **Type** "password123" into input[name=password]
4. **Click** button[type=submit]
5. **Wait** for "Dashboard" to appear
6. **Screenshot** full page

The agent calls browser_run_scenario with the scenario file and gets per-step pass/fail results back.

Accessibility Snapshots

The browser_snapshot tool captures the page's accessibility tree — a structured text representation that's much more useful for understanding page structure than a screenshot for AI agents. Each interactive element gets a unique ref that can be used in subsequent tool calls.

Architecture

Agent Conversation
       │
       ▼
  browser_* tools
       │
       ▼
  BrowserService (singleton)
       │
       ▼
  Playwright Chromium (headless or headed)

The browser is managed as a singleton service — it starts on first use and stays open for the session. On session_shutdown, it's cleanly closed.

Prerequisites

The extension requires Playwright to be installed:

npx playwright install chromium

If Playwright is not available, the extension degrades gracefully — it logs a message and skips registration without blocking the agent.

Configuration

Setting	Default	Description
`headless`	`true`	Run Chromium in headless mode
`browserName`	`chromium`	Browser engine (chromium, firefox, webkit)

When to Use

Web scraping & data extraction — the agent can navigate and extract data
Testing & QA — run automated tests from the conversation
Screenshots & audits — capture pages at different viewports
Demo recording — generate video walkthroughs of user flows
SPA debugging — inspect network requests and console errors