Browser Automation
The Browser Automation extension gives the Spectral agent full control over a real Chromium browser via Playwright. With 28 tools, the agent can navigate to URLs, click elements, fill forms, take screenshots, inspect network traffic, stream the browser, record videos, and execute multi-step scenario walkthroughs — all directly from the agent conversation.
What It Can Do
| Capability | Tools |
|---|---|
| Navigation | browser_navigate, browser_navigate_back, browser_close, browser_resize |
| Tabs | browser_tabs (list, create, close, select) |
| Input | browser_click, browser_type, browser_press_key, browser_hover, browser_drag, browser_select_option, browser_fill_form, browser_file_upload, browser_drop |
| Evaluate | browser_evaluate (JavaScript evaluation), browser_run_code_unsafe (Playwright-level code) |
| Screenshots & Snapshots | browser_take_screenshot, browser_snapshot (accessibility tree) |
| Network | browser_network_requests, browser_network_request |
| Console | browser_console_messages |
| Dialogs | browser_handle_dialog |
| Waiting | browser_wait_for (wait for text, text disappearance, or time) |
| Streaming | browser_stream_start, browser_stream_stop, browser_stream_status |
| Recording | browser_record_start, browser_record_stop, browser_record_status |
| Scenarios | browser_run_scenario |
Video Recording
One of the most powerful features: the agent can record everything that happens in the browser as a .webm video.
// The agent calls these during a session:
browser_record_start({ filename: "login-demo" })
// ... agent navigates, clicks, types ...
browser_record_stop()
// → returns the video file path, duration, and ffmpeg conversion commands
Recordings persist across navigation, clicks, and typing — everything the agent does in the browser is captured. Use this for:
- Demo creation — record walkthroughs of features
- Bug reproduction — capture exact steps that trigger an issue
- QA evidence — document end-to-end test runs
- User onboarding — generate tutorial videos automatically
Browser Streaming (WebRTC)
The agent can stream its browser view via WebRTC using canvas capture at 15 FPS. Viewers connect through the backend signaling server.
browser_stream_start({ url: "https://example.com" })
// → returns a sessionId for viewers to connect
browser_stream_stop()
Scenario Execution
Markdown-based scenario files let you define multi-step browser tests that the agent executes in a real browser:
# Login Flow
1. **Navigate** to https://myapp.com/login
2. **Type** "[email protected]" into input[name=email]
3. **Type** "password123" into input[name=password]
4. **Click** button[type=submit]
5. **Wait** for "Dashboard" to appear
6. **Screenshot** full page
The agent calls browser_run_scenario with the scenario file and gets per-step pass/fail results back.
Accessibility Snapshots
The browser_snapshot tool captures the page's accessibility tree — a structured text representation that's much more useful for understanding page structure than a screenshot for AI agents. Each interactive element gets a unique ref that can be used in subsequent tool calls.
Architecture
Agent Conversation
│
▼
browser_* tools
│
▼
BrowserService (singleton)
│
▼
Playwright Chromium (headless or headed)
The browser is managed as a singleton service — it starts on first use and stays open for the session. On session_shutdown, it's cleanly closed.
Prerequisites
The extension requires Playwright to be installed:
npx playwright install chromium
If Playwright is not available, the extension degrades gracefully — it logs a message and skips registration without blocking the agent.
Configuration
| Setting | Default | Description |
|---|---|---|
headless | true | Run Chromium in headless mode |
browserName | chromium | Browser engine (chromium, firefox, webkit) |
When to Use
- Web scraping & data extraction — the agent can navigate and extract data
- Testing & QA — run automated tests from the conversation
- Screenshots & audits — capture pages at different viewports
- Demo recording — generate video walkthroughs of user flows
- SPA debugging — inspect network requests and console errors