Browser Actions

Browser actions are the building blocks for web automation.

Overview

Available actions:

• Navigation
• Click interactions
• Text input
• Content extraction
• Screenshots
• Scrolling
• Waiting

Accessibility Tree Snapshots

Instead of relying on screenshots as the primary page representation, the browser now uses text-based accessibility tree snapshots. This approach is inspired by OpenClaw and provides a structured, token-efficient view of the page that the agent can reason about directly.

How It Works

When a snapshot is taken, the browser injects JavaScript that walks the DOM, assigns data-ref="N" attributes to interactive elements, and returns a structured text representation of the page.

The format uses two styles:

• Interactive elements: [N] role "label" attrs -- where N is the ref number used for targeting
• Landmarks and headings: - role "label" -- structural elements that provide context

Example snapshot output:

- heading "Search Results"
[1] link "Home"
[2] textbox "Search..." focused
[3] button "Search"
- main "Results"
  [4] link "First Result - Example Page"
  [5] link "Second Result - Another Page"
[6] button "Next Page"

Page text:
First Result - Example Page
A description of the first result found...
Second Result - Another Page
More details about the second result...

The snapshot also includes a page text content section at the bottom, which captures readable text from the page (search results, articles, etc.).

Getting a Snapshot

Use the snapshot action to get or refresh the accessibility tree:

{
  "action": "snapshot"
}

All other actions (click, type, press, scroll) automatically return a fresh snapshot in their response, so you always have an up-to-date view of the page after every interaction.

Snapshot sources

The default snapshot walks the DOM. On app-like pages that re-render their inputs, or when the DOM walk finds fewer than three controls, the Cloudflare provider reads Chrome's own accessibility tree over CDP instead (source: "ax"), in the same [N] role "name" format and with data-ref set on the backing elements so ref-based click/type still work. It is slower (one element handle per interactive node, capped at 150 refs), so it is opt-in:

{ "action": "snapshot", "source": "ax" }

Ref-Based Targeting

Instead of writing CSS selectors, you can target elements by their ref number from the snapshot. The ref number resolves to a [data-ref="N"] CSS selector internally.

{
  "action": "click",
  "ref": 5
}

This is equivalent to { "action": "click", "selector": "[data-ref='5']" } but is shorter and less error-prone. Ref numbers are reassigned on each snapshot, so always use refs from the most recent snapshot.

You can use ref-based targeting with click, type, and other element-targeting actions:

{
  "action": "type",
  "ref": 2,
  "text": "search query"
}

Token Efficiency

Accessibility tree snapshots are dramatically more efficient than screenshots:

Representation	Typical Size
Screenshot (PNG)	~5 MB
Snapshot (text)	~5-50 KB
Reduction	~100x

This makes it practical to include page state in every tool response without blowing up context windows.

Vision Model Fallback

For pages where the accessibility tree does not capture sufficient information (e.g., canvas-based applications, complex visual layouts, or CAPTCHA challenges), the agent can fall back to screenshot-based analysis using a vision model. The vision model is configurable via settings.llm.visionModel and follows an auto-fallback chain: configured vision model, then the main LLM if it supports vision, then Llama 4 Scout as a last resort.

Action Types

Navigate

Navigate to a URL:

{
  "action": "navigate",
  "url": "https://example.com"
}

Click

Click an element:

{
  "action": "click",
  "selector": "#submit-button"
}

Options:

{
  "action": "click",
  "selector": ".menu-item",
  "button": "left",
  "click_count": 2,
  "delay": 100
}

Type

Enter text:

{
  "action": "type",
  "selector": "#email-input",
  "text": "user@example.com"
}

With options:

{
  "action": "type",
  "selector": "#search",
  "text": "search query",
  "delay": 50,
  "clear": true
}

Press

Press keyboard keys:

{
  "action": "press",
  "key": "Enter"
}

Common keys:

• Enter, Tab, Escape
• ArrowUp, ArrowDown, ArrowLeft, ArrowRight
• Backspace, Delete
• Modifiers: Shift+Enter, Control+a

Extract

Extract content from page:

{
  "action": "extract",
  "selector": ".product-price",
  "attribute": "text"
}

Extract types:

{
  "action": "extract",
  "selector": "img.hero",
  "attribute": "src"
}

Multiple elements:

{
  "action": "extract",
  "selector": ".item-title",
  "multiple": true
}

Screenshot

Capture screenshot:

{
  "action": "screenshot"
}

With options:

{
  "action": "screenshot",
  "selector": "#main-content",
  "full_page": false,
  "format": "png"
}

Scroll

Scroll the page:

{
  "action": "scroll",
  "direction": "down",
  "amount": 500
}

Scroll to element:

{
  "action": "scroll",
  "selector": "#footer",
  "behavior": "smooth"
}

Wait

Wait for conditions:

{
  "action": "wait",
  "selector": ".loading",
  "state": "hidden"
}

Wait types:

// Wait for element
{
  "action": "wait",
  "selector": "#content",
  "state": "visible",
  "timeout": 10000
}

// Wait for time
{
  "action": "wait",
  "time": 2000
}

// Wait for navigation
{
  "action": "wait",
  "type": "navigation"
}

Upload

Attach a file from the agent's media library (R2) to a file input. The file is injected in-page through a DataTransfer, so nothing touches a filesystem:

{
  "action": "upload",
  "ref": 12,
  "mediaUrl": "https://media.uranus.im/<media-id>",
  "filename": "photo.jpg"
}

mediaUrl accepts a media.uranus.im URL or a bare media id (a chat attachment, generated image or screenshot — list_media returns them); filename and MIME type are looked up from the stored object.

Handoff

Hand the live browser to a human when the agent is blocked (CAPTCHA, login, a judgement call). Requires the Cloudflare provider.

{
  "action": "handoff",
  "instructions": "Solve the CAPTCHA and log in, then hand back",
  "timeout": 600000
}

Returns immediately with a handoffId and a hosted Live View URL (live.browser.run, valid up to an hour) to give the user; the session is marked awaiting_human. The agent then polls:

{ "action": "handoff_status" }

until active is false, after which it takes a fresh snapshot and continues. Timeouts are capped at 30 minutes. See Human-in-the-Loop.

browser_task — one call instead of a chain

For multi-step goals ("find the opening hours of the Orchard branch", "fill the contact form with these details") the agent prefers browser_task(goal, url?, inputs?), which lets the Jev decision model drive the snapshot → act loop internally and return the final page text, a needs_input request, or a needs_confirmation stop before anything consequential. browser_action remains the manual path when browser_task reports uncertain, blocked, stuck or driver_unavailable.

browser_execute — escape hatch

For what the fixed action list cannot express — reading hidden state, scrolling a specific container, dispatching custom events, extracting structured data — browser_execute(sessionId, script) runs an arrow function in the page:

{
  "sessionId": "…",
  "script": "() => [...document.querySelectorAll('tr')].map(r => r.innerText)"
}

It runs in page-script scope (not raw CDP), requires user approval on every call (it is in TOOLS_REQUIRING_APPROVAL), returns the function's JSON-serialisable result truncated to 8 KB, and is logged like any other action.

Executing Actions

Via API

curl -X POST https://your-domain.com/api/agents/{id}/browser/{sessionId}/execute \
  -H "Content-Type: application/json" \
  -d '{
    "action": "click",
    "selector": "#login-button"
  }'

Response:

{
  "success": true,
  "result": null,
  "duration_ms": 150
}

Via Workflow

{
  "nodes": [
    {
      "id": "nav-1",
      "type": "navigate",
      "data": { "url": "https://example.com" }
    },
    {
      "id": "click-1",
      "type": "click",
      "data": { "selector": "#login" }
    },
    {
      "id": "type-1",
      "type": "type",
      "data": {
        "selector": "#email",
        "text": "{{credentials.email}}"
      }
    }
  ]
}

Selectors

CSS Selectors

// By ID
{ "selector": "#my-element" }

// By class
{ "selector": ".my-class" }

// By attribute
{ "selector": "[data-testid='submit']" }

// By tag
{ "selector": "button" }

// Combined
{ "selector": "form#login input[type='email']" }

XPath Selectors

{
  "selector": "//button[contains(text(), 'Submit')]",
  "selector_type": "xpath"
}

Text Selectors

{
  "selector": "text=Click here",
  "selector_type": "text"
}

Action Chaining

Execute multiple actions:

curl -X POST .../browser/{sessionId}/execute \
  -d '{
    "actions": [
      { "action": "navigate", "url": "https://example.com" },
      { "action": "wait", "selector": "#content" },
      { "action": "click", "selector": "#menu" },
      { "action": "screenshot" }
    ]
  }'

Error Handling

Common Errors

Error	Cause	Solution
ElementNotFound	Selector doesn't match	Verify selector, wait for element
Timeout	Action took too long	Increase timeout
NavigationFailed	Page failed to load	Check URL, retry
ClickIntercepted	Element blocked	Wait, scroll into view

Retry Logic

{
  "action": "click",
  "selector": "#button",
  "retry": {
    "attempts": 3,
    "delay": 1000
  }
}

Advanced Actions

Hover

{
  "action": "hover",
  "selector": ".dropdown-trigger"
}

Select

Select dropdown option:

{
  "action": "select",
  "selector": "#country",
  "value": "US"
}

Upload

Upload file:

{
  "action": "upload",
  "selector": "input[type='file']",
  "file_path": "/path/to/file.pdf"
}

Evaluate

Execute JavaScript:

{
  "action": "evaluate",
  "script": "document.querySelector('#count').textContent"
}

Best Practices

1. Use Stable Selectors

Prefer:

• IDs and data attributes
• Semantic selectors

Avoid:

• Dynamic classes
• Position-based selectors

2. Wait for Elements

Always wait before interacting:

[
  { "action": "wait", "selector": "#form" },
  { "action": "type", "selector": "#email", "text": "..." }
]

3. Handle Dynamic Content

Wait for content to load:

{
  "action": "wait",
  "selector": ".loading",
  "state": "hidden"
}

4. Set Appropriate Timeouts

Adjust for slow pages:

{
  "action": "wait",
  "selector": "#data-table",
  "timeout": 30000
}

5. Validate Results

Check action results:

{
  "action": "extract",
  "selector": ".success-message",
  "validate": true
}

API Reference

See Browser API for complete endpoint documentation.