Human-in-the-Loop

Human-in-the-loop (HITL) allows human operators to take control of automated browser sessions.

Overview

HITL enables:

• Manual intervention when automation fails
• Complex decision-making scenarios
• CAPTCHA solving
• Quality assurance
• Training data collection

How It Works

┌─────────────────────────────────────────────────────────┐
│                    Browser Session                       │
├─────────────────────────────────────────────────────────┤
│                                                          │
│   ┌──────────────┐          ┌──────────────┐            │
│   │   Automated  │          │    Human     │            │
│   │   Control    │◀────────▶│   Takeover   │            │
│   └──────────────┘          └──────────────┘            │
│          │                         │                     │
│          │    Takeover Request     │                     │
│          ├────────────────────────►│                     │
│          │                         │                     │
│          │    Human Actions        │                     │
│          │◀────────────────────────┤                     │
│          │                         │                     │
│          │    Return Control       │                     │
│          │◀────────────────────────┤                     │
│                                                          │
└─────────────────────────────────────────────────────────┘

Requesting Takeover

Via Dashboard

1. Open the browser session
2. Click Request Takeover
3. Wait for human operator
4. Operator completes actions
5. Control returns to automation

Via API

curl -X POST https://your-domain.com/api/agents/{id}/browser/{sessionId}/takeover \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "CAPTCHA detected",
    "timeout": 300
  }'

Response:

{
  "takeover_id": "takeover-123",
  "status": "pending",
  "takeover_url": "https://browserbase.com/takeover/...",
  "expires_at": "2024-12-15T10:05:00Z"
}

Takeover URL

The takeover URL provides:

• Full browser control
• Visual interface
• Keyboard and mouse input
• Real-time view

Takeover States

State	Description
pending	Waiting for human
active	Human is controlling
completed	Human finished, control returned
timeout	No response within timeout
cancelled	Takeover cancelled

Workflow Integration

Use HITL in workflows:

{
  "type": "human-takeover",
  "data": {
    "reason": "Manual verification needed",
    "timeout": 600,
    "on_complete": "continue-workflow",
    "on_timeout": "retry-automation"
  }
}

Use Cases

CAPTCHA Solving

{
  "type": "if-else",
  "condition": "{{captcha_detected}}",
  "true": {
    "type": "human-takeover",
    "reason": "CAPTCHA requires human input"
  }
}

Complex Forms

When form logic is too complex:

{
  "type": "human-takeover",
  "data": {
    "reason": "Complex multi-step form",
    "instructions": "Please complete the insurance quote form"
  }
}

Quality Assurance

Human review of automated actions:

{
  "type": "human-takeover",
  "data": {
    "reason": "Review before submission",
    "instructions": "Verify all fields are correct before clicking Submit"
  }
}

Error Recovery

When automation encounters unexpected states:

{
  "type": "error-handler",
  "handler": {
    "type": "human-takeover",
    "reason": "Unexpected page state",
    "instructions": "Navigate to the correct page"
  }
}

Operator Interface

Features

• Live browser view
• Mouse and keyboard control
• Element inspector
• Screenshot capture
• Action recording
• Chat with automation

Keyboard Shortcuts

Shortcut	Action
Ctrl+Enter	Return control
Ctrl+R	Reload page
Ctrl+S	Screenshot
Esc	Cancel takeover

Best Practices

1. Clear Instructions

Provide specific guidance:

{
  "instructions": "1. Click the CAPTCHA checkbox\n2. Complete the puzzle\n3. Click Continue"
}

2. Reasonable Timeouts

Set appropriate wait times:

• Simple tasks: 1-5 minutes
• Complex tasks: 10-15 minutes
• Extended operations: 30+ minutes

3. Fallback Handling

Handle timeout gracefully:

{
  "on_timeout": "notify-admin",
  "fallback": "skip-step"
}

4. Logging

Track takeover events:

• Reason for takeover
• Duration
• Actions performed
• Outcome

5. Operator Training

Document common scenarios:

• Frequent takeover reasons
• Step-by-step guides
• Expected outcomes

Notifications

Alert Operators

Configure notifications:

{
  "takeover_notifications": {
    "email": ["operator@example.com"],
    "slack": "#browser-alerts",
    "urgency": "high"
  }
}

Slack Integration

{
  "type": "slack",
  "data": {
    "channel": "#operators",
    "message": "Browser takeover requested: {{takeover.reason}}\nURL: {{takeover.url}}"
  }
}

Monitoring

Metrics

Track HITL performance:

• Takeover frequency
• Average resolution time
• Success rate
• Timeout rate

Dashboard

View takeover history:

1. Go to Browser
2. Select session
3. View Takeover History

API Reference

Request Takeover

POST /api/agents/{id}/browser/{sessionId}/takeover

Get Takeover Status

GET /api/agents/{id}/browser/{sessionId}/takeover/{takeoverId}

Cancel Takeover

DELETE /api/agents/{id}/browser/{sessionId}/takeover/{takeoverId}

Return Control

POST /api/agents/{id}/browser/{sessionId}/takeover/{takeoverId}/complete