SightRig

A traceable runtime and visual workbench for GUI agents.

Build agents that see rendered interfaces, act through guarded GUI operations,
hand control to a person when needed, and turn successful runs into reusable flows.

Quick start · How it works · SDK · Runtimes · Trace & replay · Docs

SightRig is a provider-neutral TypeScript runtime for agents that operate graphical interfaces. The model receives rendered screenshots—not DOM or accessibility trees—and returns actions that SightRig validates, executes, traces, and verifies.

The same Agent Loop can drive isolated local Chromium, an AgentBay cloud browser, or an AgentBay Android device. Models and GUI runtimes are dependency-injected, so adding a new provider does not require changing the loop.

Why SightRig

Stable Agent Loop Frame-bound actions, normalized coordinates, pause and stop fences, stall detection, retries, and explicit terminal states.	Browser and Mobile Run against local Chromium, a remote browser over CDP, or Android through the same observation and action protocol.
Trace by construction Every screenshot, decision, action, guard rejection, latency, and verification result becomes an append-only event.	Human handoff Pause in waiting_human, preserve the environment, let a person take over, then resume the same run.
Explore once, replay cheaply Compile a successful model run into a semantic browser flow and execute it later without model inference.	Model-neutral core Use any ModelAdapter; OpenAI-compatible transport and Qwen UI protocols are optional extensions.

SightRig intentionally executes one action per model turn. The protocol can represent a batch, but batches larger than one are rejected until their safety semantics are explicit.

Quick start

Requirements

• Node.js 20 or newer
• pnpm 9 or newer
• An OpenAI-compatible vision model endpoint
• macOS when using persistent Keychain-backed secrets

Run the Web Console

git clone https://github.com/coetzeexu/sightrig.git
cd sightrig
pnpm install
pnpm --filter @sightrig/browser exec playwright install chromium
pnpm build
pnpm start

Open http://127.0.0.1:4318, then:

1. Open Settings → Models and add an OpenAI-compatible vision model.
2. Choose a browser or mobile runtime in the chat composer.
3. Describe the task and send it. A run starts immediately.
4. Select any Agent Step to inspect its screenshot, canonical action, result, and timing.

For source development, run pnpm dev. Vite serves the console at http://127.0.0.1:5173 and proxies /api to the local daemon.

How SightRig works

flowchart LR
    Console["Web Console"] -->|REST commands| Daemon["Local daemon"]
    Daemon -->|SSE trace| Console

    subgraph Runtime["SightRig runtime"]
      Runner["AgentRunner"] --> Model["ModelAdapter"]
      Runner --> Env["AgentEnvironment"]
      Runner --> Actions["ActionProvider"]
      Runner --> Trace["TraceStore"]
      Runner -. handoff .-> Human["HumanTakeoverProvider"]
    end

    Daemon --> Runner
    Env --> Local["Local Chromium"]
    Env --> Cloud["AgentBay browser"]
    Env --> Mobile["AgentBay Android"]

Loading

Each turn crosses the same guarded loop:

flowchart LR
    Observe["Observe\nscreenshot + frameId"] --> Decide["Decide\nmodel adapter"]
    Decide --> Guard["Guard\nschema + frame + policy"]
    Guard --> Act["Act\nprovider"]
    Act --> Settle["Settle and re-observe"]
    Settle --> Verify["Trace and verify"]
    Verify -->|continue| Observe
    Verify -->|done| End["succeeded · unverified · failed"]

Loading

Actions use 0..1000 normalized coordinates and must reference the current frameId. Unknown actions, stale frames, and out-of-bounds coordinates are rejected before they reach the environment.

SDK quickstart

import { BrowserEnvironment, VisibleBrowserTakeoverProvider } from '@sightrig/browser';
import { createAgentRunner } from '@sightrig/core';
import { QwenModelAdapter } from '@sightrig/model/qwen';
import { FileTraceStore } from '@sightrig/trace';

const browser = new BrowserEnvironment({ headless: false });

const runner = createAgentRunner({
  model: new QwenModelAdapter({
    baseURL: 'http://127.0.0.1:8000/v1',
    apiKey: 'EMPTY',
    model: 'Qwen/Qwen3.5-9B',
    dialect: 'vllm-sglang',
    uiProtocol: 'computer-use',
  }),
  environment: browser,
  actionProviders: [browser],
  humanTakeoverProvider: new VisibleBrowserTakeoverProvider(browser),
  traceStore: new FileTraceStore('/absolute/path/to/sightrig-data'),
  policy: { maxBatchSize: 1 },
});

const session = await runner.createSession({
  goal: 'Fill and submit the form',
  startUrl: 'https://example.com',
  modelProfileId: 'local-vision-model',
  thinking: false,
});

const events = (async () => {
  for await (const event of session.events()) {
    console.log(event.seq, event.type);
  }
})();

await session.start();
await session.waitForCompletion();
await events;

The public contracts stay small:

interface ModelAdapter {
  decide(context: AgentContext, signal?: AbortSignal): Promise<Decision>;
}

interface AgentEnvironment {
  acquire(signal?: AbortSignal): Promise<void>;
  reset(input: ResetInput, signal?: AbortSignal): Promise<Observation>;
  observe(signal?: AbortSignal): Promise<Observation>;
  release(): Promise<void>;
}

interface ActionProvider {
  definitions(): ToolDefinition[];
  supports(action: CanonicalAction): boolean;
  execute(action: CanonicalAction, context: ActionContext): Promise<ActionResult>;
}

Runtime support

Target	Provider	Status	Notes
Browser	Local Playwright Chromium	Supported	Isolated context per conversation; visible or headless
Browser	AgentBay over CDP	Supported	Remote sandbox lifecycle remains outside the Agent Loop
Mobile	AgentBay Android	Supported	Tap, swipe, text input, key events, screenshot observations
Desktop	Custom provider	Extension point	Implement AgentEnvironment and ActionProvider
CLI / API	Custom action provider	Protocol only	No shell or arbitrary API executor is included

Provider selection is part of session input:

await runner.createSession({
  goal: 'Open the product page',
  startUrl: 'about:blank',
  modelProfileId: 'vision-model',
  target: { kind: 'browser', provider: 'agentbay', headless: true },
});

await runner.createSession({
  goal: 'Open Android Settings',
  modelProfileId: 'vision-model',
  target: { kind: 'mobile', provider: 'agentbay' },
});

AgentBay keys and image IDs are configured from Settings → Browser/Mobile. Keys are write-only and stored through SecretStore; image IDs are deployment-specific and are not included in the repository. Environment variables remain available for CI and headless hosts—see .env.example.

Model support

The base transport in @sightrig/model supports OpenAI-compatible Chat Completions with image content and function tools. Model-family behavior lives in extensions.

Extension	Purpose
@sightrig/model/qwen	Canonical tools and native Qwen computer_use; generic, vLLM/SGLang, and DashScope dialects
@sightrig/model/grounding	Strict coordinate-only grounding with optional image preprocessing and zoom-in
Custom ModelAdapter	Any model or protocol that can produce a canonical Decision

The loop stores a short displayable reasoning summary, not a hidden chain of thought. Transport failures and invalid model decisions are recorded separately from GUI execution failures, making model quality and harness reliability independently diagnosable.

Trace, human handoff, and replay

Visual trace

SightRig writes append-only NDJSON events and content-addressed artifacts. The Web Console can reconnect through SSE, move through the timeline, and render click, drag, scroll, and guard overlays against the exact historical frame. Trace replay is time travel only—it never re-executes an action.

Human in the loop

When a model requests help, the run enters waiting_human without releasing its browser or mobile environment. Visible local Chromium supports direct takeover. Headless and remote runtimes can expose canonical human actions through the Web relay. The host then resumes, declines, or explicitly fails the intervention.

Recorded flows

A successful or user-confirmed run can be compiled into a fixed browser flow:

import { compileRecordedFlow, createRecordedFlowRun } from '@sightrig/core';

const flow = compileRecordedFlow({
  metadata: session.metadata,
  events: await traceStore.readEvents(session.id),
});

const replay = await createRecordedFlowRun({
  flow,
  environment: replayBrowser,
  actionProviders: [replayBrowser],
  traceStore,
});

await replay.start();
await replay.waitForCompletion();

Replay resolves semantic element identity first and only uses recorded coordinates as a controlled fallback in resilient mode. It creates a new audit trace and makes no model request. If the final state has no deterministic postcondition, the replay ends as unverified rather than claiming success.

Workspace

SightRig/
├── packages/
│   ├── protocol/          Canonical actions, observations, events, Zod schemas
│   ├── core/              Agent Loop, state machine, guards, flow compiler
│   ├── model/             OpenAI-compatible transport and model extensions
│   ├── browser/           Local and CDP-backed Playwright environment
│   ├── browser-agentbay/  AgentBay browser runtime connector
│   ├── mobile/            Provider-neutral mobile environment
│   ├── mobile-agentbay/   AgentBay Android connector
│   ├── trace/             NDJSON, artifacts, recovery, flow storage
│   └── secret-store/      Keychain and environment secret providers
├── apps/
│   ├── daemon/            Fastify REST, SSE, runtime settings
│   └── web/               React visual console
├── examples/browser-basic/
├── fixtures/test-site/
└── tests/

Development

pnpm build       # packages, evaluation apps, daemon, and Web Console
pnpm typecheck   # strict TypeScript checks across the workspace
pnpm test        # unit tests plus deterministic Chromium end-to-end tests

The repository also includes optional harnesses for ScreenSpot-Pro grounding evaluation and OpenAI-compatible endpoint performance testing. Datasets, predictions, profiles, screenshots, and credentials are deliberately excluded from version control.

Security and guarantees

• The daemon listens on 127.0.0.1 and rejects unknown browser origins.
• API keys are never returned by the daemon or written to logs, traces, or browser storage.
• Screenshots and model responses can still contain sensitive page content; trace storage is local and is not automatically deleted.
• cli_command and api_call are reserved protocol actions. SightRig ships no shell or arbitrary API executor.
• A model-reported completion without a verifier ends as unverified, not succeeded.
• Session deletion requires an exact ID confirmation and removes unreferenced artifacts.

Report vulnerabilities according to SECURITY.md, not through a public issue.

Documentation

• Architecture and extension boundaries
• Trace event and artifact format
• CLI provider extension contract
• ScreenSpot-Pro evaluation
• Model deployment performance evaluation
• Contributing

License

SightRig is released under the MIT License.