Skip to content

feat(desktop): add targeted native actions - #63

Merged
iamnbutler merged 2 commits into
feat/github-cachefrom
codex/desktop-actions
Oct 5, 2026
Merged

iamnbutler merged 2 commits into
feat/github-cachefrom
codex/desktop-actions

Conversation

@iamnbutler

@iamnbutler iamnbutler commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Ace's native tools can inspect windows but cannot yet complete a click-and-type workflow. This adds desktop_click, desktop_type, and desktop_key to pi for the first mutating slice of #8.

Actions use single-use observations bound to the application instance, exact window, and observed control. Text replaces a field's entire value; keys are limited to Return, Tab, Escape, deletion, and arrows. Successful actions return fresh bounded Accessibility/image evidence when available. Outcomes distinguish completed, refused, and unknown delivery, and pi never automatically replays input after interruption. Hosted workspace calls also have generation-specific IDs so a late reply cannot settle a new runtime's action.

The signed embedded bridge owns input and macOS permissions. Ace's existing collaborator-agent switch still controls every tool together. Settings now report Keyboard input permission. A pinned Peekaboo source patch moves semantic AXPress off MainActor while retaining operation ownership, and prevents an ambiguous native error from falling back to another click (#61).

Validation completed:

  • Type checks, formatting/lint, React Doctor (100/100), signed macOS build and signature verification, and workspace reply correlation across runtime generations.
  • Real signed desktop field replacement, arrow keys, and a Rename button press that closed the dialog and persisted the new channel name. Foreign observations and nonliteral element IDs were refused; simultaneous reuse of one snapshot delivered one action and refused the duplicate.
  • A real pi/Anthropic run inspected the window, clicked and edited the field, moved the caret, pressed Rename, and verified the result. Exact text, errors, and image results replayed from persisted history after worker restart.
  • Stop and worker-restart cases each withheld a real native response after the input had completed. Both preserved an unknown interrupted result and delivered the action exactly once; the restarted model reinspected without repeating it.
  • The local Workers/Durable Objects path passed collaborator-switch rejection, offline refusal before native dispatch, and workspace loss after launching the native request (unknown delivery, no retry, drained handler and cleaned temporary files). This disconnect test does not establish that its AX mutation landed.

The final hosted workflow also passed through a real pi/Anthropic run against a signed disposable AppKit form: click, replace, key, and button activation changed the visible result and persisted the intended value. Restarting the local Workers/Durable Objects runtime without a workspace replayed all ten completed/refused/unknown results with identical text, errors, and images. Hosted validation used the local Workers runtime, not a second physical machine.

During validation, Ace-dev's test window became unavailable to native inventory/Accessibility despite remaining readable through Codex's controlled view. That separate availability/diagnostics problem is tracked in #64; the successful hosted form run does not establish recovery from it. Wider application/key coverage remains follow-up work in #8.

Fixes #61.

Built in Ace

Builds, checks, runtime scripts, Git/GitHub work, and diff retrieval use Ace's terminal and channel APIs. Codex and its subagents edited and reviewed source and prepared smoke scripts; external harness integration is tracked in #9 and Ace's file editor in #22. Codex computer use prepared the disposable desktop target because the missing native action surface is the work in #8. Tests used a separate Ace profile/project and a disposable signed AppKit form, preserving existing channels. Dogfooding found the button completion problem recorded in #61.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant