Repository navigation
feat(desktop): add targeted native actions - #63
Merged
Merged
Conversation
iamnbutler
marked this pull request as ready for review
October 5, 2026 02:47
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ace's native tools can inspect windows but cannot yet complete a click-and-type workflow. This adds
desktop_click,desktop_type, anddesktop_keyto pi for the first mutating slice of #8.Actions use single-use observations bound to the application instance, exact window, and observed control. Text replaces a field's entire value; keys are limited to Return, Tab, Escape, deletion, and arrows. Successful actions return fresh bounded Accessibility/image evidence when available. Outcomes distinguish completed, refused, and unknown delivery, and pi never automatically replays input after interruption. Hosted workspace calls also have generation-specific IDs so a late reply cannot settle a new runtime's action.
The signed embedded bridge owns input and macOS permissions. Ace's existing collaborator-agent switch still controls every tool together. Settings now report Keyboard input permission. A pinned Peekaboo source patch moves semantic AXPress off MainActor while retaining operation ownership, and prevents an ambiguous native error from falling back to another click (#61).
Validation completed:
The final hosted workflow also passed through a real pi/Anthropic run against a signed disposable AppKit form: click, replace, key, and button activation changed the visible result and persisted the intended value. Restarting the local Workers/Durable Objects runtime without a workspace replayed all ten completed/refused/unknown results with identical text, errors, and images. Hosted validation used the local Workers runtime, not a second physical machine.
During validation, Ace-dev's test window became unavailable to native inventory/Accessibility despite remaining readable through Codex's controlled view. That separate availability/diagnostics problem is tracked in #64; the successful hosted form run does not establish recovery from it. Wider application/key coverage remains follow-up work in #8.
Fixes #61.
Built in Ace
Builds, checks, runtime scripts, Git/GitHub work, and diff retrieval use Ace's terminal and channel APIs. Codex and its subagents edited and reviewed source and prepared smoke scripts; external harness integration is tracked in #9 and Ace's file editor in #22. Codex computer use prepared the disposable desktop target because the missing native action surface is the work in #8. Tests used a separate Ace profile/project and a disposable signed AppKit form, preserving existing channels. Dogfooding found the button completion problem recorded in #61.