Repository navigation
Add targeted click variants and screenshot points - #82
Merged
Merged
Conversation
iamnbutler
marked this pull request as ready for review
October 5, 2026 12:27
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend
desktop_clickwith single, double, right, middle, and triple clicks on either an observed element or a normalized screenshot point. Coordinates use the snapshot's native capture geometry, preserving exact process/window checks, single-use snapshots, background delivery, and unsafe replay semantics.This is the click-only portion of #68 and #8. Scrolling and dragging remain separate work; this does not close #68.
Validation: repository type checks and formatting/lint pass, as do the signed desktop build, strict signature verification, Ace diff review, and independent source review. A real signed native fixture using the shared click implementation passed six groups: the existing AX click, a coordinate single click from a resized screenshot, and double/right/middle/triple button/count checks. That broader run later stopped on an uncertain scroll result; no scroll or drag pass is claimed here. A real pi/Anthropic run using this exact focused candidate’s tool schema, host adapter, signed client and GUI then inspected the fixture, selected a screenshot coordinate inside Activate, clicked once, and reinspected. The real button counter increased exactly once while the scroll position and prior board events stayed unchanged. Exact results/screenshots survived closing and reopening its private pi SQLite store with unchanged usage and no repeated input. Worker-crash and hosted end-to-end coverage remain in #8; this check does not claim them.
Built in Ace
Used Ace's terminal for dependencies, checks, the signed build, Git, and GitHub, and its changes/patch surface for the diff. Codex source tools performed editing and independent review because the external agent/editor integration is tracked in #9 and #22. Native validation was coordinated through Ace's real host adapter and signed desktop bridge against a disposable AppKit fixture. External CUA handled validation-app setup/quit while Ace’s lifecycle tools remain incomplete under #8.