Skip to content

Inspect native pixels and report target availability - #71

Merged
iamnbutler merged 1 commit into
feat/github-cachefrom
codex/desktop-availability
Oct 5, 2026
Merged

iamnbutler merged 1 commit into
feat/github-cachefrom
codex/desktop-availability

Conversation

@iamnbutler

@iamnbutler iamnbutler commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Failed native window inspections retain their original error and add a bounded, later read of target availability. Explicit desktop_inspect pixels mode returns a screenshot with exact target and image-digest checks, without a reusable action snapshot. Failed reinspection also preserves an already completed action's outcome.

Relates to #64 and #8. These diagnostics do not establish or fix the cause of intermittent offscreen or incomplete Accessibility observations.

This focused PR is based on merged application-state and selection/shortcut changes (9418c8e); it no longer depends on #67. Availability diagnostics retain the new is_active_known flag.

Validation: type checks, formatting/lint, the independent signed desktop build, deep/strict signature verification, diff checks, and independent source review pass. Fresh native calls against that signed candidate passed exact app/window generation, Accessibility and pixel inspection, absent-window original-error diagnostics, unknown-state flags, and invalid-mode rejection. Pixel results have a screenshot but no action snapshot, element IDs, or detection metadata. Ace's initial webview AX tree was sparse; this does not claim that separate issue is fixed.

A real AppKit Close Alpha button ran exactly once and closed its target while another fixture window remained open. The returned action retained its original completed outcome/native receipt and separately reported the failed follow-up inspection. Callback evidence confirmed the actual close; no input was replayed. The initial verifier incorrectly expected the closed window to disappear from inventory; macOS retained an offscreen pixels-only row. Rechecking the saved result against the correct contract passed without repeating the click. Existing pi result/history transport is unchanged; broader visual workflows remain tracked in #8.

A real pi/Anthropic run using this PR's desktop tool schema and host adapter passed exactly three read-only inspections: pixels, Accessibility, and an absent window. The stored tool results retained the native text, screenshots, and error flags exactly. Reopening the same pi SQLite store returned identical results and images with unchanged model usage and no repeated native calls. Evidence: /tmp/ace-availability-model-odVTLq/result.json.

Built in Ace

Ace's terminal ran installation, builds, checks, Git/GitHub operations, and prior native validation. Codex coordinated and read/edited source while Ace's external-harness support (#9) and file editor (#22) remain unfinished. Native checks ran through Ace’s terminal against the isolated signed app. External CUA was used only to quit the prior validation build and launch the disposable AppKit fixture while Ace’s application-lifecycle support remains incomplete under #8.

@iamnbutler
iamnbutler force-pushed the codex/desktop-availability branch from 3f47a50 to a99b284 Compare October 5, 2026 12:07
@iamnbutler
iamnbutler changed the base branch from codex/desktop-keyboard to feat/github-cache October 5, 2026 12:07
@iamnbutler
iamnbutler marked this pull request as ready for review October 5, 2026 12:11
@iamnbutler
iamnbutler merged commit 61c5aa9 into feat/github-cache Oct 5, 2026
@iamnbutler

Copy link
Copy Markdown
Contributor Author

The real pi/Anthropic inspection smoke passed: pixels, Accessibility, and absent-window diagnostics. Exact tool text, images, and error flags survived reopening pi's store with unchanged model usage and no repeated native calls. Evidence: /tmp/ace-availability-model-odVTLq/result.json. Validation details are recorded in the PR body and #8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant