Skip to content

Embed native desktop inspection in Ace - #47

Merged
iamnbutler merged 3 commits into
feat/github-cachefrom
codex/peekaboo
Oct 5, 2026
Merged

iamnbutler merged 3 commits into
feat/github-cachefrom
codex/peekaboo

Conversation

@iamnbutler

@iamnbutler iamnbutler commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Adds the first inspection slice for #8: pi agents can list native applications and windows, then inspect an explicit window with Accessibility text and a screenshot. The result appears in expandable tool output and survives reopening the channel through pi's existing history. Hosted channels forward inspection to their workspace. The existing collaborator-agent switch governs these tools with all other agent tools.

Peekaboo 4.8.0 is embedded in the signed Ace desktop app. Users grant Accessibility and Screen Recording to Ace in Settings → This Mac; no separate Peekaboo installation is required. Native inspection requires macOS 15 and Ace to remain open on the execution host. This slice adds observation only; clicks and typing remain follow-up work under #8. Bumps the desktop version to 0.0.10 for the next canary release.

Dogfooding exposed a timeout while Ace's synchronous folder picker was open. The picker now opens asynchronously, keeping native inspection responsive and preserving cancellation, exact folder paths, and the project-first flow.

The README, terms, and architecture now state the intended trust and deployment model: the tailnet is the sole team/access authority, teams operate their own installations and shared services, and Ace will not run a hosted service.

Validation:

  • All package type checks, formatting/lint, app build, and the signed development desktop build pass. Verified nested signatures, client identity, macOS 15 targets, dependency linkage, pinned Swift dependencies, and packaged licenses.
  • Real pi/Anthropic runs called all three desktop tools through both a local host and the local Workers/Durable Objects runtime. The collaborator switch rejected invocation when off and allowed it when on.
  • Native captures returned bounded screenshots and Accessibility text with exact-window receipts. Inspected the real folder picker; checked cancel, a path containing a comma, and opening a project without creating a channel.
  • Local host and Workers restarts replayed identical image bytes and Accessibility text. Reopened the channel in Ace and visually checked the saved image.
  • Workspace-offline errors, in-flight cancellation, capture-file cleanup, and quitting with the picker open passed. Quitting the app removed its bridge socket while the independent host stayed running.
  • React Doctor: 92/100, with one maintainability warning in the existing HostStatus function. Independent review found no outstanding blockers.

Built in Ace

Builds, checks, runtime commands, Git operations, and GitHub operations ran through Ace's terminal. Codex provided orchestration, source edits/reads, and computer use because the corresponding Ace capabilities are still being developed: #8 (native UI), #9 (external harness integration), and #22 (file editing). This PR builds the inspection part of the native UI gap while retaining pi as the native harness.

@iamnbutler iamnbutler changed the title feat(desktop): embed native window inspection in Ace Embed native desktop inspection in Ace Oct 5, 2026
@iamnbutler
iamnbutler marked this pull request as ready for review October 5, 2026 00:42
@iamnbutler
iamnbutler merged commit ce1bd9b into feat/github-cache Oct 5, 2026
@iamnbutler

Copy link
Copy Markdown
Contributor Author

Published canary 0.0.10 from merge commit ce1bd9bde77955b15521a5cbeee1236bdc14fa13 in release run 37249000711.

CI completed type/lint checks, signing, notarization, archive verification, and feed publication. The downloaded artifact matched the release manifest and requested source revision. Both the app and disk image passed local Gatekeeper and stapled-ticket checks; the published feed matches CI's signed feed and requires macOS 15.

The local Sparkle upgrade from 0.0.8 to 0.0.10 also completed: Ace Helper restarted from the installed bundle, the update/resume receipts cleared, and the existing project, channels, and selected chat history reopened. The installed app and bundled native client's signatures passed verification.

DMG SHA-256: e708ee9dedbff6f6b2f4515126a8c1c12d014ddd47418346668ab1efa1bf3640.

The terminal input glitch observed during release dogfooding is tracked separately in #48 under #5.

@iamnbutler iamnbutler mentioned this pull request Oct 5, 2026
10 of 56 tasks
iamnbutler added a commit that referenced this pull request Oct 5, 2026
* feat(app): cache GitHub pages and share the project sidebar

* chore(desktop): prepare canary 0.0.9

* chore(desktop): record published canary 0.0.9

* chore: move the work tracker to GitHub issues

* feat(channel): add names and rolling summaries

Refs #23. Show full rolling summaries in the details sidebar, with brief dashboard and channel previews. Preserve existing channel history and lane prefixes.

* Embed native desktop inspection in Ace (#47)

* feat(desktop): embed native window inspection in Ace

* fix(desktop): keep inspection responsive during folder selection

* chore(desktop): prepare canary 0.0.10

* fix(host): keep replayed terminal queries out of the shell's input
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant