Repository navigation
Agent task coverage audit #75
Description
How well desktop-tools covers everyday agent tasks on a Mac, mapped from docs/tools.md, docs/recipes.md, src/tools.ts and the native client.
Refreshed 2026-10-17 (second pass) against main at aefabc2 (0.11.0 with #107 and #109), plus open #110 (web accessibility for desktop_wait) and #111 (incomplete-read retry). The first refresh was at 6783905 (0.8.0); the original audit was at acae444. Rows changed since then say which PR changed them. None of this was run live: the screen is locked, and the live checks for each PR are in #42. Ratings come from the code, and the ones marked (unverified) depend on how an app or the OS exposes its UI.
- Gap: there is no reasonable tool sequence.
- Awkward: it works, but takes many calls, has an unverifiable result, or relies on a fragile workaround.
- Works: there is a direct sequence.
Common shorthand: inspect = desktop_inspect (it returns the snapshot that input uses). Each completed action returns a fresh inspection, so chains don't need a separate inspect.
Counts: 5 gaps (10 originally), 11 awkward (13 at the first refresh, 16 originally), 29 works (26 at the first refresh, 18 originally).
Gaps (5)
| # | Task | Tool sequence | Why it's a gap |
|---|---|---|---|
| 2 | Enter a password: a website login, joining Wi-Fi, or an admin prompt (SecurityAgent) | inspect → desktop_type / desktop_insert / desktop_dialog_answer field |
All three refuse secure fields, and so does #82's web typing. Left for the user's decision. |
| 3 | Install an app from a DMG when it replaces an existing app, or change a locked System Settings pane | desktop_open dmg → inspect → desktop_drag app → Applications |
The copy works, but admin authentication hits gap 2. Touch ID can't be done at all. |
| 4 | Connect to a secured Wi-Fi network | desktop_menubar_click Wi-Fi → inspect → desktop_click network → password |
Opening the list and choosing the network work, and an open network works. The password field hits gap 2. |
| 8 | Move a window to another Space | — | Documented as not supported: since macOS 14.5 only the Dock can do it. The Mission Control drag workaround is always unknown. |
| 10 | Select non-editable text in a PDF (to copy or highlight) | inspect → desktop_drag |
Web pages are done (row g10 under Works). A PDF view has no selected text range, and #88's WebKit text markers apply to web areas only, so a PDF needs a drag-select, which is unverifiable. desktop_read covers copying. |
Works but awkward (11)
| # | Task | Tool sequence | Why awkward |
|---|---|---|---|
| 2 | Use Spotlight to open something | desktop_system_key Cmd-Space with expect window_appears → inspect → desktop_insert → desktop_key enter |
Many steps. The system key now completes when Spotlight's window appears (expect, #89), but is still unknown otherwise. #84 has a recipe. desktop_open / desktop_launch is usually better. |
| 4 | Print to PDF | desktop_menu File > Print… → … → desktop_dialog_answer |
About 6 supported steps. #84 has a recipe, including the Export as PDF shortcut. |
| 7 | Drag a file into a web upload area | Finder inspect + browser inspect → desktop_drag with to_snapshot |
Cross-app drags are foreground and unknown unless an expect condition (#89), such as the file name appearing, holds afterwards. Prefer the upload button and desktop_dialog_answer path. |
| 8 | Install an app from a DMG (no replace) | desktop_open dmg → desktop_wait → inspect → desktop_drag → desktop_dialog Gatekeeper |
Many steps with waits. #84 has a recipe. |
| 9 | Use the share sheet | desktop_menu File > Share > , or the toolbar button → popover |
The menu path works. The popover may live in another process (unverified). #84 has a recipe. |
| 11 | Pick a color in the color panel | desktop_menu Format > Show Colors → … → desktop_type hex → desktop_key enter |
Many steps, and the panel layout varies (unverified). #84 has a recipe. |
| 12 | Choose a font | desktop_menu Format > Font > Show Fonts → … |
The lists are long. #86's table read can list them. #84 has a recipe. |
| 14 | Copy files between Finder folders | desktop_clipboard_write files → inspect target → desktop_key Cmd-V |
Verifiable but indirect. #99's desktop_insert format: "files" pastes into text areas only; a Finder folder paste confirmed by the copied names is a #3 follow-up. #84 has a recipe. Shell cp is simpler. |
| 15 | Respond to a notification | desktop_notifications → desktop_notification_action |
Built in #74 against macOS 11–15. Pending a live check on 27 (#42). |
| 16 | Use a context menu on a Dock item | desktop_dock right-click + item |
Moves the real pointer. |
| 7g | Zoom or rotate in Maps, Photos or Preview (was gap 7) | desktop_menus → desktop_menu View > Zoom In / Rotate, or desktop_key Cmd-= / Cmd-- (with repeat) |
Use the menu or keys (#87 recipe). There is no pinch or rotate gesture by decision: undocumented events, GPL prior art, and no Chrome support. Canvas views may not report the zoom level. |
Works (29)
| # | Task | Tool sequence |
|---|---|---|
| 1 | Rename a file in Finder | inspect → desktop_click file → desktop_key enter → desktop_type → desktop_key enter |
| 2 | Attach a file in Mail | desktop_menu File > Attach Files… → desktop_dialog_answer path → button Choose File |
| 3 | Change a System Settings toggle | desktop_open settings URL → desktop_wait element_appears → desktop_set_value with readback |
| 4 | Accept a cookie banner | inspect (with scope if truncated) → desktop_click Accept |
| 5 | Use a context menu | desktop_click kind right or AXShowMenu → AXPick |
| 6 | Scroll a list until an item is visible | AXScrollToVisible, or desktop_scroll + desktop_wait element_appears |
| 7 | Switch tabs in Safari or Terminal | desktop_click tab, desktop_menu, or desktop_key Cmd-Shift-] |
| 8 | Answer a permission prompt (button) | desktop_dialog → desktop_dialog_answer button |
| 9 | Choose from a popup button or web <select> |
desktop_click → AXPick |
| 10 | Handle a modal sheet or alert | desktop_wait dialog_appears → desktop_dialog → desktop_dialog_answer |
| 11 | Search the web | desktop_open URL → desktop_wait text_contains → desktop_read. In Chrome and other Chromium browsers, the page's text appears once web accessibility is on (#107 for inspect and read, #110 for the wait). |
| 12 | Read a long article or mail thread | desktop_read |
| 13 | Save a document to a given path | desktop_menu File > Save… → desktop_dialog_answer path → Save |
| 14 | Quit an app with unsaved changes | desktop_quit → desktop_dialog → desktop_dialog_answer button |
| 15 | Copy and paste between apps | Cmd-C → desktop_clipboard_read, or desktop_clipboard_write → desktop_insert |
| 16 | Run a Terminal command and read its output | desktop_insert → Enter → desktop_wait text_changes → desktop_read |
| 17 | Arrange windows: display, size, full screen | desktop_spaces → desktop_move / desktop_resize / desktop_maximize, and desktop_fullscreen with an explicit enabled (#78) |
| 18 | Reorder items by dragging within one window | desktop_drag (background, exact window) |
| g1 | Select several files or messages (Cmd-click, Shift-click) (was gap 1) | desktop_click with modifiers command/shift/option, through the foreground modifier route (#77). The result is unknown unless an expect condition holds afterwards (#89); otherwise check the selection by inspecting. |
| g5 | Set a date or time in a date picker (was gap 5) | desktop_set_value with an ISO 8601 string on AXDateTimeArea/AXDateField/AXTimeField, read back. The keyboard route is given when a picker can't be set (#79). Web <input type=date> is (unverified). |
| g6 | Take a screenshot to a file (was gap 6) | desktop_screen or desktop_inspect with save_path, region and overwrite, writing the full-resolution PNG (#81) |
| g9 | Volume, brightness or play/pause by key (was gap 9) | desktop_system_key volume_up, mute, play_pause, brightness_up… The result is unknown (#80), and these keys have no observable condition for expect. For a set level, use the Control Center slider. |
| a1 | Keyboard-only navigation (was awkward 1) | desktop_key with keys (up to 16 chords) or repeat (up to 32) against one snapshot, focus re-checked before each key (#77) |
| a3 | Fill a web form's text fields (was awkward 3) | desktop_click the field → desktop_type, which types with key events so page scripts see input, then reads the field back (#82, recipe in #84). In Chromium and Electron apps (Chrome, Slack, VS Code), the fields are listed once #107 turns on the app's web accessibility at the first inspect. |
| a5 | Read a table: Numbers, Finder list, web table (was awkward 5) | desktop_read with table: true and a scope → columns, rows of cells, outline depth, and row IDs for desktop_click (#86) |
| a13 | Adjust volume or brightness to a level (was awkward 13) | desktop_menubar_click Control Center → desktop_set_value slider with readback (recipe in #84), or the keys in g9 |
| a6 | Edit a spreadsheet or table cell (was awkward 6) | desktop_read with table: true → desktop_set_cell by row and column, confirmed by reading the table again (#92). Whether Numbers and Excel cells expose a control is (unverified), listed in #42; a cell without one is refused with CELL_NOT_EDITABLE. |
| a10 | Switch the input source (was awkward 10) | desktop_input_source with select, read back through the host, covered by pause and the activity list (#90, #101) |
| g10 | Select non-editable text on a web page (was part of gap 10) | desktop_select on an element inside a web area, through WebKit text markers, read back (#88). Chromium pages are (unverified). |
Top remaining gaps and awkward items
Password entry (gap 2, and gaps 3 and 4 behind it) is left for the user's decision. The first refresh's top three are done: table cells (#92), input source by ID (#90, #101) and expect for foreground input (#89).
- Run the pending live checks in Run the live checks deferred while the screen was locked #42. None of these ratings were run live, and most rows since 0.5.0 rest on code alone.
npm run live-check(chore: run the automatable #42 live checks with npm run live-check #102) automates part of them. - Background paste into web content (WebKit: background Command-V, and Safari page fields missing from Accessibility #5 part a). It affects a3 and Ace's composer when the app isn't frontmost. A live spike, listed in Run the live checks deferred while the screen was locked #42, must answer its three questions before any patch.
- Paste files into a Finder folder (awkward 14, a Add temporary clipboard paste and lifecycle validation #3 follow-up). Confirm the paste by the copied names appearing in the folder, so the copy is
completedrather than indirect. - PDF text selection (gap 10). First find out what Preview's PDF view exposes to Accessibility.
Activity
iamnbutler commentedon Oct 17, 2026
Refreshed against main at 6783905 (0.8.0), plus #86 (queued) and #87. The body is updated row by row. Nothing was run live, because the screen is locked; the live checks are in #42.
Counts: 5 gaps (was 10), 13 awkward (was 16), 26 works (was 18).
What moved:
desktop_fullscreen(feat: enter or exit native full screen for one window #78).systemstate, and docs: recipes for tasks that take several tool calls #84 has recipes for awkward 4, 8, 9, 11, 12 and 14.Password entry (gap 2, and gaps 3 and 4 behind it) is left for the user's decision. Text selection (gap 10) is in progress with f7.
Top 3 remaining:
desktop_typeanddesktop_set_valuetarget a cell from a feat: read a table, outline or grid as columns and rows with each row's element ID #86 table read, then read the cell back through the table, so the result iscompletedorunknown.desktop_input_source, which uses the publicTISSelectInputSourceand reads the source back, so the result iscompletedinstead of a control-space press whose result isunknown.unknowntake adesktop_wait-styleexpectcondition. They reportcompletedonly when it holds within a short timeout, and the input is never repeated.iamnbutler commentedon Oct 17, 2026
Closing the audit. Of its five gaps, passwords and the two gaps they blocked are #117. PDF text selection is #130. Moving a window to another Space isn't supported, by design. The awkward items all have recipes, and Finder folder paste shipped in #114.