[iamnbutler]

Adds desktop_hover and desktop_press.

desktop_hover

Moves the real pointer over an element (its observed center) or a normalized screenshot point, so tooltips, hover menus and hover states appear. Tracking areas follow the physical pointer, so background delivery can't do this; this is the approved foreground exception, and the tool description and docs say so.

• Before moving, the client checks with CGWindowList, front to back, that the snapshot's window is still on screen at its snapshot bounds and frontmost at the point. It skips the host's own overlay windows (by the handshake's host pid) and fully transparent windows. If another window covers the point, it refuses without moving anything.
• It takes the snapshot lease, then calls moveMouseWithOutcome(to:duration: 250, steps: 15, profile: .linear).
• Afterward it checks that the pointer is within 1 pt of the point and the window is still frontmost there. If both hold the result is completed with pointer_moved: true; otherwise it is unknown with DESKTOP_ACTION_UNVERIFIED.
• A completed hover returns a fresh observation, like other actions.
• Disabled elements can be hovered, since they still show tooltips.

desktop_press

Presses and holds an element or point for duration_ms (1–10000, default 1000) with the left or right button, then releases in place. It uses the Bridge's exact-window held-pointer lifecycle, which is background delivery and doesn't move the pointer:

1. Take the lease.
2. createExactWindowHeldPointerOwner.
3. beginExactWindowPointerHold (expiry = duration + 2 s).
4. Sleep for the duration.
5. releaseExactWindowPointerHold.
6. disconnectExactWindowHeldPointerOwner. This always runs, including on errors, and releases any button still down.
7. Finish the lease.

The result reports release (the termination reason). Anything other than released (for example windowChanged or expired) is unknown. The host also releases at expiry if the client dies.

The runtime's allowedOperations gains .moveMouse and the five held-pointer operations.

Overlay

• Hover: upstream moveMouseWithOutcome already sends showMouseMovement, so the agent cursor glides to the point just before the real pointer moves there.
• Press: the new peekaboo-press.patch (end of the stack) sends long-press feedback before a hold begins, and a single click at the receipt point after a release, revoke or disconnect ends a hold. AgentCursor now:
  • keeps the cursor squeezed for a long press (at most 30 s), then lifts it with a ripple on the release feedback;
  • holds off the idle fade while a press is held;
  • arrives at once when it is already at the point, so the release isn't delayed by travel.

Tests

• npm run check
• Signed native build on the full stack (node dist/app.js … --identity "Developer ID Application: Nathan Butler (8S43JJMN6B)").
• PeekabooAutomationKit: UIAutomationServiceAgentCursorTests (new test: a hold shows a long press then a single click at the same point, and a nil termination shows nothing) and ExactWindowHeldPointer*.
• Live, dev app (DESKTOP_TOOLS_STATE=/tmp/cursor-state): a hover or press with a bad snapshot is refused before dispatch. Out-of-range duration_ms and missing targets are refused in TypeScript.

Pending: screen locked

The GUI session was locked (IOConsoleLocked), so desktop_inspect can't capture and no snapshot-based live check could run. Exact checks to run in the morning with the dev app (open -n --env DESKTOP_TOOLS_STATE=/tmp/<x> "Desktop Tools.app") and TextEdit or Safari:

1. Hover with a tooltip: desktop_hover on a toolbar button element (for example a Safari toolbar button). Expect completed, pointer_moved: true, and the real pointer at the element's center. Inspect again after about 1 s and expect the tooltip window.
2. Hover on a covered point: put another window over part of the target and hover a covered point. Expect refused with the "Another window covers this point" message, and the pointer doesn't move.
3. Hover after the window moved: move the window after inspecting, then hover with the old snapshot. Expect refused.
4. Hover overlay: the agent cursor glides to the point, then the real pointer follows it there. The agent cursor never appears in desktop_inspect or desktop_screen images.
5. Long press: desktop_press with duration_ms: 1500 on Safari's Back button, after some history. Expect completed, release: "released", and the history menu in the follow-up observation. Then press Escape or click elsewhere.
6. Press overlay: the cursor squeezes and stays down for the whole 1.5 s, without fading. It lifts with a ripple at release, with no extra delay at release.
7. Right button: desktop_press with button: "right" on a text view. Expect the context menu. Check that the right button isn't left down afterward: a later left click works normally.
8. Window changes mid-press: start a 5 s press, then close or move the window. Expect unknown, release: "windowChanged", and no stuck button.
9. Client killed mid-press: start a 5 s press and SIGKILL the node or client process. Expect the button to be up within about 7 s (duration + 2 s expiry, or sooner on disconnect), and the overlay cursor to lift.
10. Highlight: hover and press outline the target window.