Skip to content

feat: hold modifier keys during clicks, and press key sequences against one snapshot - #77

Merged
iamnbutler merged 1 commit into
mainfrom
feat/modifier-keys
Oct 17, 2026
Merged

iamnbutler merged 1 commit into
mainfrom
feat/modifier-keys

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

Covers gaps 1 and 4 from #75.

Modifier clicks (desktop_click modifiers)

  • Route. modifiers takes any of command, option and shift, for single, double or triple clicks. Instead of adding a new route, this uses upstream Peekaboo's existing foregroundModifierClick Bridge operation, now allowed in the runtime. The host:
    • leases the snapshot itself, at the execution leaf
    • checks the exact window and the coordinate authority
    • activates the app
    • moves the real pointer and holds the keys with real key-down and key-up events around a real click
    • waits until the session has seen mouse-up and every modifier key-up
    • then restores the pointer and the previously focused app and window
      The result reports cursor_restoration and focus_restoration.
  • Element clicks with modifiers take this pointer route at the element's observed center, because AXPress ignores modifiers. The docs and tool description say so.
  • Always unknown once dispatched, with a message naming the held keys. Nothing reads back the selection.
  • Control is refused, since upstream refuses Control-click and right-click restoration. The message says a Control-click is kind: "right". Right and middle clicks with modifiers are also refused.
  • Scroll is not included: Peekaboo has no modifier scroll route, so adding one would need a new Bridge operation.

Key sequences (desktop_key keys / repeat)

  • Forms. keys takes 1 to 16 chords, each with its own modifiers. repeat takes 1 to 32 presses of one key. Either way, the whole sequence runs against one snapshot under one lease.
  • Focus checks. Every key of a sequence uses the window-focus route (expectedWindowIdentity and bounds), since focus moves as Tab runs. Before each event, the host checks that the app's focused element is still in this window and that it is still the app's key window. There is a 30 ms gap between keys.
  • Stopping. The sequence stops at the first refused key, and the result reports keys_sent and keys_requested.
    • A sequence that stopped partway is unknown.
    • A refused first key is refused.
    • An indeterminate failure partway is unknown, with keys_sent.
  • Validation. Each chord is validated like a single key press, including the Command-W/Q/H/M refusal. A single key without repeat behaves exactly as before.

Checks

  • npm run check and npm test pass (11, including a new test/input.test.mjs covering the validation refusals). Native and the signed dev app build.
  • Smoke test against the dev app (screen locked): both new shapes reach native and are refused correctly. A bogus snapshot is refused, and a sequence containing Command-W is refused before any key.
  • Live checks are pending (screen locked) and listed in Run the live checks deferred while the screen was locked #42.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant