Skip to content

Send keys to windows whose focus isn't a text field #13

Closed

Description

@iamnbutler

Found in the Ace desktop tools study (githubnext/ace#427), scenario D2 on Canary 0.0.46+dd15840.

Observed: desktop_key Escape to a Finder window, with a file row selected, was refused twice: "The snapshot has no exact focused control. Click a control, then inspect the window again." The agent clicked a row and tried again, and it was refused again.

Cause: native/sources/client/actions.swift refuses key and insert when the snapshot has no focusedElement. Peekaboo's background input driver only resolves a focused text element as the receiver (focusedTextElement in BackgroundInputDriver.swift). So keys only work while a text field has focus. Escape, arrow keys, Return, Space (Quick Look) and Cmd shortcuts in lists, outlines, canvases and the Finder can't be sent. The hint suggests a click that can't help.

Options:

  • Admit a non-text focused receiver (a list, outline, table or web area) with the same exact process, window and focus checks, using background delivery when the app accepts it.
  • Or expose foreground key delivery for an exact window that has been activated, with the outcome left unknown when focus can't be read back. This is the --foreground gap from [Meta] Complete native computer use in Ace #11.

Short term, the refusal should say that keys need a focused text field, and point to desktop_menu when the command is in a menu.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions