Skip to content

feat: turn on Chromium and Electron web accessibility before inspect and read, once per app launch - #107

Merged
iamnbutler merged 1 commit into
mainfrom
feat/web-accessibility
Oct 17, 2026
Merged

iamnbutler merged 1 commit into
mainfrom
feat/web-accessibility

Conversation

@iamnbutler

@iamnbutler iamnbutler commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Part (b) of #5: Chromium and Electron build their web content's Accessibility tree only after an assistive client asks, so their web fields were missing from desktop_inspect and desktop_read.

What changes

  • New Bridge op enableWebAccessibility (native/patches/peekaboo-web-accessibility.patch, last in the stack):
    • The host sets AXManualAccessibility on one exact app generation, on the main actor with a 1 s AX messaging timeout. It never sets AXEnhancedUserInterface.
    • It sets the attribute at most once per launch: it remembers the generations it turned on or found on, and reads the attribute before setting it.
    • Result: bool (true = turned on now). Outcome is confirmedChange when it reads back on, dispatchedUnverified otherwise, and confirmedNoChange when it was already on. An app that rejects the attribute gets operationNotSupported.
    • Contract: request-pinned, process-scoped, needs Accessibility. Wire count goes 132→133 (on top of feat: paste an image or files into a native text area with desktop_insert, or leave the clipboard replaced on request #99); the op joins the mutating set and the Accessibility partition.
  • Client (native/sources/client/webaccessibility.swift): inspect (accessibility mode) and read call the op in two cases:
    • Before reading, when the app bundle ships Chromium: an Electron or Chromium Embedded framework, or a framework whose helpers include a crashpad_handler (Chrome, Edge, Atlas). When it turned on now, they wait 500 ms first.
    • After a read that was not truncated and found an AXWebArea with at most one element under it. When this turns it on, the client discards that read (cleaning its snapshot), waits, and reads once more.
    • Failures never fail the read.
    • Results report web_accessibility: enabled, already_on, unsupported or unknown. It is absent when the op didn't run.
  • Opt-out: web_accessibility: false on desktop_inspect and desktop_read. Their descriptions say they turn it on (tools/list +688 B).
  • Pause and activity:
    • The op is pause-exempt: it only lets the app's web content be read and sends no input.
    • All Activity lists "Turn on web accessibility" once per app launch (keyed by pid and process start time). Refusals are not listed.
  • Docs: tools.md (Input section and the inspect details), app.md (pause and activity), patches README.
  • Live checks:

Tests

  • npm run check, npm test (57 pass, including the new test/webaccessibility.test.mjs).
  • Full node dist/build.js applies the patch stack.
  • Swift tests: PeekabooBridgeRequestPlanTests, PeekabooBridgeOperationSemanticPlanTests and PeekabooBridgeOperationResultSemanticsTests pass (72 tests).
  • Live checks are pending: the screen is locked.

…and read, once per app launch

The host sets AXManualAccessibility (never AXEnhancedUserInterface) through a new
enableWebAccessibility Bridge operation, before reading an app that ships Chromium
or after a read finds a nearly empty web area, then reads again. Results report
web_accessibility; web_accessibility false on inspect and read opts out. It runs
while input is paused and is listed once per launch in the activity list.

Also lists the #5 background-paste spike questions as a manual live check.
@iamnbutler
iamnbutler force-pushed the feat/web-accessibility branch from 50b1396 to 81b83af Compare October 17, 2026 09:01
@iamnbutler
iamnbutler merged commit b218b1c into main Oct 17, 2026
2 checks passed
@iamnbutler
iamnbutler deleted the feat/web-accessibility branch October 17, 2026 09:06
iamnbutler added a commit that referenced this pull request Oct 17, 2026
…#111 read check (#116)

Each Chromium check (#107, #110, #111) launches its own Chrome on the fixture
page with a temporary --user-data-dir, --no-first-run and
--no-default-browser-check, then quits only that pid and deletes the profile.
They no longer skip while the user's Chrome runs, and each starts with web
accessibility off. #110 now waits for a page-only element (a table) on a fresh
launch, and #111 reads the page the moment it opens: it passes on page text,
or on a failure that says retried. Pause stays manual and user-only.

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant