diff --git a/docs/app.md b/docs/app.md index fa42e91..c0a5e71 100644 --- a/docs/app.md +++ b/docs/app.md @@ -13,10 +13,12 @@ Anything running as the user can use those grants while the app serves, so grant - the target app and window title; - the agent (`DESKTOP_TOOLS_AGENT`, or the name of the process that ran the client, such as `node` for the MCP server) and that process's pid; - the time; -- the outcome: completed, unknown or refused. +- the outcome: completed, unknown, refused, or paused for a request refused because input was paused. Identical operations in a row fold into one line with a count. Every desktop change is listed, and so are screen captures and clipboard reads. Listings and snapshot bookkeeping are not. Only the operation and its target are kept, never typed text, clipboard contents or values. The list lives in memory and is gone when the app quits. +**Pause All Input** stops agents from changing anything until you turn it off again. While paused, the app refuses every click, key, typing, scroll, drag, menu command, app or window change, dialog action and clipboard write before it is sent. The refusal tells the agent to ask you to resume. Agents can still observe: listings, the accessibility tree, screen captures and clipboard reads keep working. The menu bar icon becomes a pause sign while input is paused. Only this menu item changes it; no client can pause or resume. Pausing isn't saved, so the app starts resumed. `desktop_input_source` select runs in the client rather than through the app, so pausing does not stop it. + ## Build ```sh diff --git a/docs/embed.md b/docs/embed.md index 0969285..560bbc8 100644 --- a/docs/embed.md +++ b/docs/embed.md @@ -7,11 +7,12 @@ | Symbol | Meaning | | --- | --- | | `void desktop_tools_start(const char *socket, const char *client)` | Serve the Bridge on `socket` for the exact signed `client` identifier from this app's team | -| `char *desktop_tools_status(void)` | JSON `{ state, error?, accessibility, screenRecording, eventSynthesizing, clipboardRead, cursor, highlight }`; free with `desktop_tools_free` | +| `char *desktop_tools_status(void)` | JSON `{ state, error?, accessibility, screenRecording, eventSynthesizing, clipboardRead, cursor, highlight, paused }`; free with `desktop_tools_free` | | `void desktop_tools_stop(void)` | Drain native operations and stop; poll status until `stopped` | | `void desktop_tools_permission(const char *kind)` | Request `accessibility`, `screenRecording`, or `eventSynthesizing` | | `void desktop_tools_set_cursor(bool)` | Show the agent cursor for pointer input; it is off until turned on | | `void desktop_tools_set_highlight(bool)` | Outline and label the window the agent is using; it is off until turned on | +| `void desktop_tools_set_paused(bool)` | Pause all input: refuse every operation that changes the desktop, and clipboard writes, before dispatch, while reads continue. It starts resumed, is held in memory only, and only the embedding app can change it | | `char *desktop_tools_activity(void)` | JSON array of the last 20 operations, newest first: `{ time, agent, pid, operation, app?, window?, outcome, count }`; free with `desktop_tools_free` | | `void desktop_tools_free(void *)` | Release a status or activity string | diff --git a/native/patches/README.md b/native/patches/README.md index eb28c33..a6253ce 100644 --- a/native/patches/README.md +++ b/native/patches/README.md @@ -527,6 +527,8 @@ response, descriptor and semantics with the read; the operation inventory tests - **Advisory only.** The report changes no routing or response. Without an observer, the server does no extra work. - **No new operation.** The wire count is unchanged. +The same patch adds an `operationAdmission` hook, for the embedding app's pause switch. The server asks it for every authorized request, after operation access is validated and before the target window is reported or anything is dispatched. It passes the operation and whether its result contract mutates the desktop. A returned message fails the request as a pre-dispatch `permissionDenied`, so it may not have completed and is recorded as `refused`. Without a hook, nothing changes. + When updating Peekaboo, review the upstream fixes and these patches together. Update the revision guard and patches deliberately, or remove each patch when the dependency includes its fix. Revalidate real button clicks, changed-focus refusal, literal multiline insertion, clipboard diff --git a/native/patches/peekaboo-operation-record.patch b/native/patches/peekaboo-operation-record.patch index 2966cdc..704cd95 100644 --- a/native/patches/peekaboo-operation-record.patch +++ b/native/patches/peekaboo-operation-record.patch @@ -1,20 +1,22 @@ diff --git a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeBootstrap.swift b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeBootstrap.swift -index b0627bc94..b6d2bab3f 100644 +index b0627bc94..07059f0ae 100644 --- a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeBootstrap.swift +++ b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeBootstrap.swift -@@ -19,6 +19,7 @@ public enum PeekabooBridgeBootstrap { +@@ -19,6 +19,8 @@ public enum PeekabooBridgeBootstrap { let requestDrainTimeoutSec: TimeInterval let screenCaptureKitOwnershipPreparer: @Sendable () async throws -> Void var targetWindowObserver: (@Sendable (PeekabooBridgeTargetWindow) -> Void)? = nil + var operationObserver: (@Sendable (PeekabooBridgeOperationRecord) -> Void)? = nil ++ var operationAdmission: (@Sendable (PeekabooBridgeOperation, Bool) -> String?)? = nil } @discardableResult -@@ -118,6 +119,7 @@ public enum PeekabooBridgeBootstrap { +@@ -118,6 +120,8 @@ public enum PeekabooBridgeBootstrap { automationActivityObserver: configuration.automationActivityObserver, browserSessionBootstrapProvider: services.browserSessionBootstrapProvider) server.targetWindowObserver = configuration.targetWindowObserver + server.operationObserver = configuration.operationObserver ++ server.operationAdmission = configuration.operationAdmission let host = PeekabooBridgeHost( socketPath: configuration.socketPath, server: server, @@ -121,7 +123,7 @@ index 000000000..a2ad0d774 + } +} diff --git a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer+Routing.swift b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer+Routing.swift -index aca49389f..6f02f5140 100644 +index aca49389f..4d7846995 100644 --- a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer+Routing.swift +++ b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer+Routing.swift @@ -18,6 +18,15 @@ extension PeekabooBridgeServer { @@ -139,6 +141,20 @@ index aca49389f..6f02f5140 100644 + peer: PeekabooBridgePeer?) async throws -> PeekabooBridgeHandledResponse { let request = plan.request + do { +@@ -83,6 +92,13 @@ extension PeekabooBridgeServer { + plan: plan, + stage: .preDispatch(PeekabooBridgeOperationResultSemantics.preDispatchReason(for: envelope))) + } ++ if let refusal = self.operationAdmission?(op, plan.result.completion.mutatesDesktop) { ++ failed = true ++ throw PeekabooBridgeOperationResultSemantics.canonicalFailure( ++ .init(code: .permissionDenied, message: refusal), ++ plan: plan, ++ stage: .preDispatch(.permissionDenied)) ++ } + await self.reportTargetWindow(plan, peer: peer) + do { diff --git a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer+TargetWindow.swift b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer+TargetWindow.swift index 728abef12..5a3d5cdb6 100644 @@ -180,47 +196,54 @@ index 728abef12..5a3d5cdb6 100644 private func snapshotWindow(_ snapshotId: String) async -> (pid_t, CGWindowID)? { diff --git a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer.swift b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer.swift -index 5a1c3c3e3..9f6fbd161 100644 +index 5a1c3c3e3..b0d87cdc2 100644 --- a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer.swift +++ b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooBridgeServer.swift -@@ -141,6 +141,8 @@ public final class PeekabooBridgeServer { +@@ -141,6 +141,11 @@ public final class PeekabooBridgeServer { let automationActivityObserver: (@Sendable (pid_t) -> Void)? /// Receives each authorized request's exact target window before dispatch. var targetWindowObserver: (@Sendable (PeekabooBridgeTargetWindow) -> Void)? + /// Receives each authorized request once it has finished, with its target and outcome. + var operationObserver: (@Sendable (PeekabooBridgeOperationRecord) -> Void)? ++ /// Refuses an authorized request before dispatch when it returns a message: the operation and whether it ++ /// mutates the desktop. ++ var operationAdmission: (@Sendable (PeekabooBridgeOperation, Bool) -> String?)? let encoder: JSONEncoder let decoder: JSONDecoder let logger = Logger(subsystem: "boo.peekaboo.bridge", category: "server") diff --git a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooEmbeddedBridgeRuntime.swift b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooEmbeddedBridgeRuntime.swift -index a3723e271..09e92c69b 100644 +index a3723e271..814aba2bc 100644 --- a/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooEmbeddedBridgeRuntime.swift +++ b/Core/PeekabooCore/Sources/PeekabooBridge/PeekabooEmbeddedBridgeRuntime.swift -@@ -156,6 +156,8 @@ public actor PeekabooEmbeddedBridgeRuntime { +@@ -156,6 +156,11 @@ public actor PeekabooEmbeddedBridgeRuntime { public let requestDrainTimeoutSeconds: TimeInterval /// Receives each authorized request's exact target window before dispatch, on the main actor. public var targetWindowObserver: (@Sendable (PeekabooBridgeTargetWindow) -> Void)? + /// Receives each authorized request once it has finished, with its target and outcome, on the main actor. + public var operationObserver: (@Sendable (PeekabooBridgeOperationRecord) -> Void)? ++ /// Refuses an authorized request before dispatch when it returns a message, given the operation and whether ++ /// it mutates the desktop. Called on the main actor for every request, so it must be quick. ++ public var operationAdmission: (@Sendable (PeekabooBridgeOperation, Bool) -> String?)? let screenCaptureKitProcessCapabilityRegistrar: @Sendable () throws -> Void let screenCaptureKitOwnershipPreparer: @Sendable () async throws -> Void -@@ -390,7 +392,8 @@ public actor PeekabooEmbeddedBridgeRuntime { +@@ -390,7 +395,9 @@ public actor PeekabooEmbeddedBridgeRuntime { requestTimeoutSec: configuration.requestTimeoutSeconds, requestDrainTimeoutSec: configuration.requestDrainTimeoutSeconds, screenCaptureKitOwnershipPreparer: configuration.screenCaptureKitOwnershipPreparer, - targetWindowObserver: configuration.targetWindowObserver)) + targetWindowObserver: configuration.targetWindowObserver, -+ operationObserver: configuration.operationObserver)) ++ operationObserver: configuration.operationObserver, ++ operationAdmission: configuration.operationAdmission)) } let host = preparedHost.host diff --git a/Core/PeekabooCore/Tests/PeekabooTests/PeekabooBridgeOperationRecordTests.swift b/Core/PeekabooCore/Tests/PeekabooTests/PeekabooBridgeOperationRecordTests.swift new file mode 100644 -index 000000000..814c42440 +index 000000000..3381bf8b5 --- /dev/null +++ b/Core/PeekabooCore/Tests/PeekabooTests/PeekabooBridgeOperationRecordTests.swift -@@ -0,0 +1,101 @@ +@@ -0,0 +1,142 @@ +import CoreGraphics +import Foundation +import PeekabooAutomationKit @@ -286,6 +309,34 @@ index 000000000..814c42440 + } + + @Test ++ func `an admission refusal stops the request before dispatch and is recorded as refused`() async throws { ++ let server = PeekabooBridgeServer( ++ services: StubServices(snapshots: InMemorySnapshotManager()), ++ hostKind: .gui, ++ allowlistedTeams: [], ++ allowlistedBundles: []) ++ let records = OperationRecords() ++ server.operationObserver = { records.append($0) } ++ let asked = AdmissionQuestions() ++ server.operationAdmission = { operation, mutates in ++ asked.append(operation, mutates) ++ return operation == .listApplications ? "Input is paused." : nil ++ } ++ ++ do { ++ _ = try await server.route(Self.plan(.listApplications), peer: Self.peer) ++ Issue.record("The paused request was dispatched.") ++ } catch let envelope as PeekabooBridgeErrorEnvelope { ++ #expect(envelope.message == "Input is paused.") ++ #expect(envelope.operationMayHaveCompleted == false) ++ } ++ ++ #expect(asked.values.map(\.0) == [.listApplications]) ++ #expect(asked.values.map(\.1) == [false]) ++ #expect(records.values.map(\.outcome) == [.refused]) ++ } ++ ++ @Test + func `mutation states map to completed, unknown or refused`() { + #expect(PeekabooBridgeServer.recordedOutcome(nil) == .completed) + #expect(PeekabooBridgeServer.recordedOutcome(.confirmedChange) == .completed) @@ -310,6 +361,19 @@ index 000000000..814c42440 + } +} + ++private final class AdmissionQuestions: @unchecked Sendable { ++ private let lock = NSLock() ++ private var questions: [(PeekabooBridgeOperation, Bool)] = [] ++ ++ var values: [(PeekabooBridgeOperation, Bool)] { ++ self.lock.withLock { self.questions } ++ } ++ ++ func append(_ operation: PeekabooBridgeOperation, _ mutates: Bool) { ++ self.lock.withLock { self.questions.append((operation, mutates)) } ++ } ++} ++ +private final class OperationRecords: @unchecked Sendable { + private let lock = NSLock() + private var records: [PeekabooBridgeOperationRecord] = [] diff --git a/native/sources/app/app.swift b/native/sources/app/app.swift index 194a1f6..d86f00b 100644 --- a/native/sources/app/app.swift +++ b/native/sources/app/app.swift @@ -11,6 +11,7 @@ private struct Runtime { let setCursor: @convention(c) (Bool) -> Void let setHighlight: @convention(c) (Bool) -> Void let activity: @convention(c) () -> UnsafeMutablePointer? + let setPaused: @convention(c) (Bool) -> Void init(_ path: String) { guard let handle = dlopen(path, RTLD_NOW) else { fatalError(String(cString: dlerror())) } @@ -26,6 +27,7 @@ private struct Runtime { setCursor = symbol("desktop_tools_set_cursor") setHighlight = symbol("desktop_tools_set_highlight") activity = symbol("desktop_tools_activity") + setPaused = symbol("desktop_tools_set_paused") } } @@ -35,6 +37,7 @@ private struct Status: Decodable { var accessibility: Bool var screenRecording: Bool var eventSynthesizing: Bool + var paused: Bool? } /// One recent operation from the runtime, newest first; it never includes typed text or values. @@ -60,6 +63,8 @@ private final class Host: NSObject, NSApplicationDelegate, NSMenuDelegate { private let item = NSStatusBar.system.statusItem(withLength: NSStatusItem.variableLength) private let menu = NSMenu() private var serving = true + // Pausing lasts until the user resumes or the app quits; it is never saved. + private var paused = false // The app shows the agent cursor and window outline unless the user turned them off; embedding apps start with them off. private var cursor = UserDefaults.standard.object(forKey: "agentCursor") as? Bool ?? true private var highlight = UserDefaults.standard.object(forKey: "agentWindow") as? Bool ?? true @@ -87,7 +92,7 @@ private final class Host: NSObject, NSApplicationDelegate, NSMenuDelegate { } func applicationDidFinishLaunching(_: Notification) { - item.button?.image = NSImage(systemSymbolName: "cursorarrow.rays", accessibilityDescription: "Desktop Tools") + updateIcon() menu.delegate = self item.menu = menu runtime.setCursor(cursor) @@ -114,6 +119,7 @@ private final class Host: NSObject, NSApplicationDelegate, NSMenuDelegate { menu.removeAllItems() let status = read() let state = status.map { $0.error.map { "Error: \($0)" } ?? $0.state.capitalized } ?? "Unavailable" + + (paused ? " (input paused)" : "") menu.addItem(withTitle: "Desktop Tools: \(state)", action: nil, keyEquivalent: "") menu.addItem(withTitle: socket, action: nil, keyEquivalent: "") menu.addItem(.separator()) @@ -133,6 +139,11 @@ private final class Host: NSObject, NSApplicationDelegate, NSMenuDelegate { menu.addItem(entry) } menu.addItem(.separator()) + let pause = NSMenuItem(title: "Pause All Input", action: #selector(togglePause), keyEquivalent: "") + pause.target = self + pause.state = paused ? .on : .off + pause.toolTip = "Refuse every click, key, typing, window change and clipboard write from agents until resumed. Reading the screen still works." + menu.addItem(pause) let toggle = NSMenuItem(title: "Serve agents", action: #selector(toggle), keyEquivalent: "") toggle.target = self toggle.state = serving ? .on : .off @@ -183,7 +194,8 @@ private final class Host: NSObject, NSApplicationDelegate, NSMenuDelegate { keyEquivalent: "") let time = Date(timeIntervalSince1970: entry.time).formatted(date: .omitted, time: .standard) item.subtitle = "\(entry.agent) (\(entry.pid)) · \(time) · \(entry.outcome)" - let symbol = ["completed": "checkmark.circle", "refused": "xmark.circle"][entry.outcome] ?? "questionmark.circle" + let symbol = ["completed": "checkmark.circle", "refused": "xmark.circle", "paused": "pause.circle"][entry.outcome] + ?? "questionmark.circle" item.image = NSImage(systemSymbolName: symbol, accessibilityDescription: entry.outcome) return item } @@ -192,6 +204,21 @@ private final class Host: NSObject, NSApplicationDelegate, NSMenuDelegate { (sender.representedObject as! String).withCString(runtime.permission) } + @objc private func togglePause() { + paused.toggle() + runtime.setPaused(paused) + updateIcon() + } + + /// The menu bar icon shows a pause sign while input is paused, so the state is visible at a glance. + private func updateIcon() { + let symbol = paused ? "pause.circle.fill" : "cursorarrow.rays" + item.button?.image = NSImage( + systemSymbolName: symbol, + accessibilityDescription: paused ? "Desktop Tools, input paused" : "Desktop Tools") + item.button?.toolTip = paused ? "Desktop Tools: input paused" : "Desktop Tools" + } + @objc private func toggle() { serving.toggle() if serving { runtime.start(socket, client) } else { runtime.stop() } diff --git a/native/sources/runtime/activity.swift b/native/sources/runtime/activity.swift index 65a4dd2..1b52567 100644 --- a/native/sources/runtime/activity.swift +++ b/native/sources/runtime/activity.swift @@ -107,6 +107,46 @@ private let activityHidden: Set = [ .disconnectExactWindowHeldPointerOwner, .prepareDialogAction, ] +/// Pause all input: while on, the runtime refuses every operation that changes the desktop, and clipboard writes, +/// before dispatch. Only the embedding app sets it, from its own menu; clients have no way to change it. +final class Pause: @unchecked Sendable { + static let shared = Pause() + static let message = "Input is paused in Desktop Tools, so nothing was sent. Ask the user to resume input from the Desktop Tools menu; reading the desktop still works." + private let lock = NSLock() + private var paused = false + + var isPaused: Bool { + lock.lock() + defer { lock.unlock() } + return paused + } + + func set(_ value: Bool) { + lock.lock() + paused = value + lock.unlock() + } + + /// The refusal for one request, or nil when it may run. + func refusal(_ operation: PeekabooBridgeOperation, mutatesDesktop: Bool) -> String? { + isPaused && Self.blocks(operation, mutatesDesktop: mutatesDesktop) ? Self.message : nil + } + + /// Desktop changes and clipboard writes, but never snapshot bookkeeping or releasing a held button. + static func blocks(_ operation: PeekabooBridgeOperation, mutatesDesktop: Bool) -> Bool { + if pauseExempt.contains(operation) { return false } + return mutatesDesktop || clipboardWrites.contains(operation) + } +} + +private let clipboardWrites: Set = [.clipboardTextWrite, .clipboardImageWrite, .clipboardFilesWrite] + +/// Reads and cleanup keep working while paused, so a paused agent can still observe and nothing stays held. +private let pauseExempt: Set = [ + .createSnapshot, .cleanSnapshot, .ownsSnapshot, .getDetectionResult, .finishSnapshotMutation, + .releaseExactWindowHeldPointer, .revokeExactWindowHeldPointer, .disconnectExactWindowHeldPointerOwner, +] + /// Records one finished request when it is worth listing. Runs while the client still waits for its reply, /// so the client's environment and parent are still readable. func recordActivity(_ record: PeekabooBridgeOperationRecord) { @@ -125,7 +165,8 @@ func recordActivity(_ record: PeekabooBridgeOperationRecord) { operation: name, app: record.targetProcessIdentifier.flatMap { NSRunningApplication(processIdentifier: $0)?.localizedName }, window: record.targetWindowID.flatMap(windowTitle), - outcome: record.outcome.rawValue, + outcome: record.outcome == .refused && Pause.shared.isPaused + && Pause.blocks(operation, mutatesDesktop: record.mutatesDesktop) ? "paused" : record.outcome.rawValue, count: 1 )) } diff --git a/native/sources/runtime/runtime.swift b/native/sources/runtime/runtime.swift index c0f75c9..f68dc4e 100644 --- a/native/sources/runtime/runtime.swift +++ b/native/sources/runtime/runtime.swift @@ -16,6 +16,7 @@ private struct Status: Encodable { var clipboardRead = ClipboardPolicy() var cursor = false var highlight = false + var paused = false } private final class State: @unchecked Sendable { @@ -37,6 +38,7 @@ private final class State: @unchecked Sendable { result.screenRecording = CGPreflightScreenCaptureAccess() result.eventSynthesizing = result.eventSynthesizing || CGPreflightPostEventAccess() result.clipboardRead = ClipboardPolicy.current() + result.paused = Pause.shared.isPaused return result } @@ -106,6 +108,9 @@ private final class Desktop { } } configuration.operationObserver = recordActivity + configuration.operationAdmission = { operation, mutatesDesktop in + Pause.shared.refusal(operation, mutatesDesktop: mutatesDesktop) + } let next = PeekabooEmbeddedBridgeRuntime.make(configuration: configuration, feedbackClient: AgentCursor.shared) runtime = next try await next.startChecked() @@ -170,6 +175,13 @@ public func setHighlight(_ enabled: Bool) { DispatchQueue.main.async { AgentWindow.shared.enabled = enabled } } +/// Pauses or resumes all input: while paused, every operation that changes the desktop, and clipboard writes, +/// are refused before dispatch, and reads keep working. Only the embedding app calls this; it starts resumed. +@_cdecl("desktop_tools_set_paused") +public func setPaused(_ paused: Bool) { + Pause.shared.set(paused) +} + /// Returns the recent operations as a JSON array, newest first; release it with `desktop_tools_free`. /// They are kept in memory only and never include typed text, clipboard contents or values. @_cdecl("desktop_tools_activity")