Repository navigation
feat(drivers): agent-desktop and macos-harness on macOS, with executable TCC grants and MacOSDriver - #501
Conversation
MacOSDriver serves computer/v1 on macOS from ScreenCaptureKit and Quartz events, for drivers whose software brings its own tools. A macOS layer's grants take an executable under its prefix, granted by path; its venv is a copy of the pinned Python and its services start on it, so its grant covers them and what they run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A builder that hasn't stopped within boot_timeout_s is synced again, its guest's log of the last ten minutes saved and warned of, then stopped from outside and kept; the compose's probe judges it. Fixes FD-5257. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A driver's Settings, ComputerSettings or a subclass adding fields, is what it reads at start; its declaration carries the schema when it adds any, and a deploy's driver_settings are checked against it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-desktop 1.2.0 serves its own MCP server's instructions and tools, pinned in mcp.json and forwarded to a server of its own, --headed when the driver's headed setting is on. macos-harness 0.2.1 is one tool that runs a list of its command lines with no shell, its pinned skill as the instructions. Both serve computer/v1 from MacOSDriver and hold TCC permissions through a grant to their own Python. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ness Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xed half second The slow-reader check slept 0.5 s and assumed the fake grab's 30 frames had all been made by then; on a loaded machine they hadn't, and the reader drained 23. It now waits until the source has made every frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Made through BIN's python3 link, a --copies venv's home was BIN, where no standard library is, so its ensurepip failed. It's made from the pinned Python's own path now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Posted back to back, macOS delivered three notches down as one and two left as one, failing the probe's P7 count; 50 ms apart, as a wheel turned by hand, each arrives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
| try: | ||
| out, err = await asyncio.wait_for(process.communicate(), COMMAND_TIMEOUT_S) | ||
| except TimeoutError: | ||
| process.kill() | ||
| await process.wait() | ||
| return 124, f"macos-harness {words[0]} didn't finish within {COMMAND_TIMEOUT_S} s" |
There was a problem hiding this comment.
Cancelled commands keep running
_run kills the harness on TimeoutError, but not when the tool call is cancelled. A cancelled command can keep clicking or typing after the caller has stopped waiting. Stop and wait for the child in a cancellation-safe cleanup path.
The same cleanup gap exists in _agent_desktop: its skill timeout raises without killing the child.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/drivers/envs/macos-harness/driver.py
Line: 110-115
Comment:
**Cancelled commands keep running**
`_run` kills the harness on `TimeoutError`, but not when the tool call is cancelled. A cancelled command can keep clicking or typing after the caller has stopped waiting. Stop and wait for the child in a cancellation-safe cleanup path.
The same cleanup gap exists in `_agent_desktop`: its skill timeout raises without killing the child.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in e17a809: _run now stops the harness process (_stopped) when its call times out or is cancelled, so a cancelled macos_harness call doesn't leave the command running. test_a_cancelled_call_stops_the_command_it_was_running checks the child is gone after the cancel.
A right click released the instant it was pressed was lost now and then, failing P4 on one of two probes; the pointer now rests 0.2 s before a press, as cua-driver's macOS driver found Tk needs, and each press holds 30 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its last 2000 lines were a Hearing daemon's once a second, the moment the shutdown was asked for long gone; the log now keeps every line of the last 15 minutes about shutdown, halt, reboot, logout or a kill signal, then the last 300. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A press at the pointer right after a move took macOS's position before the move reached it, and landed where the pointer was, failing the pointer group's check on one probe; a move now waits up to 0.2 s for macOS to report the pointer where it was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d from outside One that shuts down does so in seconds; waiting boot_timeout_s, ten minutes, cost every hung builder that long before the stop from outside. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
press holds Shift for a character typed with it; a cancelled macos-harness call kills and reaps its command; an executable's path is quoted for TCC's SQL; the shutdown log's ssh timeout and quoted lines are named. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oting Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l instead of holding it Claude Code cuts an MCP server's instructions at 4,096 characters, so the skill (9,166) reached the agent cut off before its notes, failure handling, finishing and ask-first rules. The instructions now map the harness's shell to the tool and tell the agent to run [["skill"]] first, which prints it whole, as the vendor's own setup loads it as a skill. help, the vendor's synonym for --help, runs too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two more computer-use drivers for macOS computers, agent-desktop and macos-harness, beside cua-driver. They need macOS permissions granted to a program that isn't an app bundle, and
computer/v1without cua-driver, so this PR adds both of those.What's in it
Permissions for an executable (
env/envs/computer/utils/install.py,computers/vm_bundle/macos/lib.sh,macos_vm_bundle.py)grantscan name anexecutableunder its prefix as well as anappbundle.grant_executablewrites TCC rows for that path (client type 1), bound to the file's code signature. It also records the screen-capture approval under the path, which is how macOS keys it for a bare executable.--copiesvenv made from the pinned Python's own folder, so the layer has an interpreter of its own. Its services startagentenv-serviceon that interpreter, and macOS charges what a process starts to the program launchd started. So a grant tovenv/bin/pythoncovers the layer's driver and every program it runs, and no other layer.MacOSDriver(agentenv_protocol.computer._macos, with the protocol's newmacosextra)computer/v1on macOS, including the pointer and keyboard groups. ScreenCaptureKit captures the display, and Quartz events send input from the driver's own process.key_downride on every later event.computer-readyreports any of Accessibility, Post Event and Screen Recording that macOS hasn't granted.Driver settings (
computer/driver.py,_declare.py,env/envs/computer/utils/params.py)Settingssubclass ofComputerSettings. Its declaration carries the schema, and core checks a deploy'sdriver_settingsagainst it. This mirrors what apps already have.The drivers (
drivers/envs/agent-desktop/,drivers/envs/macos-harness/)agent-desktop mcp, serves, pinned inmcp.jsonthe way cua-driver's tools are pinned. That's its instructions and its 67desktop_*tools, each forwarded to a server of its own. Itsheadedsetting starts the server with--headed.computer-readychecks both executables' sha256, the tools the server lists, and the permissions.macos_harness. It runs the harness on a list of command lines in order and stops at the first that fails, with no shell. A line that doesn't start with one of the harness's commands runs nothing. Its skill, pinned inskill.md, is the instructions. A read's screenshot comes back as an image. The harness gets none of the driver's environment, and its telemetry is off.Lume: a builder whose shutdown hangs (
providers/sandbox_providers/lume/provider.py)SHUTDOWN_WAIT_S(two minutes), it's synced again, and its guest's log is saved and warned of: the shutdown lines of the last 15 minutes and the last 2 minutes in full. Then it's stopped from outside and kept, and the compose's probe judges it as usual. Before, the compose failed after ten minutes and the evidence was lost.Also
Why this shape
computer/v1. Both drivers serve the probe, recordings and readiness from the same module, so the arms differ only in their own tools.Tested
--copiesinterpreter gets all three permissions as a LaunchAgent;MacOSDriver, each fixed and re-probed: wheel notches posted back to back reached the app as fewer (P7), a right click was sometimes lost (P4), and a press right after a move landed where the pointer had been. It now passes all 55.computer_*tools from the agent).cua_driver_test.pytests fail when the unit suite runs on a Mac host instead of Linux. They fail onmaintoo, so they're not from this PR.Linear: FD-5213 (the other arms on macOS), FD-5257 (the builder that didn't shut down).
🤖 Generated with Claude Code
The PR still needs the remaining child-process cleanup from the previous finding before merging.
Fix with agent prompt
Summary
Adds
agent-desktopandmacos-harnessdrivers for macOS.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR Agent --> Desktop["agent-desktop tools"] Agent --> Harness["macos_harness tool"] Desktop --> MCP["Vendor MCP server"] Harness --> CLI["Pinned harness"] Computer["computer_* tools"] --> MacOSDriver MacOSDriver --> Capture["ScreenCaptureKit"] MacOSDriver --> Input["Quartz input"]Reviews (7) · Last reviewed commit: "fix(drivers): macos-harness's instructio..." · Reviewed by Greptile