Skip to content

feat(drivers): agent-desktop and macos-harness on macOS, with executable TCC grants and MacOSDriver - #501

Open
earakely-scale wants to merge 17 commits into
mainfrom
edgararakelyan/fd-5213-b7-the-other-arms-on-macos-agent-desktop-and-macos-harness
Open

earakely-scale wants to merge 17 commits into
mainfrom
edgararakelyan/fd-5213-b7-the-other-arms-on-macos-agent-desktop-and-macos-harness

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Two more computer-use drivers for macOS computers, agent-desktop and macos-harness, beside cua-driver. They need macOS permissions granted to a program that isn't an app bundle, and computer/v1 without cua-driver, so this PR adds both of those.

What's in it

Permissions for an executable (env/envs/computer/utils/install.py, computers/vm_bundle/macos/lib.sh, macos_vm_bundle.py)

  • A macOS layer's grants can name an executable under its prefix as well as an app bundle. grant_executable writes TCC rows for that path (client type 1), bound to the file's code signature. It also records the screen-capture approval under the path, which is how macOS keys it for a bare executable.
  • Each macOS layer's venv is now a --copies venv made from the pinned Python's own folder, so the layer has an interpreter of its own. Its services start agentenv-service on that interpreter, and macOS charges what a process starts to the program launchd started. So a grant to venv/bin/python covers the layer's driver and every program it runs, and no other layer.

MacOSDriver (agentenv_protocol.computer._macos, with the protocol's new macos extra)

  • computer/v1 on macOS, including the pointer and keyboard groups. ScreenCaptureKit captures the display, and Quartz events send input from the driver's own process.
  • Typing uses a US keyboard's key codes, so apps see real key events. Characters off that layout are sent as the characters themselves.
  • Modifiers go down and up around whatever they modify, Shift with any shifted character, and keys held with key_down ride on every later event.
  • Events are paced so apps see them all: a click is held and settles, wheel notches are spaced, and a pointer move waits until macOS reports the pointer there before anything presses at it.
  • computer-ready reports any of Accessibility, Post Event and Screen Recording that macOS hasn't granted.

Driver settings (computer/driver.py, _declare.py, env/envs/computer/utils/params.py)

  • A driver can declare a Settings subclass of ComputerSettings. Its declaration carries the schema, and core checks a deploy's driver_settings against it. This mirrors what apps already have.

The drivers (drivers/envs/agent-desktop/, drivers/envs/macos-harness/)

  • agent-desktop 1.2.0: agents get what its own MCP server, agent-desktop mcp, serves, pinned in mcp.json the way cua-driver's tools are pinned. That's its instructions and its 67 desktop_* tools, each forwarded to a server of its own. Its headed setting starts the server with --headed. computer-ready checks both executables' sha256, the tools the server lists, and the permissions.
  • macos-harness 0.2.1: it ships only a command line, so agents get one tool, macos_harness. It runs the harness on a list of command lines in order and stops at the first that fails, with no shell. A line that doesn't start with one of the harness's commands runs nothing. Its skill, pinned in skill.md, is the instructions. A read's screenshot comes back as an image. The harness gets none of the driver's environment, and its telemetry is off.

Lume: a builder whose shutdown hangs (providers/sandbox_providers/lume/provider.py)

  • When a builder hasn't stopped within SHUTDOWN_WAIT_S (two minutes), it's synced again, and its guest's log is saved and warned of: the shutdown lines of the last 15 minutes and the last 2 minutes in full. Then it's stopped from outside and kept, and the compose's probe judges it as usual. Before, the compose failed after ten minutes and the evidence was lost.

Also

  • A recording test that assumed 30 fake frames are made within half a second now waits for the source. It was flaky under load.

Why this shape

  • Grants by executable path. These two tools are a plain binary and a Python package, not app bundles. A copied interpreter per layer gives each driver its own identity without an app wrapper per arm.
  • Each arm reaches the agent as its vendor ships it. agent-desktop has had its own MCP server since this release, so it's forwarded the way cua-driver's is. macos-harness is a command line whose skill chains actions in one call, so it's one tool that takes a list of command lines.
  • One shared computer/v1. Both drivers serve the probe, recordings and readiness from the same module, so the arms differ only in their own tools.

Tested

  • Unit: the touched protocol, core, driver and bundle suites pass on an EC2 Mac.
  • TCC on a macOS 26.5.2 VM, in a clone of a composed computer:
    • a granted --copies interpreter gets all three permissions as a LaunchAgent;
    • an ungranted program it starts inherits them;
    • the pinned Python and other copies are denied;
    • a granted copy started by an ungranted one is denied;
    • the grant holds after a reboot;
    • exec'ing on the same copy keeps the grant.
  • ScreenCaptureKit capture: 15–50 ms a frame from the granted copy.
  • Composed and fully probed on EC2 Macs (gate 3): both computers pass all 55 checks.
    • macos-harness passed on its first compose.
    • agent-desktop's probes found three timing problems in MacOSDriver, each fixed and re-probed: wheel notches posted back to back reached the app as fewer (P7), a right click was sometimes lost (P4), and a press right after a move landed where the pointer had been. It now passes all 55.
  • Bench trials: both drivers run the macOS Cua Driver Bench's trials end to end with only their own tools (the bench hides the computer_* tools from the agent).
  • Two cua_driver_test.py tests fail when the unit suite runs on a Mac host instead of Linux. They fail on main too, so they're not from this PR.

Linear: FD-5213 (the other arms on macOS), FD-5257 (the builder that didn't shut down).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

The PR still needs the remaining child-process cleanup from the previous finding before merging.

Fix All in CursorFindings

  1. P1 Cancelled commands keep running ▶
Fix with agent prompt
### Issue 1
src/agent_env/drivers/envs/macos-harness/driver.py:112-116
`_run` kills the harness on `TimeoutError`, but not when the tool call is cancelled. A cancelled command can keep clicking or typing after the caller has stopped waiting. Stop and wait for the child in a cancellation-safe cleanup path.

The same cleanup gap exists in `_agent_desktop`: its skill timeout raises without killing the child.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds agent-desktop and macos-harness drivers for macOS.

  • Each macOS layer can grant permissions to its own executable.
  • MacOSDriver captures screens and sends computer input on macOS.
  • Drivers can declare settings that deploys can check.
  • Agents can use agent-desktop’s pinned macOS tools through a computer driver.
  • Agents can run macos-harness commands through one computer tool.
  • Lume keeps builders whose shutdown takes longer than two minutes.
  • The recording test waits for its frames before checking the stream.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Agent --> Desktop["agent-desktop tools"]
    Agent --> Harness["macos_harness tool"]
    Desktop --> MCP["Vendor MCP server"]
    Harness --> CLI["Pinned harness"]
    Computer["computer_* tools"] --> MacOSDriver
    MacOSDriver --> Capture["ScreenCaptureKit"]
    MacOSDriver --> Input["Quartz input"]
Loading

Reviews (7) · Last reviewed commit: "fix(drivers): macos-harness's instructio..." · Reviewed by Greptile

earakely-scale and others added 10 commits November 4, 2026 22:52
MacOSDriver serves computer/v1 on macOS from ScreenCaptureKit and Quartz
events, for drivers whose software brings its own tools. A macOS layer's
grants take an executable under its prefix, granted by path; its venv is a
copy of the pinned Python and its services start on it, so its grant
covers them and what they run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A builder that hasn't stopped within boot_timeout_s is synced again,
its guest's log of the last ten minutes saved and warned of, then
stopped from outside and kept; the compose's probe judges it. Fixes
FD-5257.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A driver's Settings, ComputerSettings or a subclass adding fields, is
what it reads at start; its declaration carries the schema when it adds
any, and a deploy's driver_settings are checked against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-desktop 1.2.0 serves its own MCP server's instructions and tools,
pinned in mcp.json and forwarded to a server of its own, --headed when
the driver's headed setting is on. macos-harness 0.2.1 is one tool that
runs a list of its command lines with no shell, its pinned skill as the
instructions. Both serve computer/v1 from MacOSDriver and hold TCC
permissions through a grant to their own Python.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ness

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xed half second

The slow-reader check slept 0.5 s and assumed the fake grab's 30 frames
had all been made by then; on a loaded machine they hadn't, and the
reader drained 23. It now waits until the source has made every frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Made through BIN's python3 link, a --copies venv's home was BIN, where no
standard library is, so its ensurepip failed. It's made from the pinned
Python's own path now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Posted back to back, macOS delivered three notches down as one and two
left as one, failing the probe's P7 count; 50 ms apart, as a wheel turned
by hand, each arrives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 5, 2026 08:17
@socket-security

socket-security Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​macos-harness@​0.2.197100100100100
Addedpypi/​pyobjc-framework-quartz@​13.099100100100100
Addedpypi/​pyobjc-framework-screencapturekit@​13.0100100100100100

View full report

Comment on lines +110 to +115
try:
out, err = await asyncio.wait_for(process.communicate(), COMMAND_TIMEOUT_S)
except TimeoutError:
process.kill()
await process.wait()
return 124, f"macos-harness {words[0]} didn't finish within {COMMAND_TIMEOUT_S} s"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Cancelled commands keep running

_run kills the harness on TimeoutError, but not when the tool call is cancelled. A cancelled command can keep clicking or typing after the caller has stopped waiting. Stop and wait for the child in a cancellation-safe cleanup path.

The same cleanup gap exists in _agent_desktop: its skill timeout raises without killing the child.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/drivers/envs/macos-harness/driver.py
Line: 110-115

Comment:
**Cancelled commands keep running**

`_run` kills the harness on `TimeoutError`, but not when the tool call is cancelled. A cancelled command can keep clicking or typing after the caller has stopped waiting. Stop and wait for the child in a cancellation-safe cleanup path.

The same cleanup gap exists in `_agent_desktop`: its skill timeout raises without killing the child.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e17a809: _run now stops the harness process (_stopped) when its call times out or is cancelled, so a cancelled macos_harness call doesn't leave the command running. test_a_cancelled_call_stops_the_command_it_was_running checks the child is gone after the cancel.

earakely-scale and others added 7 commits November 5, 2026 00:24
A right click released the instant it was pressed was lost now and then,
failing P4 on one of two probes; the pointer now rests 0.2 s before a press,
as cua-driver's macOS driver found Tk needs, and each press holds 30 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its last 2000 lines were a Hearing daemon's once a second, the moment the
shutdown was asked for long gone; the log now keeps every line of the last
15 minutes about shutdown, halt, reboot, logout or a kill signal, then the
last 300.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A press at the pointer right after a move took macOS's position before the
move reached it, and landed where the pointer was, failing the pointer
group's check on one probe; a move now waits up to 0.2 s for macOS to
report the pointer where it was sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d from outside

One that shuts down does so in seconds; waiting boot_timeout_s, ten
minutes, cost every hung builder that long before the stop from outside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
press holds Shift for a character typed with it; a cancelled macos-harness
call kills and reaps its command; an executable's path is quoted for TCC's
SQL; the shutdown log's ssh timeout and quoted lines are named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oting

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l instead of holding it

Claude Code cuts an MCP server's instructions at 4,096 characters, so the skill (9,166) reached the agent cut off before its notes, failure handling, finishing and ask-first rules. The instructions now map the harness's shell to the tool and tell the agent to run [["skill"]] first, which prints it whole, as the vendor's own setup loads it as a skill. help, the vendor's synonym for --help, runs too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant