Skip to content

fix(daemon): start one daemon when clients race on a cold start - #336

Open
hatemadi15 wants to merge 1 commit into
cocoindex-io:mainfrom
hatemadi15:fix/daemon-single-start
Open

hatemadi15 wants to merge 1 commit into
cocoindex-io:mainfrom
hatemadi15:fix/daemon-single-start

Conversation

@hatemadi15

@hatemadi15 hatemadi15 commented Nov 9, 2026 •

Copy link
Copy Markdown

When two clients find no daemon at the same moment, both start one. A fresh ccc mcp session does this on its first search, which races the background index. So do two ccc commands run together, two clients that both notice a global_settings.yml change, and ccc daemon restart under the Docker entrypoint's supervisor loop. Nothing serializes the start, and on POSIX each new daemon unlinks the socket path and binds its own. The daemon that bound first keeps running where no client can reach it: it is reparented to init, and if it already served a request it holds the project's LMDB and SQLite files open. ccc daemon stop stops the reachable daemon, then reports "Daemon is not running." while the other is still alive. The unreachable daemon exits only at its idle timeout: 180 minutes by default, never under the supervisor.

Change

  • Daemon lock. A daemon takes an exclusive lock on daemon.lock in its runtime dir (flock; msvcrt.locking on Windows) before anything else, and releases it right after closing its listener. A daemon that finds the lock held exits at once with status 75 (DAEMON_ALREADY_RUNNING_EXIT_CODE), before it reads settings or touches the PID file, last-exit marker, log or socket.
  • Crash safety. The OS drops the lock when its holder dies, so a crashed daemon's leftover lock file and socket file never block the next start. _UnixListener still removes a stale socket, but only a lock holder gets that far now.
  • Losing client. _wait_for_daemon() treats status 75 as "another daemon got there first" and keeps waiting for that daemon's socket, so both clients end up on one daemon. After the handshake, a client whose own process is not the daemon that answered waits for that process to exit; otherwise the MCP server would keep it as a zombie ([ccc] <defunct>) until its next spawn.
  • Supervised restart. Under a supervisor (COCOINDEX_CODE_DAEMON_SUPERVISED=1, the Docker image), ccc daemon restart now waits for the supervisor's respawn instead of starting a daemon itself, as auto-start already does. Starting its own raced the entrypoint loop.

#243 takes the other approach: a lock file around the client's check → spawn → wait. This PR puts the lock in the daemon, because the displacement happens there and not every daemon is started by a client that would take a client-side lock. The Docker entrypoint's loop and a plain ccc run-daemon start one directly, and ccc daemon restart calls start_daemon() outside #243's lock. noVaSon reported the ccc run-daemon case in #243's thread and suggested this fix: a lock the daemon holds for its lifetime. The two PRs change the same start-up code in client.py, so they conflict. #243 also raises the listener backlog from 1 to 64; this PR leaves the backlog alone. Racing clients still each spawn a process; the loser exits about 0.3 s after it starts, without reading settings or serving anything.

On Windows, PipeListener creates its first pipe instance with FILE_FLAG_FIRST_PIPE_INSTANCE, so a second daemon could never displace the first: it failed at Listener() with PermissionError, but only after overwriting daemon.pid, deleting the last-exit marker and truncating daemon.log. It now exits before touching any of them.

Validation

  • uv run pytest tests/: 353 passed, 8 deselected (docker_e2e). uv run prek run --all-files: all hooks pass. uv run mypy .: 2 errors in scripts/find_best_models.py, identical on main; CI's hook (mypy src/ tests/) is clean.
  • New tests: test_concurrent_cold_starts_leave_one_daemon (two clients start daemons at once; one exits with status 75, the other serves both), test_crashed_daemon_leftovers_do_not_block_the_next_start, test_daemon_restart_leaves_the_start_to_the_supervisor. Against main's daemon.py, client.py and cli.py all three fail (expected one daemon left running and one exited at once, got [None, None]; no daemon.lock; ccc daemon restart started a daemon under a supervisor). The two e2e tests passed 15/15 repeated runs, and 5/5 with three CPU-bound processes competing.
  • Real binaries, in a throwaway HOME/COCOINDEX_CODE_DIR, counting that dir's ccc run-daemon processes with ps:
Scenario main this branch
two concurrent ccc daemon status, cold 2 daemons, one unreachable; ccc daemon stop leaves it running and then says "Daemon is not running." 1 daemon; the second ccc run-daemon exited 75; ccc daemon stop leaves 0
six concurrent ccc daemon status, cold (the case #243 tests) 6 daemons in 3 of 3 runs; ccc daemon stop leaves 5 running 1 daemon in 3 of 3 runs; all six clients succeed; ccc daemon stop leaves 0
ccc daemon status, then a second ccc run-daemon (the case from #243's thread) the new daemon takes over daemon.pid and the socket; ccc daemon stop stops it, the first keeps running, and a second stop says "Daemon is not running." the new one exits 75 within 0.42 s (3 of 3 runs); daemon.pid and the socket are unchanged; ccc daemon stop leaves 0
ccc mcp + search right after initialize 2 daemons, both PPID 1 after the server exits 1 daemon; search returned results; no zombie under the live server
two ccc index 0.3 s apart 2 daemons in 1 of 3 runs; the unreachable one at 1009 MB RSS with the project's LMDB open 1 daemon in 3 of 3 runs
ccc daemon restart under docker/entrypoint.sh 2 daemons; the supervisor's child unreachable 1 daemon, the supervisor's child; no respawn loop
settings touched, then two concurrent ccc daemon status 2 daemons 1 daemon
  • Lifecycle on this branch: a settings change restarts the daemon (new pid); ccc daemon restart, stop, stop again ("Daemon is not running.") and restart with nothing running behave as before; a SIGKILLed daemon leaves daemon.sock, daemon.lock and daemon.pid behind and the next ccc daemon status starts a new daemon; with idle_timeout_minutes: 1 the daemon idle-exited after 122 s (reason: idle_timeout) and the next command started a new one without a warning.

Notes

  • Windows and macOS were not run locally; the Windows lock relies on msvcrt.locking raising PermissionError on a held byte and releasing it on process exit, so CI's windows-latest job is its first real run.
  • Pre-upgrade daemons don't take the lock, so during an upgrade a new daemon can still displace an old one; the version handshake restarts old daemons anyway.
  • The lock is released right after the listener closes, not at process exit, so a new daemon can start while the old one is still closing projects: the same overlap as today for a client arriving mid-shutdown.
  • CLAUDE.md's Process Model section gains one sentence about the lock.

@hatemadi15
hatemadi15 marked this pull request as ready for review November 9, 2026 19:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant