Skip to content

🤖 fix: harden remote base repo normalization#3496

Merged
ethanndickson merged 7 commits into
mainfrom
workspace-setup-jets
Jun 12, 2026
Merged

🤖 fix: harden remote base repo normalization#3496
ethanndickson merged 7 commits into
mainfrom
workspace-setup-jets

Conversation

@ethanndickson

@ethanndickson ethanndickson commented Jun 9, 2026

Copy link
Copy Markdown
Member

Summary

Hardens SSH/Coder remote workspace setup so Mux treats .mux-base.git as an internal bare common git directory rather than a checkout. Shared base repos are normalized by removing leaked core.bare / core.worktree config, keeping the base repo's own HEAD off user branches, and healing poisoned warm-path caches before materializing new workspaces.

Background

A remote Coder host ended up with .mux-base.git/config containing shared checkout config (core.bare=false and core.worktree=<synthetic path>). Linked worktrees inherit that common config, so a repair that made one workspace look healthy poisoned future sibling workspaces. Separately, leaving the base repo HEAD on trunk makes checkout-aware tools such as Graphite interpret the bare cache as the real trunk checkout.

Implementation

The base repo invariant is now explicit: .mux-base.git is bare by layout, has no shared checkout config, and does not advertise a user branch in git worktree list --porcelain. Empty base repos point HEAD at an unborn internal branch so Git porcelain stays branch-shaped; once Mux selects a real worktree base commit, it detaches the base repo HEAD at that commit. This keeps Graphite from seeing trunk checked out at the base repo while preserving normal Git behavior for worktree add -B, worktree add -b, fsck, and branch -D.

Both the slow sync path and the fused warm fast-path normalize poisoned base repos before checkout. The warm path treats cleanup failures as cache misses so the slow path can run its retry/error handling instead of materializing from still-poisoned shared config. Branch cleanup after workspace deletion keeps Git's own active-worktree guard by neutralizing base HEAD and using git branch -D instead of raw ref deletion; the HEAD rewrite is scoped to common git dirs named .mux-base.git so deleting a worktree of an unmanaged real checkout never strands that checkout on the internal branch. Regression coverage exercises pre-existing poisoned core.bare / core.worktree, neutral base HEAD, warm fast-path healing, missing-object repair, and deleted-worktree branch cleanup.

Validation

Focused SSH integration coverage was run for the poisoned-config, neutral-HEAD, warm-path healing, missing-object repair, and worktree-delete cases. Local static validation also passed with MUX_ESLINT_CONCURRENCY=1 make static-check.

Risks

This touches remote SSH workspace materialization and deletion. The riskiest area is old/corrupted shared base repos, but the change is intentionally defensive: normalization is idempotent, warm-path cleanup failures fall back to the slow path, missing-object repair remains in the checkout path, and branch cleanup is best-effort.


Generated with mux • Model: openai:gpt-5.5 • Thinking: xhigh • Cost: $47.88

Keep SSH remote .mux-base.git repositories from leaking shared checkout config into linked worktrees. The runtime now strips core.bare/core.worktree, keeps the base HEAD on an internal sentinel ref, and creates new worktrees via detached checkout before attaching the workspace branch so the base repo never masquerades as trunk.

---

_Generated with `mux` • Model: `openai:gpt-5.5` • Thinking: `xhigh` • Cost: `$27.40`_

<!-- mux-attribution: model=openai:gpt-5.5 thinking=xhigh costs=27.40 -->
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson ethanndickson added this pull request to the merge queue Jun 11, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 11, 2026
Git versions used by CI can report the intentional refs/mux-internal/base-head sentinel as fsck badHeadTarget. Treat that HEAD-only complaint as healthy for explicit base-repo connectivity probes while preserving missing-object failures.

---

_Generated with `mux` • Model: `openai:gpt-5.5` • Thinking: `xhigh` • Cost: `$29.32`_

<!-- mux-attribution: model=openai:gpt-5.5 thinking=xhigh costs=29.32 -->
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

Follow-up pushed for the CI Git/fsck behavior: base repo connectivity probes now ignore only the expected HEAD-only badHeadTarget complaint for refs/mux-internal/base-head, while preserving real missing-object failures.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: abd4c52851

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/runtime/SSHRuntime.ts Outdated
Avoid git branch -D in SSH base repos because the base HEAD intentionally points at refs/mux-internal/base-head. Delete the workspace branch ref directly so cleanup still works with the sentinel HEAD invariant.

---

_Generated with `mux` • Model: `openai:gpt-5.5` • Thinking: `xhigh` • Cost: `$29.32`_

<!-- mux-attribution: model=openai:gpt-5.5 thinking=xhigh costs=29.32 -->
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

Addressed the branch deletion finding by replacing git branch -D with git --git-dir ... update-ref -d refs/heads/<workspace>, and added coverage that deleteWorkspace cleans the branch ref while the base repo HEAD points at the internal sentinel.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 42693ea000

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/runtime/SSHRuntime.ts Outdated
Keep the sentinel-HEAD-safe update-ref cleanup, but only delete the workspace branch ref after verifying no worktree still reports that branch and by passing the expected old OID.

---

_Generated with `mux` • Model: `openai:gpt-5.5` • Thinking: `xhigh` • Cost: `$29.32`_

<!-- mux-attribution: model=openai:gpt-5.5 thinking=xhigh costs=29.32 -->
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

Addressed the active-branch guard concern: the update-ref cleanup now first verifies no worktree still owns the branch and deletes with the expected old OID.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb6a747899

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/runtime/SSHRuntime.ts Outdated
Use an unborn internal branch only as the empty-repo neutral state, then detach the shared base repo HEAD at the selected worktree base commit when one is available. This keeps Graphite from seeing a user branch checked out at .mux-base.git while preserving Git's normal worktree, fsck, and branch-delete porcelain behavior.

---

_Generated with `mux` • Model: `openai:gpt-5.5` • Thinking: `xhigh` • Cost: `7.50`_

<!-- mux-attribution: model=openai:gpt-5.5 thinking=xhigh costs=37.50 -->
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

I pushed an elegant-redesign pass that keeps the base repo HEAD Git-native: unborn internal branch while empty, detached at the selected base commit once materializing a workspace. This lets the PR keep normal Git porcelain (worktree add -B, worktree add -b, fsck, branch -D) while still ensuring .mux-base.git never advertises a user branch checked out at the base repo.

@ethanndickson

Copy link
Copy Markdown
Member Author

Resolved the remaining Codex thread from the prior update-ref -d approach: the redesign removes raw ref deletion entirely and returns branch cleanup to git branch -D after neutralizing the base repo HEAD, so Git's active-worktree guard is preserved by porcelain instead of reimplemented.

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5dc4ac8588

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/runtime/SSHRuntime.ts Outdated
Treat warm-path base repo config cleanup failures as cache misses so poisoned shared config never leaks into a newly materialized worktree. The slow path already owns retry and hard-error handling for lock conflicts and other cleanup failures.

---

_Generated with `mux` • Model: `openai:gpt-5.5` • Thinking: `xhigh` • Cost: `$47.88`_

<!-- mux-attribution: model=openai:gpt-5.5 thinking=xhigh costs=47.88 -->
@ethanndickson

Copy link
Copy Markdown
Member Author

Addressed the warm-path cleanup feedback: core.bare / core.worktree cleanup now only continues on success or "key absent"; any other cleanup failure becomes WARM_MISS:base-config-normalization-failed, so the slow path handles retry/error semantics before any worktree materialization.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b19d59a3bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/runtime/SSHRuntime.ts Outdated
deleteWorkspace resolves the worktree common git dir from the workspace
itself, which for hand-crafted/legacy worktrees can be a real checkout
.git rather than the Mux-owned .mux-base.git. Rewriting that HEAD onto
the unborn internal branch would strand the user checkout, so gate the
neutralization on the resolved dir basename being .mux-base.git.

---

_Generated with `mux` • Model: `anthropic:claude-fable-5` • Thinking: `xhigh` • Cost: `$56.34`_

<!-- mux-attribution: model=anthropic:claude-fable-5 thinking=xhigh costs=56.34 -->
@ethanndickson

Copy link
Copy Markdown
Member Author

Addressed the P1 (PRRT_kwDOPxxmWM6I_oAn) in cab4098: deleteWorkspace now gates the HEAD neutralization on the resolved common git dir actually being a managed base repo (basename === ".mux-base.git", which covers both the canonical and legacy hashed layouts). For unmanaged common dirs (e.g. a worktree of a real checkout), the symbolic-ref HEAD rewrite is skipped entirely; the best-effort branch -D still runs and remains protected by Git's own checked-out-branch guard. Added an integration regression test ("deleteWorkspace leaves an unmanaged source checkout's HEAD untouched") that creates the workspace as a worktree of a normal checkout and asserts the checkout's HEAD ref and commit survive deletion — verified it fails against the previous code and passes with the fix.

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: cab409839d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: cab409839d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson ethanndickson added this pull request to the merge queue Jun 12, 2026
Merged via the queue into main with commit f1ba2b8 Jun 12, 2026
40 of 42 checks passed
@ethanndickson ethanndickson deleted the workspace-setup-jets branch June 12, 2026 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant