Redo the semantic domain import until it finishes

[imnasnainaec]

Split out of #4352, part 2 of 4 — see
#4352 (comment) for the split and how it was verified.

The database pod's postStart hook imported the semantic domains whenever either collection was empty. A count cannot
tell a finished import from one killed part way through, so an interrupted import left the collections non-empty but
incomplete and was never redone.

• Guard the import with a completion record in CombineDatabase.SemanticDomainImportStatus, written only when the
  import finishes. The record is read by exit status, so anything short of a completed import imports again — a merge,
  and safe to repeat.
• Write the record from update-semantic-domains.sh rather than from the hook, so that the documented manual rerun
  counts too and does not leave the next pod start redoing the whole import.
• Fail the hook on a failed import, so the container restarts and retries. The Combine cannot be used without the
  semantic domains, and a database that serves without them looks healthy.
• Add a readiness probe on a marker the hook writes last, so the pod stays out of the database Service until the
  replica set advertises the current pod IP — which changes on every restart, and which the backend needs since it
  connects with ?replicaSet=rs0.
• Append to /data/db/postStart.log instead of truncating it, stamp each start, and trim to the last 200 lines, since
  the log is on the database's persistent volume with nothing rotating it.

Scope

Four files, none of them touched by parts 1 or 4:

• database/init/update-semantic-domains.sh
• deploy/helm/thecombine/charts/database/templates/database.yaml
• docs/deploy/README.md
• the timeout row of installer/README.md

That last one is a deliberate move from where the split analysis put it. It documents that the installer's timeout has
to cover the semantic domain import, which is a consequence of the readiness probe here, so it belongs with the change
that causes it rather than with part 4. #4353 changes the update row of the same table; the two are verified
conflict-free, as is this branch against #4353 and part 1 as a whole.

QA and prod

Affected, and this is the part to review with prod in mind. The chart template has no target conditionals and
updateStrategy is Recreate on every profile, so on the first upgrade to this chart the database pod is replaced and
then held out of the database Service by the new readiness probe until its postStart hook finishes. An existing QA
or prod installation has no completion record, so that hook imports the semantic domains once more: several minutes
with no database endpoint.

QA and prod deploys run setup_combine.py through .github/actions/combine-deploy-update without --wait, so helm
itself does not block on the import — but the endpoint gap is real and worth planning around. The manual rerun,
kubectl -n thecombine exec deployment/database -- /opt/thecombine/update-semantic-domains.sh, now also writes the
completion record.

Testing

Not yet exercised on hardware. Worth covering both a fresh install, where the import runs inside the hook, and an
upgrade of an installation that already has the domains but no record, since that is the case QA and prod will hit
first.

🤖 Generated with Claude Code

---

This change is 

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9e09254c-4001-42a6-8838-5a3dc663f019

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.96%. Comparing base (2e23f4d) to head (7191cb2).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4355   +/-   ##
=======================================
  Coverage   75.96%   75.96%           
=======================================
  Files         305      305           
  Lines       11384    11384           
  Branches     1411     1411           
=======================================
  Hits         8648     8648           
  Misses       2332     2332           
  Partials      404      404           

Flag	Coverage Δ
backend	87.22% <ø> (ø)
frontend	66.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[imnasnainaec]

Delivery note — publish the release before building an installer

database/init/update-semantic-domains.sh is baked into the database image (database/Dockerfile, into
/opt/thecombine/), while database.yaml ships in the deploy/ tree inside combine-installer.run. This is the PR
where those two stop being independent of each other.

installer_release.yml takes the image tag from gh release view — the latest published release — while
actions/checkout gives it master's deploy/ tree. Those normally coincide, since the installer build follows a
release deploy. But an installer built from master after this merges and before the next release pairs the new chart
with the previous release's images.

In that pairing this breaks hard rather than degrading. e34efeb8 moved the completion-record write out of the
postStart hook and into the script, so a new hook calling an old script records nothing, and every pod start reimports
forever. With #4356 also in, wait-for-semantic-domains never passes: the install fails after
IMPORT_CHECK_TIMEOUT_SECONDS with a manual-import hint that cannot help either, because the manual run is the same old
script.

combinectl update <tag> reaches the same state from the other direction, since it moves image tags and leaves the
charts alone.

Two ways out:

1. Publish the release before building any installer that carries this chart.

2. Restore the hook-side record write as a fallback, after the script succeeds — the line e34efeb8 removed:

   mongosh --quiet --host 127.0.0.1 --eval "db.getSiblingDB('CombineDatabase').SemanticDomainImportStatus.replaceOne({ _id: 'semantic-domains' }, { _id: 'semantic-domains', completed: true }, { upsert: true });"

   Keeping both writes is idempotent, and each covers a case the other does not: the script's write is what makes a
   manual rerun count, the hook's is what makes an older image work.

Option 2 is a one-line change to this PR and removes the ordering constraint entirely, which is the way I would lean.

---

Update: Option 2 applied.