[WIP] CAS draft (adopting to CI/CD, not for review / merge) by filimonov · Pull Request #2073 · Altinity/ClickHouse

filimonov · 2026-07-17T17:36:22Z

Changelog category (leave one):

Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

content addressable storage - draft PR

Documentation entry for user-facing changes

TBD.

Exclude tests:

Regression jobs to run:

github-actions · 2026-07-17T17:37:47Z

Workflow [PR], commit [81642c8]

…ent_addressed stateless lanes) Config Workflow check failed with 'Workflows are outdated' for master.yml, pull_request.yml, pull_request_community.yml, release_builds.yml. Regenerated via 'python3 -m praktika yaml'. The regeneration adds the two CAS stateless jobs to the generated workflows: 'Stateless tests (arm_binary, content_addressed storage, parallel)' and 'Stateless tests (arm_binary, content_addressed s3 storage, parallel)' (the rustfs-backed lane). CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=927ea142c9cb14759623861eb004261d0b4b1c8f&name_0=PR&name_1=Config+Workflow PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

…teless lane The lane's start_rustfs expected a pre-extracted binary at ci/tmp/rustfs and failed on CI runners where nothing provisions it (the workflow wipes ci/tmp on every run). Download the static musl build for the runner architecture from the RustFS GitHub release (1.0.0-beta.9) when the binary is absent, mirroring how setup_minio.sh downloads minio/mc. Validated locally: the beta.9 binary passes the conditional-operation semantics the CA pool requires (second 'If-None-Match: *' PUT -> 412, wrong-etag conditional DELETE -> 412, right-etag DELETE succeeds), and download_rustfs provisions an executable binary end-to-end. PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

Fast test fails at cmake generation: 'Target "dbms" links to ch_contrib::crc32c but the target was not found' — the fast-test job initializes a limited submodule list that does not include contrib/crc32c, so the unconditional add_contrib is skipped while the dbms link line still references the target. The dependency is dead: it was wired in for per-block CRC32C in the early CAS run-file format (5f1272c), which was later replaced by the text record-stream codecs; no source file includes the library today. Restore the pre-CAS state: crc32c is built only for google-cloud-cpp, and dbms does not link it. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=835251f81cb5af73ad9eaa3a835f50f0c8b678db&name_0=PR&name_1=Fast+test PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

Fast test builds without SSL and failed on the unconditional 'openssl/evp.h' include in CasBlobHashingWriteBuffer.cpp. Wrap the OpenSSL-backed Sha256 hashing write buffer and the one-shot digest in '#if USE_SSL'; on non-SSL builds selecting blob_hash = 'sha256' now fails closed with SUPPORT_IS_DISABLED. CityHash128 and XXH3-128 blob hashes are unaffected. PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

… test regression) CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Fast+test PR: #2073 A CAS parser commit grouped `RELOAD_DICTIONARY`/`RELOAD_MODEL`/ `RELOAD_FUNCTION` with `CONTENT_ADDRESSED_GARBAGE_COLLECTION` into a format case that prints only the optional disk, dropping the reload targets: `SYSTEM RELOAD MODEL my_model` formatted as `SYSTEM RELOAD MODEL` (failed 04117_parser_system_query_variants and 04124_parser_system_query_extra in Fast test). Fold all four types back into the generic target-printing case (table / target_model / target_function / disk else-if chain) — for the CA GC command the disk branch produces the identical output. Both stateless tests verified locally via clickhouse-local against their references. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

…als (arm_tidy, T13 batch 1) CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Build+(arm_tidy) PR: #2073 Removes default arguments from all virtual/override methods flagged by `google-default-arguments` (147 sites: `CasBackend.h` interface, `IObjectStorage.h`/`S3ObjectStorage.h`, all backend implementers, test helpers/fixtures) and adds non-virtual convenience overloads on the base classes that forward the previous default values. Derived classes gain `using` declarations to unhide the base overloads. Qualified parent-implementation calls in test fault backends switched to the explicit 3-arg form — the 2-arg form would now route through the base forwarder and re-enter the derived override virtually (double fault injection; caught by the battery). Bulk edits produced by codex (gpt-5.6-luna) per the T13 brief; overload visibility and qualified-call fixes plus verification by Claude. Battery 919/919 green. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Build+(arm_tidy) PR: #2073 Semantics-preserving conformance for the remaining flagged classes: readability-container-contains, readability-isolate-declaration, google-runtime-int (AWS SDK retry-API overrides keep `long` with targeted NOLINT — the override contract owns the type), readability-duplicate-include, cppcoreguidelines-init-variables, cert-msc, modernize-raw-string-literal, modernize-use-starts-ends-with, bugprone-empty-catch (comments only — no new behavior), googletest naming, bugprone-argument-comment, bugprone-optional-value-conversion, bugprone-misplaced-widening-cast (CasTypes.h site audited: not a real precision bug — the value is range-validated to 0-5; cast made explicit without value change). CasRefCowMap's own `contains` keeps its `find` with NOLINT (self-recursion). Bulk edits by codex (gpt-5.6-luna) per the T13 brief (.superpowers/sdd/task-13-batch2-report.md); one over-removed include (PartFolderAccess.h) restored and verification by Claude. Battery 919/919. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

…n't abort the server (STID 0883) CI PR#2073 (content_addressed storage lanes) crashed with "Too large size (9223372036854775870) passed to allocator" (LOGICAL_ERROR, server abort) running the regression test 04070_no_crash_extreme_compress_block _size. Root cause: an extreme max_compress_block_size (2^63-1) flows into ContentAddressedTransaction::writeFile's buf_size and, unclamped, reaches the CaContentWriteBuffer base-class allocation (Memory::alloc), where checkSize (>= 0x8000000000000000) fires. The ordinary MergeTree writers clamp compress -block sizes to 256 MiB (MergeTreeWriterSettings::MAX_COMPRESS_BLOCK_SIZE) for exactly this reason; the CAS write path received the value unclamped. Fix: clamp buf_size and adaptive_write_buffer_initial_size to 256 MiB at the CAS allocation site (both CaContentWriteBuffer ctors), mirroring the ordinary clamp. New gtest CasContentWriteBuffer.ExtremeBufferSizeIsClampedNotPassedToAllocator reproduces the exact crash number without the clamp (verified RED) and passes with it. CA gtest gate 1057/1057. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=latest&name_0=PR Related: #2073

…cle (UNMOUNT/FSCK) Design for STID 3982-3b48 (CI PR#2073). Five parts: (1) MountLeaseKeeper's vanished-backing-store case throws FILE_DOESNT_EXIST (not LOGICAL_ERROR) so a background renewal thread never aborts the server + new CasMountLeaseLost counter; (2) SYSTEM CONTENT ADDRESSED UNMOUNT <disk> (shutdown() the pool, auto-remount on next access) as a clean alternative to rm -rf under a live mount; (3) online SYSTEM CONTENT ADDRESSED FSCK <disk> via read-only runFsck(Pool&); (4) rewrite the no-leftovers test teardown to GC RUN -> FSCK -> UNMOUNT -> rm -rf; (5) rename the offline clickhouse-disks fsck applet to ca-fsck for consistency. Auto-teardown-on-DROP stays out of scope. Related: #2073

…ence Task 7 added the SYSTEM_CONTENT_ADDRESSED_FSCK AccessType but only updated the AccessType.h macro list, not the 01271_show_privileges reference — the test enumerates every privilege, so the new row (right after MOUNT, matching the AccessType declaration order) was missing and fasttest failed with a one-line diff. 01271 was not in the local gtest/FSCK-access test runs, so the gap surfaced only in CI. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=d780fa7473807203916125e96cf73a9774c84464&name_0=PR&name_1=Fast%20test PR: #2073 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb

Add two DESIRABLE items under §3 (GC correctness/observability) surfaced by triaging the CAS-s3 stateless run (Altinity PR #2073, run 30019911967): - [RECOVERED-INDEGREE-ATTRIBUTION] the "delete_pending recovered in-degree — structurally impossible … investigate" warning is a false alarm (dedup-adopt vs condemn TOCTOU, spared, no data loss). Downgrade the GC log to a ProfileEvent + Debug and move the real adopt-without-resurrect detector to the writer's edge-commit. - [CONDEMN-GRACE-WINDOW] a cool-down before condemning a just-zeroed blob to kill hot-dedup churn (tiny system-log blobs) at the source; flagged higher-risk (condemn-timing/ack-floor, TLA-gated, protocol-veto). Docs-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…GABRT) CI-confirmed on Altinity PR #2073 (run 30019911967, asan_ubsan CAS-s3 stateless): a mount-lease renewal PUT that times out client-side (ambiguous — may have applied server-side) gets soft-retried with a stale token, mismatches against its own bumped body, and falls through MountLeaseKeeper's classifier (same uuid/epoch, unfenced — none of the 3 cases match) into the base class's generic LOGICAL_ERROR, aborting the server under ASan. A third variant of the STID 3982-3b48 family (parts 1a/1b covered vanished/absent-at-release). Also notes: amd_msan/amd_tsan CAS-s3 stateless jobs in the same run hit the 6h job timeout with zero artifacts — unknown if same crash-loop or a plain hang. Docs-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The unsharded CA-s3 sanitizer lanes do not fit the 6h GitHub job timeout, and the kill lands after the test loop but before result upload, so the lane reports NO artifacts at all and praktika's aggregate stays RUNNING forever. In run 30203355812 (sha e2d04bf): tsan finished all 10990/10990 tests at 5h59m and was killed 16s later during teardown; msan was at 5824/10990 (53%, steady progress, no hang) when killed; asan_ubsan passed at 5.4h — too close. This is the full explanation of the "msan/tsan CA-s3 cancelled at exactly 6h with zero artifacts" pattern seen in three consecutive runs. Shard asan_ubsan and tsan 2 ways and msan (the slowest, ~11h projected) 3 ways, following the existing "amd_tsan, s3 storage, parallel, N/M" convention; ci/jobs/functional_tests.py already parses the N/M batch token generically. Workflow YAML regenerated with praktika yaml. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

…irectory The refusal in prepareAdoptFromManifest throws LOGICAL_ERROR, which aborts the whole process in debug/sanitizer builds instead of behaving like a catchable exception, so the EXPECT_THROW form killed unit_tests_dbms on all three sanitizer CI lanes (asan_ubsan/tsan/msan, 0s failures with "Logical error: 'Relink target ... does not address a content-addressed part directory of a live table'"). Split it the same way as the CasWiringOpsDeathTest precedent in this file: EXPECT_THROW stays for plain release builds, and a CasWiringExchangeDeathTest EXPECT_DEATH variant proves the same refusals positively abort under DEBUG_OR_SANITIZER_BUILD. Verified: build_asan runs the death test (1 test, OK), build (release) runs the EXPECT_THROW variant (9 tests from 2 suites, all OK). Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

The pre-scrape cleanup removed only filesystem_caches/*/status (one level), but dynamic cache disks created by tests nest their path — the binary CA-s3 lane died on filesystem_caches/disks/cache_03517/status with "Cannot lock file ... Another server instance in same directory is already running" (StatusFile.cpp flock, EWOULDBLOCK) raised from FileCache::initialize when clickhouse-local initialized the disk map. Make the glob recursive so any depth is covered, same mechanism as before: removing the file lets clickhouse-local create and lock a fresh inode even when the not-fully-stopped server still holds the old one. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

…lanes Tag it no-content-addressed-storage: the coverage is the inline content-addressed disk the test creates itself, so it stays meaningful on every ordinary lane. On lanes whose DEFAULT MergeTree storage is content-addressed, system.remote_data_paths (no disk_name pushdown — the applyFilters TODO in StorageSystemRemoteDataPaths.cpp) also walks the huge shared default pool with the whole run's data, and on the S3 (RustFS) variant that walk exceeds the 600s test timeout (recurred in two consecutive runs, plus a "Some queries hung" ride-along). The BACKLOG pushdown item stays open as the real fix; also note the ConnectionGroup Disk-session pressure signal from the same run. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

Same class as 5f292e0: PreparedPartWrite's one-shot-terminal rejection throws LOGICAL_ERROR, which aborts the whole process in debug/sanitizer builds (Exception.cpp's handle_error_code) instead of behaving like a catchable exception, so CasPartFolderAccess.PreparedPartWriteRejectsASecondTerminal killed unit_tests_dbms on all three sanitizer CI lanes (0s failures). The file's author knew the class (the NETWORK_ERROR/MEMORY_LIMIT_EXCEEDED choices further down cite it) — these two spots were the leftovers: - PreparedPartWriteRejectsASecondTerminal: split per the CasWiringOpsDeathTest precedent — expectThrowsCode stays for plain release builds, a CasPartFolderAccessDeathTest EXPECT_DEATH twin proves the same rejections positively abort under DEBUG_OR_SANITIZER_BUILD. - PreparedPartWriteMoveTransfersTheTerminalDuty: the single moved-from abort check is guarded in place (EXPECT_DEATH forks, so the rest of the test continues in the parent). A sweep of all CAS gtest files for unguarded LOGICAL_ERROR throw expectations found no other gaps (blob_upload_pool, ref_writer, ref_install_safety, upload_detached, upload_fanout all carry guards). Verified: build_asan 39/39 with both death tests OK; build (release) 30/30 with the EXPECT_THROW variants. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=8417137aeb842f463066fd68a75b60ffeae30d63&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

CasRefInstallSafetyDeathTest.DenyGuardStopsAnAllocation dispatched on DEBUG_OR_SANITIZER_BUILD, but DENY_ALLOCATIONS_IN_SCOPE itself is gated on MEMORY_TRACKER_DEBUG_CHECKS (MemoryTracker.h: defined only under !NDEBUG). Sanitizer builds define NDEBUG, so there the guard compiles to static_assert(true) and the death test "failed to die" on all three sanitizer CI lanes — visible verbatim in the CI error output. (These lanes only now reached this test at all: the earlier LOGICAL_ERROR aborts in gtest_ca_wiring / gtest_cas_part_folder_access killed the process before it, so each fix un-shadows the next tail failure.) Gate the death test on MEMORY_TRACKER_DEBUG_CHECKS instead, and drop the throw-only EXPECT_ANY_THROW branch as dead code: MEMORY_TRACKER_DEBUG_CHECKS implies !NDEBUG implies DEBUG_OR_SANITIZER_BUILD, so whenever the guard exists its LOGICAL_ERROR aborts — death is the only observable outcome (the old comment claimed the opposite implication). Verified: build_debug runs the death test (OK, guard fires); build_asan compiles it out, remaining CasRefInstallSafety* 21 tests x5 repeats green. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=07f8398acddff2c676aded5f804f122984845a4a&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

RCA of the fence cascades on the CA-s3 sanitizer lanes (run for 07f8398, msan 2/3 lane): clickhouse-server logged 351 bursts of 'Connection refused' (e.code() = 111) to rustfs at localhost:11121 spread over the whole run — the service stayed up between bursts, so this is fd/accept exhaustion, not a crash. rustfs was launched WITHOUT the open-files-limit raise that start_azurite performs for exactly this failure mode, while the server under parallel sanitizer load holds 10k+ active Disk-group S3 sessions. Most bursts were absorbed by the CAS request retries (max_attempts=16 / 90s deadline); the one at ~02:00-02:01 outlived (a) in-flight writes' 90s budgets (Code 210 UNCERTAIN) and (b) the mount-lease renewal confirm window (TTL 30s), so the mount fence tripped (fail-closed by design) and every writing test failed with Code 668/210 from 02:02:52 until remount completed ~02:05:00, with straggler failures to ~02:10 — ~40 of the run's 66 test failures. Also add rustfs.log to the uploaded artifacts: it was already written to ci/tmp/rustfs.log but never uploaded (azurite/kafka/minio logs are), which was exactly the missing evidence for this class of triage. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=07f8398acddff2c676aded5f804f122984845a4a&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

ilejn · 2026-07-28T21:09:16Z

+
+```sql
+SYSTEM CONTENT ADDRESSED GC RUN [ON CLUSTER cluster_name] [disk_name]
+```


GC DISABLE ?

… razor RefWriterStalePrecommitSweep.BoundedBatchesAndInterruptionResumeAcrossMounts failed on 5 of 6 sanitizer-lane runs across two CI rounds (asan/tsan/msan, 'refLaneWedgedForTest — Actual: false, Expected: true') while passing every quiet local run. Reproduced locally 14/14 by adding full CPU load with the test otherwise unchanged and filtered alone — so not test-ordering contamination, pure timing. Root cause: the test set attempt_timeout_ms == operation_deadline_ms == 100, which turns the request controller's pre-send gate (putIfAbsentControlled: 'now + attempt_timeout > deadline' returns Unresolved WITHOUT sending) into a zero-width race — it only passes when no millisecond tick elapses between the deadline capture and the gate. This test uniquely burns that window encoding the ~1700-op removal chunk, so on a loaded or sanitizer-slow machine the gate fired first, the injected ambiguous fault was never reached, the sweep failed CLEAN — nothing sent, nothing ambiguous, and the product CORRECTLY did not wedge the lane. The test had over-specified the failure mode it would meet. Ambiguity is guaranteed by max_attempts = 1 alone; widen operation_deadline_ms to 5000 so the PUT is always actually sent and the wedge is deterministic on any machine. Verified: 20/20 green (5 repeats x 4 sweep tests) on ASan under half-core CPU load; the sibling wedge tests keep the tight budget deliberately — their capture-to-gate window is empty. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=834c9517f56a83927b0c941ac08d8dfd82460579&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

… (PR 2073 review) Addresses the three review recommendations from strtgbb on #2073 - All 10 CAS `Job.ParamSet` entries move out of `JobConfigs.functional_tests_jobs` into a new `AltinityJobConfigs.cas_functional_tests_jobs`. Every workflow that picked them up implicitly -- including `release_builds` (`binary` filter), `release_branches` and `backport_branches` (`asan` filters) -- now concatenates the two lists in the original order, so the job sets are unchanged. Proven by an empty `diff -r` of the regenerated `.github/workflows` and by an identical dump of `workflow.jobs` names for every `ci/workflows/*.py` module before and after. - The CAS ParamSet comments are cut to the load-bearing constraints: RustFS rather than MinIO OSS for enforced conditional deletes, and sharded sanitizer lanes because an unsharded one exceeds the 6h GitHub job timeout. - The CAS comments in `ci/jobs/scripts/clickhouse_proc.py` are compressed the same way, keeping why the disks are opened read-only, why the substitution is keyed on the `<metadata_type>cas</metadata_type>` marker, and why `grep -R` and `sed --follow-symlinks` are required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN

CASRefCheckpointJoin.EncodedCkptSizeIsIndependentOfCardinality and CASRefWriterStalePrecommitSweep.BoundedBatchesAndInterruptionResumeAcrossMounts failed identically on all three sanitizer unit-test lanes (asan_ubsan, tsan, msan) at 9ad3e15, thrown from ordinary product code, not a sanitizer report -- both are the same class of bug `8f9e63c7a19` already fixed once for the second test: a CPU-bound encode step (committing `MANY_REFS`/ ~`ref_txn_max_ops` ref ops in one call) measured against a real-wall-clock deadline (`mount_lease_ttl_ms` for the first test, `operation_deadline_ms` for the second), which a slow-enough sanitizer build -- msan above all -- can outrun regardless of how wide the deadline is set. Both the mount fence and the ref-log request controller (`CasRefLedger`'s `controller_boot_ms_fn`) already read time through `PoolConfig::boot_ms_fn`, the same injectable seam a few lines below already uses (the `resumer` pool in the second test). Freeze it for the pools that do the CPU-bound work instead of widening the deadline further, removing the race rather than giving it more room. Not build-verified: the shared dev box was at 63 GB free disk (existing full sanitizer builds run 43-51 GB each) and near swap exhaustion, so no fresh build was attempted. Verified by tracing the exact production wiring (PoolConfig::boot_ms_fn -> CasRefLedger::controller_boot_ms_fn -> CasRequestController::now_ms, and CasMountRuntime::bootMsNow) and matching the pattern already in use for this file's `resumer` pool. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=9ad3e15b68823633cc1c2a1539292f20de3e7b5e&name_0=PR PR: #2073 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN

`gcs_max_conditional_put_bytes` is a property of the GCS conditional-write dialect, not a CAS policy: it takes effect only on a generation-token store, and what CAS owns is the policy that such a write must not go multipart. That policy is already a per-write flag. Only the number was in the wrong place. The setting moves to `S3AuthSettings`, beside `gcs_issue_compose_request`, keeping its config spelling, and `S3ObjectStorage` applies it itself when the write carries the flag. `WriteSettings::s3_single_part_upload_max_bytes_override` had exactly one producer and disappears with it. The key is added to the CAS `non_cas_keys` skip-list in this commit so that the still-scanning CAS loader keeps accepting it; the whole list is removed next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A CAS disk block is shared with the object storage, the generic disk layer and the proxy resolver, and the CAS settings were the only consumer in it that scanned every key and rejected whatever was not in a hand-written skip-list. Any legal key nobody had enumerated failed server startup: all `s3_*` request settings, most S3 client settings, repeated `<header>` elements, `<proxy>`, and every Azure key, so a CAS disk over Azure could not be configured at all. The CAS settings now live under a `cas_` config-key prefix and nothing else in the block is read, so the skip-list is gone with nothing in its place. A mis-spelled CAS setting is still rejected, because that check is now made over a namespace CAS actually owns. The unprefixed spelling is accepted for a bounded period, since configurations using it already exist in external CI/CD scripts; each disk reports its superseded keys once. A key written in both spellings, or twice in one spelling, is rejected rather than resolved silently. `skip_access_check` keeps its bare spelling and stops being a CAS setting: the generic disk layer reads the same key, and one key must not become two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Classify duplicate and ambiguous `cas_` settings before parsing values, logging migration success, or changing the settings object. This keeps malformed values from masking the required fail-closed exceptions.

Validate every owned `cas_` and legacy setting in a temporary settings state before reporting migration success or changing the live configuration.

Run `loadFromConfig` post-processing and semantic validation on the temporary CAS settings state before reporting migration success or replacing the live state.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`http_keep_alive_timeout` in a CAS disk block used to fail server startup, which is what blocked the mitigation suggested in #2243. The CAS integration disk now carries that key and five more of the same class, so the lane fails if the disk block ever stops accepting them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Operator-facing XML snippets still advertised `server_root_id` and `blob_hash_allow_new` after CAS keys moved to the `cas_` namespace. Use `cas_server_root_id` and `cas_blob_hash_allow_new`, and pin the hash-admission exception in the pluggable-hash suite.

Report externally configured CAS settings with their `cas_*` names in validation, parser failures, staging capability checks, and mount-recovery guidance. Preserve the original prefixed spelling for unknown keys while keeping internal setting names and persisted fields unchanged. Refresh stale integration comments to describe namespace ownership instead of the deleted `non_cas_keys` list.

(cherry picked from commit 058b678)

(cherry picked from commit f76335f)

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com> (cherry picked from commit cc73c9b)

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com> (cherry picked from commit 3f288c1)

(cherry picked from commit eac9fc9)

(cherry picked from commit 5d14298)

…eservation` release at query finish" This reverts commit cfd11fdcdebbf1e196c8c4e0d0aa4dc5eeb3b7b2. The backport is correct and 26.6 wants it -- it fixes a `MemoryReservation` use-after-free that TSan reports in `Stress test (amd_tsan)`, left behind when cherry-pick ClickHouse#108562 was auto-closed with conflicts on 2026-07-06. But it was applied directly onto this content-addressed-storage development branch instead of onto `antalya-26.6`, so it exists on no upstream ref: it reaches this branch through the first parent of the upstream merge, not through the merge itself. That makes it invisible to the release branch while silently diverging eight upstream-scope files (`Context`, `ProcessList`, `executeQuery`, `BlockIO`, and their test) between this branch and every branch carved from it, and it renames `releaseWorkloadResources` to `releaseQuerySlot` across `src/Interpreters` and `src/QueryPipeline` -- an upstream API rename carried by a CAS branch. Reverted here so the two branches are comparable again. The fix itself is not abandoned: it belongs in its own pull request against `antalya-26.6`, from where this branch will receive it through the next upstream merge. The original commit remains in history for that purpose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

CI runs newly added stateless tests several times against the same server (https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=5828352eddfa33a42915e26314999c358b84db95&name_0=PR, #2073): the first run passed and every repeat failed. 05024_cas_freeze_two_roots reused a fixed table UUID, so a repeated run collided with the previous run's asynchronous cleanup twice over: the Atomic database still held the UUID mapping (`TABLE_ALREADY_EXISTS`), and once it let go, the CAS namespace was still waiting for its terminal fold (`CAS write could not be committed ... is Removing`), which never comes with the test's GC interval. The UUID is now generated per run; within a run the two roots still share it, which is all the cross-root lookup needs. 05025_cas_attach_partition_cross_disk gave its local disks fixed names with a database-dependent path. Custom disks live for the whole server lifetime, so the repeated run's CREATE was a redefinition with different settings (`The disk ... is already configured as a custom disk in another table`). The database is now part of the disk name as well. Verified locally with `--count 3` (each test 3 times in one server): failing before, 6/6 OK after. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The asan_ubsan unit-test job aborted at process exit with a heap-use-after-free in `Poco::Logger::shutdown`: a logger channel was released once more than its reference count (https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=5828352eddfa33a42915e26314999c358b84db95&name_0=PR, #2073). Bisection landed on `ScopedCasSettingsLogCapture`: its `old_channel` member was a `Poco::AutoPtr` constructed from `logger->getChannel()`, and that constructor takes ownership WITHOUT calling `duplicate` -- every capture stole one reference from whatever channel the logger had at the time. In a full run the victim was the root `ConsoleChannel` installed by `CoordinationChangelogTest::SetUp`, so the double release only surfaced at exit; in a reduced run the second capture dropped the count to zero inside its own constructor and the destructor's `duplicate` touched freed memory. Fix: construct the `AutoPtr` with `shared=true`, which buys the reference the fixture holds. The sibling fixtures that parked the previous channel as a raw pointer are swept to the same shape: a raw parked pointer is count-balanced, but nothing kept the channel alive while the capture channel was installed, so the restore could resurrect a dangling pointer. Verified: `CASContentAddressedSettings.* + *CoordinationChangelog*` under ASan reproduced the use-after-free before and is clean after; the full unit suite (CI filter) now exits 0 with 12037 tests passed and no ASan report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The tsan unit-test job failed `CASShutdownExitTest.EmitAfterResetSharedContextExitsCleanly` (https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=5828352eddfa33a42915e26314999c358b84db95&name_0=PR, #2073): the default "fast" death-test style forks the heavily multithreaded, sanitizer-instrumented test runner, and this exit test then builds a real `LocalObjectStorage`-backed storage inside the child, where a filesystem call failed spuriously (`weakly_canonical: Invalid argument` from the bootstrap's residual LIST) and the fail-closed refusal escaped the death statement. The "threadsafe" style re-executes the binary instead, so the child never inherits the runner's threads (same approach as `gtest_backup_info.cpp`). Also make `makeLocalObjectStorageForTest` fail loudly when it cannot clear or create its scratch root instead of discarding the error code and letting the storage layer report something bewildering later. Verified under ASan: the `CAS*` battery passes (2186 tests, exit 0) with the death tests going through the re-exec path. Not verified under TSan locally (no fresh tsan build); the next CI push measures that lane. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ection stays cheaper The test did not pin `index_granularity`, so the projection was not reliably cheaper than the base table and the test flapped in CI. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073 PR: #2073 Squashed from: 95ba719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

The "proven absent" presence probe and the cold-reader admission both required two whole-catalog reads to be byte-identical before trusting an observation. The catalog is pool-global, so under a parallel workload the condition almost never holds and both paths starve: in CI, `01069_database_memory` failed 193 of 194 retries with "catalog changed while probing table-root cleanup completeness" on its `deduplication_logs` namespace. Each catalog read is one token-CAS'd full value, so this namespace's row is what the proof actually needs: - presence probe: row absent in BOTH reads => proven absent, linearized at the second read; a row that appears in between answers present; - cold reader: revalidate THIS row by value; unrelated churn admits, an own-row change still forces a fresh observation. The second cut now runs the ambiguity validation explicitly -- an aliasing incarnation admitted between the reads left the target row byte-identical and was only refused by the whole-catalog comparison it replaces. Two tests pinning whole-catalog stillness updated to the per-row contract; regression tests added for unrelated-churn starvation and the aliasing incarnation. PR: #2073 Squashed from: 684161d Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

Four independent defects, all of them in test or harness code, none in the product. **The mount-renewal integration test could not start at all.** Its appended compose file bind-mounted the fault-injecting proxy from `utils/ca-soak/proxy/`, a soak-harness path that is deliberately not part of this branch. The mount resolved to nothing, so `python3 /proxy/s3_fault_proxy.py` failed immediately, the `s3proxy` healthcheck never passed, `node` never started behind its `condition: service_healthy`, and `cluster.start()` threw in the module fixture. The test is now self-contained: the proxy script lives beside it and the bind is relative to the test directory. `utils/ca-soak/proxy/` keeps its own copy for the soak harness. **The `FREEZE` and cross-disk `ATTACH PARTITION FROM` tests were not re-runnable.** `05024_cas_freeze_two_roots` and `05025_cas_attach_partition_cross_disk` left state that made a second run in the same server fail, so a retry could not distinguish a real regression from leftovers. **CAS gtest fixtures leaked a parked logger channel.** The fixtures parked a Poco logger channel without holding a real reference to it, so the channel could be released while still installed -- an AutoPtr steal that ASan reports as a use-after-free. Five fixtures now hold a genuine reference. **The exit tests forked a multithreaded sanitizer-instrumented runner.** The tsan unit-test job failed `CASShutdownExitTest.EmitAfterResetSharedContextExitsCleanly`: the default "fast" death-test style forks the test runner, and the exit test then builds a real `LocalObjectStorage`-backed storage inside that child, where a filesystem call failed spuriously (`weakly_canonical: Invalid argument` from the bootstrap's residual LIST) and the fail-closed refusal escaped the death statement. These tests now use the "threadsafe" style, which re-executes the binary so the child never inherits the runner's threads -- the same approach as `gtest_backup_info.cpp`. This removes the whole hazard class rather than one symptom: a forked child of a threaded process can also hang outright if it inherits a held lock, which is what a `std::exit` in such a child did before `d7a02b45f23` replaced it with `std::_Exit`. `makeLocalObjectStorageForTest` also stops discarding the error code when it cannot clear or create its scratch root, so a broken fixture says so instead of letting the storage layer report something bewildering later. Verified under ASan: the `CAS*` battery passes (2186 tests, exit 0) with the exit tests going through the re-exec path. The tsan lane was not reproduced locally (no fresh tsan build); the next CI push measures it. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=5828352eddfa33a42915e26314999c358b84db95&name_0=PR PR: #2073 Squashed from: 0e623d2 d2a0599 81642c8 (plus the compose bind-mount repair, authored here: the soak-harness path it referenced does not exist on this branch.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

filimonov mentioned this pull request Jul 17, 2026

[WIP] CAS draft (adopting to CI/CD, not for review / merge) #2066

Closed

ilejn reviewed Jul 28, 2026

View reviewed changes

alsugiliazova mentioned this pull request Jul 30, 2026

CAS — consolidated static analysis audit findings (tracking) #2031

Open

svb-alt added antalya antalya-26.6 roadmap Key features and improvements for Antalya project labels Jul 30, 2026

strtgbb requested changes Jul 31, 2026

View reviewed changes

Comment thread ci/defs/job_configs.py Outdated

Comment thread ci/defs/job_configs.py Outdated

Comment thread ci/jobs/scripts/clickhouse_proc.py Outdated

filimonov and others added 26 commits August 25, 2026 11:06

fix: validate CAS config keys before applying them

3ed7f5f

Classify duplicate and ambiguous `cas_` settings before parsing values, logging migration success, or changing the settings object. This keeps malformed values from masking the required fail-closed exceptions.

fix: load CAS disk settings transactionally

aae83c1

Validate every owned `cas_` and legacy setting in a temporary settings state before reporting migration success or changing the live configuration.

fix: validate CAS settings before committing them

9fad804

Run `loadFromConfig` post-processing and semantic validation on the temporary CAS settings state before reporting migration success or replacing the live state.

test: move the XML CAS disk configurations to the cas_ prefix

71401fc

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

test: move the inline CAS disk() stateless tests to the cas_ prefix

2671092

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

test: move the Python-assembled CAS configurations to the cas_ prefix

07d7416

test: move the CAS config literals in C++ tests to the cas_ prefix

d1f8fe9

docs: document the cas_ config-key prefix and its migration window

3522179

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs: complete CAS config-key namespace coverage

a923b98

docs: clarify CAS scratch-path defaults

65fcfce

Fix CI Tests fail: Ignore unknown config keys in ClickHouseService

3e7037b

(cherry picked from commit 058b678)

remove ci tests from blocking jobs

080b17c

(cherry picked from commit f76335f)

Skip the EXPORT PART stateless tests on CAS storage

9ffc620

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com> (cherry picked from commit cc73c9b)

Shard the CAS aggregate_functions, lightweight_delete and alter jobs

269e77c

Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com> (cherry picked from commit 3f288c1)

Update CAS jobs and add CASS3Cache configurations

243f063

(cherry picked from commit eac9fc9)

Update job names to include 's3_cache' prefix

5828352

(cherry picked from commit 5d14298)

Merge antalya-26.6 into cas-gc-rebuild

6ab9794

filimonov mentioned this pull request Aug 26, 2026

[experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 #2159

Merged

28 tasks

Navigation Menu

Uh oh!

[WIP] CAS draft (adopting to CI/CD, not for review / merge) - #2073

[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2073
filimonov wants to merge 4474 commits into
antalya-26.6from
cas-gc-rebuild

filimonov commented Jul 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

ilejn Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

filimonov commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilejn Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

filimonov commented Jul 17, 2026 •

edited

Loading

github-actions Bot commented Jul 17, 2026 •

edited

Loading