From 8e60d0263725a9df984f89811fb7dc69ee0eb345 Mon Sep 17 00:00:00 2001 From: Boris Tyshkevich Date: Tue, 25 Aug 2026 16:43:43 +0200 Subject: [PATCH] Simplify export-system-logs article The article was 3313 words against a section median of 498 - the longest in Setup & maintenance by 3x, and much of the length was ClickHouse internals that an operator does not need: SystemLogQueue::pop() and queue->confirm() semantics, C++ stack traces, and a measurement table justifying the findings rather than stating them. Cut to 1278 words. Every tested recipe is kept; the reasoning behind them is replaced by the operational rule. - Replace the flush-mechanism explanation with the consequence: a failed flush is not retried, so the rows are gone. - Drop both stack traces and the push-time vs sink-construction analysis. The rule that matters is "S3 target failures lose local rows, URL target failures do not". - Collapse the 4-row measurement table into one sentence with the same numbers. - Drop the reconciliation / Distributed-sender section - design prose that made no tested claim. - Merge the References list into Related resources. - Reflow the DDL onto fewer lines. All four DDL blocks were re-executed verbatim on 26.7.5.10 after the reflow, plus the glob read-back, monitoring and upgrade-check queries: object sequence, session event types, file append and dependency output all match what the article states. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01AsB1E5eSBXpaZT4jGzY57L --- .../altinity-kb-export-system-logs.md | 436 ++++-------------- 1 file changed, 89 insertions(+), 347 deletions(-) diff --git a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-export-system-logs.md b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-export-system-logs.md index 58e2399743..5fcac6cbff 100644 --- a/content/en/altinity-kb-setup-and-maintenance/altinity-kb-export-system-logs.md +++ b/content/en/altinity-kb-setup-and-maintenance/altinity-kb-export-system-logs.md @@ -10,33 +10,21 @@ keywords: - clickhouse session_log --- -ClickHouse® records query activity in `system.query_log` and authentication activity in `system.session_log`. There is no separate "audit log file" switch for these structured logs — they are system tables, not files. Getting them into S3, an HTTP collector, or a SIEM therefore means exporting the tables. +`system.query_log` and `system.session_log` are tables, not log files, so shipping them to S3, an HTTP collector, or a SIEM means exporting the table. -There are two ways to do it: +Keep the local system log and attach a materialized view (MV) that writes to an external target. Do not replace the system log engine itself unless you accept losing the local copy — see [Direct external engine](#direct-external-engine). -1. keep the normal local system log table and attach an incremental materialized view (MV) that sends each flushed block to an external target; or -2. replace the system log table engine with an external engine such as `S3` or `URL`. - -**Use the first pattern.** It keeps a local queryable copy. But it is not free of risk either: a broken export can destroy local log data, for reasons explained below. - -Everything here was tested on 26.7.5.10, 26.4.5.143, 26.3.21.7 (LTS), 25.8.32.4, and 24.10.1.2812 against MinIO and a local HTTP collector. Observed object names, paths, and error text are pasted verbatim. +Tested on 26.7.5.10, 26.4.5.143, 26.3.21.7 (LTS), 25.8.32.4 and 24.10.1.2812 against MinIO. {{% alert title="Warning" color="warning" %}} -A materialized view pointing at an `S3` target can permanently delete rows from your local `system.query_log`. This is not theoretical: with a misconfigured S3 target, **1 of 20 test queries survived** in the local table; with an unreachable S3 endpoint, **0 of 10**. See [When the export destroys the local log](#when-the-export-destroys-the-local-log) before deploying this pattern. -{{% /alert %}} - -## How system log flushing works - -System logs are buffered in memory and flushed to their system tables in batches. For `query_log` the default `flush_interval_milliseconds` is 7500 ms; size thresholds can trigger an earlier flush, and `SYSTEM FLUSH LOGS` forces one. +A **broken S3 export deletes rows from your local `system.query_log`.** The failing insert kills the whole flush, and failed flushes are never retried. Measured on 26.7.5.10: with a working target 20 of 20 test queries were logged locally; with a misconfigured S3 target only **1 of 20**; with an unreachable S3 endpoint **0 of 10**. Same on 26.3 and 25.8. -Two properties of the flush loop matter for auditing: - -- A system-log table can be given a full table `engine` expression in the server config. By default the persistent system logs use `MergeTree`. -- **A failed flush is not retried.** There is no retry mechanism at all — not one that gets skipped on error. `SystemLogQueue::pop()` detaches the batch from the queue before the flush is attempted (`result.logs.swap(queue)` after `queue_front_index += queue_size`, in [`SystemLogBase.cpp`](https://github.com/ClickHouse/ClickHouse/blob/master/src/Common/SystemLogBase.cpp)), so those rows are already unrecoverable by the time the insert runs. `SystemLog::flushImpl` then catches any exception, increments `SystemLogErrorOnFlush`, and logs `Failed to flush system log ...`; `queue->confirm()` afterwards only advances the counter that `SYSTEM FLUSH LOGS` waits on. A batch that fails to flush is simply gone. +`URL` targets do not have this problem — a dead collector loses nothing locally (25.8+). +{{% /alert %}} -For query auditing, also check the effective settings for the audited users: `log_queries=1` enables query logging and `log_queries_probability=1` avoids sampling. +## Before you start -`system.session_log` is **off by default** in self-managed ClickHouse. Verified on 26.7.5.10 and 25.8.32.4: without the config section, `EXISTS TABLE system.session_log` returns `0`. Enable it before relying on it: +`system.session_log` is disabled by default. Enable it, then restart: ```xml @@ -49,98 +37,13 @@ For query auditing, also check the effective settings for the audited users: `lo ``` -## Recommended pattern: local system log plus materialized view - -```text -query/session event - | - v -system-log in-memory buffer - | - | flush interval / size threshold - v -system.query_log or system.session_log (local MergeTree) - | - | incremental materialized view - v -external target (S3 / HTTP / file) -``` - -The MV must be created **after** the system log table exists. On a freshly started server the table is created only on the first flush, and the MV DDL fails: - -```text -Code: 60. DB::Exception: Unknown table expression identifier 'system.query_log' -in scope SELECT ... FROM system.query_log. (UNKNOWN_TABLE) -``` - -Run `SYSTEM FLUSH LOGS` once first. - -### When the export destroys the local log - -Since 23.3 the system-log flush sets `materialized_views_ignore_errors=true`, and this is usually described as making MV export failures harmless to the local table. That is the stated intent — the setting is applied with the comment *"We always want to deliver the data to the original table regardless of the MVs"* — but it is only half true, and the half that is false loses data. - -`materialized_views_ignore_errors` covers exceptions thrown while **pushing rows** through an already-built view chain. It does not cover exceptions thrown while **building the target sink**. The S3 sink checks whether the destination object already exists at construction time, so its errors are raised early and escape the guard. The captured stack trace on 26.7.5.10 is: - -```text -DB::checkAndGetNewFileOnInsertIfNeeded(...) -DB::StorageObjectStorage::write(...) -DB::InsertDependenciesBuilder::createSink(...) -DB::InsertDependenciesBuilder::createPostSink(...) -DB::InterpreterInsertQuery::execute() -``` - -The exception therefore fails the whole system-log flush, and because a failed flush is not retried, the batch never reaches the local `MergeTree` table either. - -Measured on 26.7.5.10, 20 marker queries with one flush each: - -| MV target state | Markers surviving in local `system.query_log` | -|---|---| -| No MV at all (control) | 20 / 20 | -| MV to a working S3 target (control) | 20 / 20 | -| MV to S3, `s3_create_new_file_on_insert` not effective | **1 / 20** | -| MV to S3, endpoint unreachable | **0 / 10** | - -The same loss reproduces on 26.3.21.7 and 25.8.32.4. `SystemLogErrorOnFlush` incremented on every failed flush, and the server log shows: - -```text - ... SystemLog::flushImpl(...): Failed to flush system log -system.query_log with 2 entries up to offset 14: Code: 36. DB::Exception: Object in bucket -audit with key node-a/query-log.ndjson already exists. If you want to overwrite it, enable -setting s3_truncate_on_insert, if you want to create a new file on each insert, enable -setting s3_create_new_file_on_insert. (BAD_ARGUMENTS) -``` - -`URL` targets behave differently and better. With the collector returning HTTP 503, and again with the collector refusing connections, all 10 of 10 markers survived locally on 26.7.5.10, 26.3.21.7, and 25.8.32.4, and `SystemLogErrorOnFlush` stayed at 0 — those errors occur at row-push time and are correctly ignored. The rows were simply absent from the collector, which is the intended best-effort behavior. - -On 24.10.1.2812, which predates [PR #75679](https://github.com/ClickHouse/ClickHouse/pull/75679), the same connection-refused test left **0 of 10** markers in the local table — every batch was lost — with the exception escaping through `DB::buildPushingToViewsChain(...)`. The fix was backported to 24.12.6.70 and 25.1.6.34 and is in 25.2.1.3085 and later. - -{{% alert title="Warning" color="warning" %}} -Before enabling an S3-backed MV export in production, verify the target actually accepts a second insert. Test it, confirm `SystemLogErrorOnFlush` stays at zero, and monitor it afterwards. An export that starts failing silently takes your local query history with it. -{{% /alert %}} - -```sql -SELECT value -FROM system.events -WHERE event = 'SystemLogErrorOnFlush'; -``` - -An empty result means the event has never incremented, which is the healthy state. The event was added in **25.2** ([PR #75466](https://github.com/ClickHouse/ClickHouse/pull/75466); the changelog lists it under 25.3, since changelogs are written after the tag is cut). On earlier releases this query returns no row regardless of health — verified absent in 24.10.1.2812, present in 25.8.32.4. +Create the MV only after the system log table exists — on a fresh server it appears at the first flush, so run `SYSTEM FLUSH LOGS` once, or the DDL fails with `UNKNOWN_TABLE`. -{{% alert title="Note" color="info" %}} -The event's own description in `system.events` reads *"Number of times any of the system logs have failed to flush to the corresponding system table. Attempts to flush are repeated."* The final sentence is wrong for this path: the failed batch is never re-flushed. Do not treat a non-zero value as a transient condition that will resolve itself — every increment is lost log data. -{{% /alert %}} +Also confirm the audited users have `log_queries=1` and `log_queries_probability=1`. -## Example: export query logs to S3 as JSONEachRow +## Export to S3 -Two settings decisions have to be right, and the obvious placements for one of them do not work. - -S3 objects are not appendable. The S3 engine refuses an insert into an object that already exists, so every flush after the first fails unless `s3_create_new_file_on_insert=1` makes each insert create a new object with a numeric suffix. - -That setting must be applied **in a settings profile**, in `users.d`. Verified on 26.7.5.10: - -- `CREATE TABLE ... ENGINE = S3(...) SETTINGS s3_create_new_file_on_insert = 1` — the clause is **silently dropped**. It does not appear in `SHOW CREATE TABLE`, and the second flush fails. -- `CREATE MATERIALIZED VIEW ... AS SELECT ... SETTINGS s3_create_new_file_on_insert = 1` — the clause is preserved in `SHOW CREATE TABLE` but has **no effect** on the write into the target table. The second flush still fails. The MV `SETTINGS` clause applies to the view's `SELECT`, not to the insert into its target. -- A profile default **works**. +S3 objects cannot be appended, so each flush must create a new object. `s3_create_new_file_on_insert=1` does that — but **only from a settings profile**: ```xml @@ -153,11 +56,16 @@ That setting must be applied **in a settings profile**, in `users.d`. Verified o ``` -{{% alert title="Note" color="info" %}} -This is a profile-wide default: it changes behavior for every S3 insert made by users on that profile, not just the audit export. Apply it to the profile the audit export runs under and review the effect on other S3 writes. Do not substitute `s3_truncate_on_insert`, which overwrites the previous object instead of adding one. +{{% alert title="Warning" color="warning" %}} +Putting this setting anywhere else silently fails and triggers the data loss above: + +* `CREATE TABLE ... ENGINE = S3(...) SETTINGS s3_create_new_file_on_insert = 1` — dropped without error, absent from `SHOW CREATE TABLE`. +* `CREATE MATERIALIZED VIEW ... SETTINGS s3_create_new_file_on_insert = 1` — kept in `SHOW CREATE TABLE`, but ignored for the write into the target. + +The setting is a profile-wide default, so it affects other S3 inserts by users on that profile. Do not use `s3_truncate_on_insert` instead — it overwrites the previous object. {{% /alert %}} -For credentials, use a named collection so keys stay out of the DDL: +Keep credentials in a named collection: ```xml @@ -171,184 +79,128 @@ For credentials, use a named collection so keys stay out of the DDL: ``` -The example exports a stable subset of `system.query_log` rather than `SELECT *`: system-log schemas change across releases, and selecting only the audited fields reduces coupling. Replace `NODE_ID` with a value unique to each server. +Export a fixed column list rather than `SELECT *` — system log schemas change between releases. Replace `NODE_ID` with a per-server value. ```sql CREATE DATABASE IF NOT EXISTS audit; CREATE TABLE audit.query_log_s3 ( - hostname String, - type String, - event_time_microseconds DateTime64(6), - query_id String, - initial_query_id String, - is_initial_query UInt8, - user String, - address String, - query String, - exception_code Int32, - exception String + hostname String, type String, event_time_microseconds DateTime64(6), + query_id String, initial_query_id String, is_initial_query UInt8, + user String, address String, query String, + exception_code Int32, exception String ) ENGINE = S3(audit_s3, filename = 'NODE_ID/query-log.ndjson', format = 'JSONEachRow'); CREATE MATERIALIZED VIEW audit.query_log_to_s3 TO audit.query_log_s3 AS SELECT - toString(hostname) AS hostname, - toString(type) AS type, - event_time_microseconds, - query_id, - initial_query_id, - is_initial_query, - user, - toString(address) AS address, - query, - exception_code, - exception + toString(hostname) AS hostname, toString(type) AS type, event_time_microseconds, + query_id, initial_query_id, is_initial_query, + user, toString(address) AS address, query, exception_code, exception FROM system.query_log; ``` -Each flush produces one object, with the suffix inserted before the extension: - -```text -NODE_ID/query-log.ndjson -NODE_ID/query-log.1.ndjson -NODE_ID/query-log.2.ndjson -``` - -Read the whole export back with a glob: +Each flush adds an object: `query-log.ndjson`, `query-log.1.ndjson`, `query-log.2.ndjson`, ... Read them back with a glob: ```sql -SELECT count() -FROM s3(audit_s3, filename = 'NODE_ID/query-log*', format = 'JSONEachRow', - structure = 'query String, type String'); +SELECT count() FROM s3(audit_s3, filename = 'NODE_ID/query-log*', + format = 'JSONEachRow', structure = 'query String, type String'); ``` {{% alert title="Warning" color="warning" %}} -Never point two servers at the same object-name sequence. Two ClickHouse servers inserting concurrently into one S3 key sequence silently lose rows: in a test on 26.7.5.10, two servers made **300 inserts that were all acknowledged with zero errors, but only 267 objects existed afterwards** — 33 rows vanished. Per the analysis in [issue #112419](https://github.com/ClickHouse/ClickHouse/issues/112419) (open as of 26.7.5.10), the free-suffix probe is a check-then-act race, so both writers settle on the same `key.N` and one overwrites the other. `s3_create_new_file_on_insert` does not protect against this — it is the mechanism that races. Give every server its own prefix, which is required anyway since system logs are per-server. +Never point two servers at the same object name. Concurrent writers on one key sequence lose rows silently: two servers made 300 inserts, all acknowledged with no errors, and left only 267 objects ([issue #112419](https://github.com/ClickHouse/ClickHouse/issues/112419), open as of 26.7.5.10). Give every server its own prefix. {{% /alert %}} -## Example: export session logs to S3 +### Session log ```sql CREATE TABLE audit.session_log_s3 ( - hostname String, - type String, - event_time_microseconds DateTime64(6), - user String, - auth_type String, - client_address String, - client_port UInt16, - interface String, - failure_reason String + hostname String, type String, event_time_microseconds DateTime64(6), + user String, auth_type String, client_address String, client_port UInt16, + interface String, failure_reason String ) ENGINE = S3(audit_s3, filename = 'NODE_ID/session-log.ndjson', format = 'JSONEachRow'); CREATE MATERIALIZED VIEW audit.session_log_to_s3 TO audit.session_log_s3 AS SELECT - toString(hostname) AS hostname, - toString(type) AS type, - event_time_microseconds, - ifNull(user, '') AS user, - ifNull(toString(auth_type), '') AS auth_type, - toString(client_address) AS client_address, - client_port, - toString(interface) AS interface, - failure_reason + toString(hostname) AS hostname, toString(type) AS type, event_time_microseconds, + ifNull(user, '') AS user, ifNull(toString(auth_type), '') AS auth_type, + toString(client_address) AS client_address, client_port, + toString(interface) AS interface, failure_reason FROM system.session_log; ``` -Tested on 26.7.5.10 and 25.8.32.4: `LoginSuccess`, `LoginFailure`, and `Logout` all export, with `failure_reason` populated for failures. - -The `ifNull()` calls are not cosmetic. `system.session_log.user` and `auth_type` are `Nullable`. The MV will accept the DDL without them — ClickHouse converts `Enum8`, `IPv6`, `LowCardinality(String)`, and `Nullable(String)` to `String` implicitly — but a row with an actual `NULL` then raises: - -```text -Code: 349. DB::Exception: Cannot convert NULL value to non-Nullable type: ... -while pushing to view audit.session_log_to_s3. (CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN) -``` - -That is a push-time error, so the flush is ignored and the local table is safe — but the row disappears from the export with no error surfaced to anyone. Keep the `ifNull()` calls, or declare the target columns `Nullable`. +Exports `LoginSuccess`, `LoginFailure` and `Logout`. Keep the `ifNull()` calls — `user` and `auth_type` are `Nullable`, and a `NULL` reaching a non-`Nullable` column drops that row from the export without any error. -## HTTP/HTTPS target +## Export to HTTP -The `URL` engine maps an `INSERT` to an HTTP `POST`. The receiver must accept the posted format and handle `Transfer-Encoding: chunked`, which is what ClickHouse sends. +The `URL` engine turns each flush into a `POST`; the collector must accept chunked transfer encoding. ```sql -CREATE TABLE audit.query_log_url -( - query String, - type String, - event_time_microseconds DateTime64(6) -) +CREATE TABLE audit.query_log_url (query String, type String, event_time_microseconds DateTime64(6)) ENGINE = URL('https://audit-collector.example/clickhouse/query-log', 'JSONEachRow'); CREATE MATERIALIZED VIEW audit.query_log_to_url TO audit.query_log_url -AS SELECT query, toString(type) AS type, event_time_microseconds -FROM system.query_log; +AS SELECT query, toString(type) AS type, event_time_microseconds FROM system.query_log; ``` -Verified on 26.7.5.10 against a test collector; each row arrives as one JSON object: +Delivery is best effort: if the collector is down those rows never arrive, but the local table is unaffected on 25.8 and later. On older lines confirm the release contains [PR #75679](https://github.com/ClickHouse/ClickHouse/pull/75679) (24.12.6.70, 25.1.6.34, 25.2.1.3085+) — without it a dead collector also wipes local rows. -```json -{"query":"SELECT 'URLMARKER-1'","type":"QueryStart","event_time_microseconds":"2026-08-25 12:09:05.651860"} -``` +## Export to a file -This is the safer of the two external targets: as shown above, both a 503 response and a refused connection leave the local `system.query_log` intact on 25.8.32.4 and later. Confirm your release line contains PR #75679 (24.12.6.70, 25.1.6.34, 25.2.1.3085+). +The `File` engine only writes inside `/var/lib/clickhouse/user_files` — any other path is rejected with `DATABASE_ACCESS_DENIED`. With no path at all it writes to an unpredictable UUID directory, so always give one: -## Local flat-file target +```sql +CREATE TABLE audit.query_log_file (query String, type String, event_time_microseconds DateTime64(6)) +ENGINE = File(JSONEachRow, '/var/lib/clickhouse/user_files/query-audit.jsonl'); -The `File` engine can write newline-delimited JSON. Its path handling is more capable than it first appears. +CREATE MATERIALIZED VIEW audit.query_log_to_file TO audit.query_log_file +AS SELECT query, toString(type) AS type, event_time_microseconds FROM system.query_log; +``` -With no path, ClickHouse writes inside the table's own data directory, which for an `Atomic` database is a UUID path that is not stable across a table recreation: +Point Filebeat or Fluent Bit at that file. ClickHouse never rotates it — rotation is up to you. -```text -/var/lib/clickhouse/store/ee3/ee356d26-9a82-4ac2-a16e-9f028bb60e7e/data.JSONEachRow -``` +## Monitoring -An arbitrary absolute path is rejected: +Any non-zero value means log rows were lost and will not come back: ```sql -CREATE TABLE audit.f (a String) ENGINE = File(JSONEachRow, '/var/log/audit.jsonl'); +SELECT value FROM system.events WHERE event = 'SystemLogErrorOnFlush'; ``` -```text -Code: 291. DB::Exception: File `/var/log/audit.jsonl` is not inside -`/var/lib/clickhouse/user_files`. (DATABASE_ACCESS_DENIED) -``` +An empty result is healthy. The event exists from 25.2 onwards. Its built-in description says failed flushes are repeated — they are not. -But a path **inside `user_files` is accepted**, which does give a stable, shipper-friendly filename. Verified on 26.7.5.10 — the file is appended across flushes: +## Check the export after every upgrade -```sql -CREATE TABLE audit.query_log_file -( - query String, - type String, - event_time_microseconds DateTime64(6) -) -ENGINE = File(JSONEachRow, '/var/lib/clickhouse/user_files/query-audit.jsonl'); +When an upgrade changes the log schema, ClickHouse renames `system.query_log` to `query_log_0` and creates a new table. **The MV stays attached to the renamed table and the export stops** — with no error, and `SHOW CREATE TABLE` still showing `FROM system.query_log`. -CREATE MATERIALIZED VIEW audit.query_log_to_file TO audit.query_log_file -AS SELECT query, toString(type) AS type, event_time_microseconds -FROM system.query_log; +```sql +SELECT name, dependencies_table +FROM system.tables +WHERE database = 'system' AND name LIKE 'query_log%'; ``` -Point Filebeat or Fluent Bit at `/var/lib/clickhouse/user_files/query-audit.jsonl`. Note that the file grows without bound — ClickHouse does not rotate it, so external rotation is your responsibility, and rotating it out from under the engine requires care. +Broken — the dependency sits on the old table: -## Advanced pattern: make the system log itself remote +```text +query_log [] +query_log_0 ['query_log_to_s3'] +``` -Instead of a local MergeTree plus an MV, the system-log config can name an external engine directly. +Fix by recreating the view: -{{% alert title="Warning" color="warning" %}} -With this pattern the external engine *is* the storage for the system log. There is no local copy for reconciliation, and remote availability becomes part of the flush path. Combined with the no-retry behavior below, an outage means permanent audit gaps. -{{% /alert %}} +```sql +DROP TABLE audit.query_log_to_s3; +CREATE MATERIALIZED VIEW audit.query_log_to_s3 TO audit.query_log_s3 AS SELECT ... FROM system.query_log; +``` -S3-backed system logs need `skip_alias_columns`, added in **26.4** ([PR #102669](https://github.com/ClickHouse/ClickHouse/pull/102669), shipped in v26.4.1.1141). Without it the flush fails on every version tested: +## Direct external engine -```text -Code: 36. DB::Exception: Special columns like MATERIALIZED, ALIAS or EPHEMERAL -are not supported for s3 storage. (BAD_ARGUMENTS) -``` +The system log can use an external engine directly, dropping the local MergeTree copy entirely. A flush that fails is not retried, so any outage becomes a permanent gap in the audit trail — with a dead S3 endpoint, 14 of 16 rows were lost and never reappeared after recovery. Prefer the MV pattern. + +S3-backed system logs need `skip_alias_columns`, added in **26.4**. Earlier versions ignore the setting silently and every flush fails with `Special columns like MATERIALIZED, ALIAS or EPHEMERAL are not supported for s3 storage`. ```xml @@ -365,130 +217,20 @@ are not supported for s3 storage. (BAD_ARGUMENTS) ``` -Tested: works with zero flush errors on 26.7.5.10 and 26.4.5.143. `system.query_log` then reports `engine = S3` and remains readable through `SELECT`. On 26.3.21.7 and 25.8.32.4 the same config still fails with the ALIAS error — and **the unknown `skip_alias_columns` element is ignored without any warning**, so the only symptom is the flush error in the log. - -`replace="replace"` matters: the shipped `config.xml` already defines `` with ``, and config.d merging would otherwise combine them. Specifying `engine` alongside `partition_by`, `order_by`, or `ttl` makes the **server refuse to start** (exit code 36): - -```text -Code: 36. DB::Exception: If 'engine' is specified for system table, PARTITION BY -parameters should be specified directly inside 'engine' and 'partition_by' setting -doesn't make sense. (BAD_ARGUMENTS) -``` - -Put any such clauses inside the engine expression instead. - -### Direct-engine failure behavior - -A failed flush increments `SystemLogErrorOnFlush` — and drops the batch. There is no retry, and recovery of the endpoint does not bring the missed rows back. - -Reproduced on 26.7.5.10 with a direct S3-backed `query_log`: marker queries were run while the endpoint was healthy, then the S3 endpoint was cut, then eight more markers were run, then the endpoint was restored and logs flushed again. - -| Phase | Rows reaching S3 | -|---|---| -| A — endpoint healthy | 6 | -| B — endpoint down (8 markers, 16 rows) | **2** | -| C — endpoint restored | 2 | - -Fourteen of the sixteen phase-B rows were lost permanently. The two that survived were still buffered when the endpoint came back; every batch whose flush had already been *attempted* was gone. `SystemLogErrorOnFlush` reached 6. - -An in-memory system-log pipeline is not a durable queue. For a compliance trail, treat external availability, restarts, and crash behavior as part of the delivery design rather than assuming a remote table engine delivers reliably. - -## Stronger delivery: separate fast export from reconciliation - -If missing external records are unacceptable, keep the local system logs and add a repair path: - -```text -system.query_log ------------------> local retained history - | - +-- MV ----------------------> low-latency external export - | - +-- periodic reconciliation -> repair external gaps -``` - -A reconciler can checkpoint on `hostname`, `event_time_microseconds`, `query_id`, and `type`, then re-export rows that did not reach the destination. - -Alternatively, put a buffered sender between the MV and the final destination. ClickHouse upstream CI exports system logs through an MV into a `Distributed` sender table, which queues delivery to a remote cluster, and explicitly flushes those queues before teardown because pending rows can otherwise remain unsent. - -Neither makes the sink exactly-once, but both separate local persistence from remote delivery and give the export path an observable queue. - -## Upgrade and schema-change check - -When a system-log schema changes after an upgrade, ClickHouse renames the existing table and creates a new one. **The materialized view follows the renamed table**, so the export stops silently. - -Reproduced by starting 25.8.32.4 with a working MV export, then booting 26.7.5.10 on the same data directory: - -```text - SystemLog (system.query_log): Existing table system.query_log for system log -has obsolete or different structure. Renaming it to query_log_0. -``` - -After the upgrade, marker queries produced **0 exported rows**. No error was logged and `SystemLogErrorOnFlush` never incremented. `SHOW CREATE TABLE` still read `FROM system.query_log`, because the MV tracks its source by UUID, not by name — and that UUID now belongs to `query_log_0`. - -Detect it by checking which table actually carries the dependency: - -```sql -SELECT name, dependencies_database, dependencies_table -FROM system.tables -WHERE database = 'system' AND name LIKE 'query_log%'; -``` - -Broken state — the dependency sits on the renamed table, and the live table has none (the export view was named `mv` in this test): - -```text -query_log [] [] -query_log_0 ['audit'] ['mv'] -``` - -The fix is to recreate the view, after which exports resume immediately: - -```sql -DROP TABLE audit.query_log_to_s3; - -CREATE MATERIALIZED VIEW audit.query_log_to_s3 TO audit.query_log_s3 -AS SELECT ... FROM system.query_log; -``` - -Run this check after **every** upgrade, together with an end-to-end smoke test: - -```sql -SELECT 'audit-export-smoke-test'; -SYSTEM FLUSH LOGS; -``` - -Then confirm the record appears both in `system.query_log` and at the external destination. +`replace="replace"` is required, otherwise this merges with the shipped `` section. Never combine `` with ``, `` or `` — **the server refuses to start**; put those clauses inside the engine expression. ## Version notes -| ClickHouse version | Relevant behavior | +| Version | Behavior | |---|---| -| 23.3+ | System-log flushes set `materialized_views_ignore_errors=true`. This covers row-push errors only; sink-construction errors, including the S3 object-exists and endpoint checks, still fail the flush. | -| 24.12.6.70, 25.1.6.34, 25.2.1.3085+ | Contain PR #75679, so `URL`-target connectivity errors are ignored as intended. Tested: 24.10.1.2812 loses local rows, 25.8.32.4 does not. | -| 25.2+ | `SystemLogErrorOnFlush` profile event added (PR #75466; listed in the 25.3 changelog). Verified absent in 24.10.1.2812, present in 25.8.32.4. Its description claims flushes are repeated; they are not. | -| 26.4+ | Adds `default_system_log_flush_policy.skip_alias_columns`, required for S3-backed system logs. Earlier versions ignore the element silently. Verified working on 26.4.5.143 and 26.7.5.10; still failing on 26.3.21.7 and 25.8.32.4. | -| Open as of 26.7.5.10 | Issue #112419: concurrent writers on the same S3 key sequence lose rows with no error. Reproduced here on 26.7.5.10, the newest stable tested; the reporter reproduced it on a prestable 26.8 build. No fix is released, so treat every current line as affected and keep object namespaces unique per server. | - -Verify backports against your exact release before relying on any boundary above. +| 24.12.6.70, 25.1.6.34, 25.2.1.3085+ | Contain PR #75679. Earlier releases lose local rows when a `URL` target is unreachable. | +| 25.2+ | `SystemLogErrorOnFlush` available. | +| 26.4+ | `skip_alias_columns` available, required for S3-backed system logs. | +| Open as of 26.7.5.10 | Issue #112419 — concurrent writers on one S3 key lose rows silently. | ## Related resources -- [System tables eat my disk](/altinity-kb-setup-and-maintenance/altinity-kb-system-tables-eat-my-disk/) — retention and TTL for the local system log tables that this export reads from. -- [Logging](/altinity-kb-setup-and-maintenance/logging/) — the server's text log files, which are a separate concern from these system tables. - -## References - -- [ClickHouse system tables overview](https://clickhouse.com/docs/reference/system-tables/overview) -- [`system.query_log`](https://clickhouse.com/docs/reference/system-tables/query_log) -- [`system.session_log`](https://clickhouse.com/docs/reference/system-tables/session_log) -- [Materialized views](https://clickhouse.com/docs/reference/statements/create/view) -- [S3 table engine](https://clickhouse.com/docs/reference/engines/table-engines/integrations/s3) -- [URL table engine](https://clickhouse.com/docs/reference/engines/table-engines/special/url) -- [File table engine](https://clickhouse.com/docs/reference/engines/table-engines/special/file) -- [Named collections](https://clickhouse.com/docs/concepts/features/configuration/server-config/named-collections) -- [`SystemLog.cpp`](https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/SystemLog.cpp) — flush loop and error handling -- [`SystemLogBase.cpp`](https://github.com/ClickHouse/ClickHouse/blob/master/src/Common/SystemLogBase.cpp) — queue `pop()`/`confirm()`, showing no retry path -- [PR #75466: add `SystemLogErrorOnFlush`](https://github.com/ClickHouse/ClickHouse/pull/75466) -- [23.3 changelog: `materialized_views_ignore_errors` for system logs](https://github.com/ClickHouse/ClickHouse/blob/master/docs/changelogs/archive/v23.3.1.2823-lts.md) -- [PR #75679: respect `materialized_views_ignore_errors` for URL-engine MVs](https://github.com/ClickHouse/ClickHouse/pull/75679) -- [PR #102669: `skip_alias_columns` for S3-backed system logs](https://github.com/ClickHouse/ClickHouse/pull/102669) -- [Issue #112419: concurrent writes to the same S3 object key](https://github.com/ClickHouse/ClickHouse/issues/112419) -- [Upstream system-log export used by ClickHouse CI](https://github.com/ClickHouse/ClickHouse/blob/master/ci/jobs/scripts/functional_tests/setup_log_cluster.sh) +- [System tables eat my disk](/altinity-kb-setup-and-maintenance/altinity-kb-system-tables-eat-my-disk/) — retention and TTL for the local system log tables. +- [Logging](/altinity-kb-setup-and-maintenance/logging/) — the server text log files, a separate concern. +- [S3 table engine](https://clickhouse.com/docs/reference/engines/table-engines/integrations/s3), [URL table engine](https://clickhouse.com/docs/reference/engines/table-engines/special/url), [File table engine](https://clickhouse.com/docs/reference/engines/table-engines/special/file), [Named collections](https://clickhouse.com/docs/concepts/features/configuration/server-config/named-collections) +- [`system.query_log`](https://clickhouse.com/docs/reference/system-tables/query_log), [`system.session_log`](https://clickhouse.com/docs/reference/system-tables/session_log)