diff --git a/CHANGELOG.md b/CHANGELOG.md index a4d48df..dc29d64 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,8 +3,17 @@ v0.3.1 (Unreleased) ### Query translation * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results. +### Types +* **`DateTimeOffset` is now a supported property type**, and maps to `DateTime64(7, 'UTC')`. The provider previously had no mapping for it, so EF Core fell back to `DateTimeOffsetToStringConverter` and made a `String` column with no warning — which broke queries against a real `DateTime64` column with `TYPE_MISMATCH`, and stopped `SaveChanges` from writing the value at all. The store type pins the timezone to `'UTC'` so the instant does not depend on the server's `session_timezone`, and precision 7 is one .NET tick (100 ns), so the round trip is exact. `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin. Note that ClickHouse has no type that stores a UTC offset: `DateTime64` holds an instant, and a declared timezone only decides how that instant is rendered. The instant is kept and the offset is not, so a value read back carries the offset of the column's declared timezone. A non-UTC column such as `DateTime64(6, 'Asia/Tokyo')` reads correctly, as does a fixed-offset column such as `DateTime64(7, 'Fixed/UTC+05:30:00')`, and `DateTimeOffset` composes into `Array`, `Map` and `Tuple` columns. Where an instant cannot be reproduced exactly the read throws rather than returning a value that is quietly wrong: the repeated hour when clocks go back in a zone whose candidate offsets are both non-zero (`Europe/Paris`), and a value before 1900 in a named zone, where the offset is Local Mean Time to the second. Writing a value the store type cannot hold throws for the same reason — ClickHouse wraps it rather than reporting it, which matters for precision 8 and 9, whose ranges are narrower than `DateTimeOffset`. **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')` — add `HasConversion()` to keep the previous shape. See the [DateTimeOffset section of the README](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore#datetimeoffset) for the daylight-saving and `MinValue`/`MaxValue` limits. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53)) + ### Bug fixes * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. +* **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)` and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran, and any component whose CLR type differs from the driver's type threw `InvalidCastException`. This is not new with `DateTimeOffset` — `DateOnly[]`, `Dictionary` and `Tuple` were already affected, because `DateOnly` also arrives from the driver as a `DateTime`. The composite is now rebuilt component by component, with the same two steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. An `enum` component, a `List` component and a nested composite therefore all read correctly, and a component that needs no conversion keeps the direct cast. **Known limit:** *writing* a component that needs a `ValueConverter` still does not work, because the bulk insert path passes model values to the driver without applying converters ([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)) — an `enum` inside a composite is written as its raw ordinal. +* **`Array(Nullable(T))` DDL is no longer double-wrapped.** For a value-type element the store type came out as `Array(Nullable(Nullable(T)))`, which ClickHouse rejects with `Nested type Nullable(T) cannot be inside Nullable type`, so `EnsureCreated` and migrations both failed. The component mapping is resolved from a store type that already carries the wrapper, and `HasColumnType(...)` text is kept verbatim, so the nullable-element wrapper added a second one. It now adds the wrapper only when the inner store type does not already have one, including through `LowCardinality(Nullable(T))`. Reference-type elements were never affected. +* **Composite component mappings now honour the CLR component type.** One ClickHouse store type can serve more than one CLR type: `DateTime64` serves both `DateTime` and `DateTimeOffset`, and `Date32` serves both `DateTime` and `DateOnly`. Resolving a component from an explicit store type such as `HasColumnType("Array(Nullable(DateTime64(7, 'UTC')))")` always picked the default CLR type, which gave the composite the wrong element type and broke change tracking. Array, Map and Tuple now pass the component CLR type from the model when they have one. +* **Composite components now resolve to the CLR type the model declares.** Three separate causes gave a composite the wrong element type, and the query then failed to compile with a coercion error naming a type the user never wrote. A converter on the component erased the `Nullable<>` — EF Core takes a mapping's CLR type from its converter's model type — so `Colour?[]` over `Array(Nullable(Enum8(...)))` resolved as `Colour[]`. A component resolved from an explicit store type kept that store type's default CLR type, so `DateTime[]` over `Array(Date32)` resolved as `DateOnly[]`. And the store-type parsers disagreed about whitespace, so `Array( Nullable ( Date32 ) )` — which ClickHouse accepts and normalizes — lost its nullable element. The parsers are now one shared helper. +* **Reading a `Tuple(...)` column no longer allocates per row.** The converter array was built inside the materializer, so every row allocated a fresh array of the same delegates, including rows the fast path returned untouched. It is now built once. `Array` and `Map` were never affected. +* **A nested composite whose innermost component converts is no longer returned untouched.** The pass-through check looked only at the immediate component mapping, and a composite component carries no converter of its own, so an `Array(Array(T))` could skip the conversion of its innermost values. The check now recurses. * `Sum`/`SumAsync` over a `double` or `float` column no longer throws `InvalidCastException`. EF Core wraps a top-level aggregate so the empty case returns `0`, supplying that fallback as a boxed `Int32` carrying the `Float64`/`Float32` mapping; the literal generators now convert rather than unbox. The `Float32` read path also converts, since ClickHouse widens `sum(Float32)` to `Float64` (which the driver's `GetFloat()` refuses to downcast). ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46)) * **SummingMergeTree with multiple sum columns**: `HasSummingMergeTreeEngine("A", "B")` now generates valid DDL (`SummingMergeTree((A, B))`). Previously it emitted a comma-separated argument list (`SummingMergeTree(A, B)`), which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH`. Single-column and no-column usage are unaffected. diff --git a/README.md b/README.md index dc7d9c8..5371e84 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ public class PageView | **Bool** | `Bool` | `bool` | | **Strings** | `String`, `FixedString(N)` | `string` | | **Enums** | `Enum8(...)`, `Enum16(...)` | `string` or C# `enum` | -| **Date/time** | `Date`, `Date32`, `DateTime`, `DateTime64(P, 'TZ')` | `DateOnly`, `DateTime` | +| **Date/time** | `Date`, `Date32`, `DateTime`, `DateTime64(P, 'TZ')` | `DateOnly`, `DateTime`, `DateTimeOffset` (see [below](#datetimeoffset)) | | **Time** | `Time`, `Time64(N)` | `TimeSpan` | | **UUID** | `UUID` | `Guid` | | **Network** | `IPv4`, `IPv6` | `IPAddress` | @@ -80,6 +80,105 @@ public class PageView | **Geographic** | `Point`, `Ring`, `LineString`, `Polygon`, `MultiLineString`, `MultiPolygon`, `Geometry` | `Tuple` and arrays thereof; `object` for Geometry | | **Wrappers** | `Nullable(T)`, `LowCardinality(T)` | Unwrapped automatically | +### DateTimeOffset + +A `DateTimeOffset` property maps to `DateTime64(7, 'UTC')` by default: + +```csharp +public class Reading +{ + public long Id { get; set; } + public DateTimeOffset RecordedAt { get; set; } // DateTime64(7, 'UTC') +} +``` + +Three things to know: + +**The offset is not kept.** ClickHouse has no type that stores a UTC offset. `DateTime64` holds an +instant, and a declared timezone only decides how that instant is rendered. A value written with +any offset is stored as the correct instant, and a value read back carries the offset of the +column's timezone — `+00:00` for the default store type. With that default store type, comparisons +and ordering are instant-correct, so these two values match the same row: + +```csharp +// The same instant, written two ways. +var a = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)); +var b = new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero); +``` + +If you must keep the offset, store it yourself in a second column alongside a `DateTime`. To keep +the whole value as text, ask for the conversion explicitly with `HasConversion()` — note +that `HasColumnType("String")` on its own is not enough, because it adds no converter. Such a +column is read-only for now: `SaveChanges` cannot write any property that has a value converter +([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)). + +**Precision 7 makes the round trip exact.** One .NET tick is 100 ns, which is precision 7, so a +stored value never comes back truncated. Precision 7 also covers the full `DateTimeOffset` range, +which lets you use `DateTimeOffset.MinValue` and `MaxValue` as open-ended range limits on the +default store type. Keep those two sentinels to a column whose timezone offset is zero, such as the +default `'UTC'`. Both sit at the edge of the `DateTime` range, and the driver has to build a wall +clock in the column's timezone to return a value, so any non-zero offset pushes one end outside +`DateTime`: reading `MaxValue` from a `DateTime64(7, 'Asia/Tokyo')` column throws. Choose a smaller +precision if you prefer, but be aware that it discards the digits below it: + +```csharp +b.Property(e => e.RecordedAt).HasColumnType("DateTime64(3, 'UTC')"); // milliseconds +b.Property(e => e.RecordedAt).HasPrecision(3); // the same thing +``` + +Precision 8 and 9 are accepted, but they hold a narrower range of dates. ClickHouse stores a +`DateTime64(P)` as an `Int64` count of 10^-P seconds, so precision 9 reaches only 1678 to 2262 and +precision 8 about 1970 ± 2900 years. A value outside the range wraps rather than reporting, so the +provider checks it and throws on write instead. Precision 7 has no such limit — it spans roughly +29 000 years, which is why it is the default. Note also that .NET cannot represent more than 7 +fractional digits, so the extra precision stores no extra detail. + +**Keep `'UTC'` in the store type** unless you have a reason to change it. For a timezone-less type +such as `DateTime64(7)`, the server reads the query parameter in its `session_timezone`, which moves +the instant when that setting is not UTC. + +A column that declares a different timezone, for example `DateTime64(6, 'Asia/Tokyo')`, is read +correctly and returns that zone's offset. Two limits apply to such a column: + +- The host operating system must know the timezone, or the read throws. Minimal Linux images may + need the `tzdata` package. +- In a zone with daylight saving, the repeated hour when clocks go back is ambiguous, because the + driver gives a wall clock and drops the offset. The provider recovers the instant where the zone's + standard offset is zero, such as `Europe/London`. Where both candidate offsets are non-zero, such + as `Europe/Paris`, the instant cannot be recovered and the read throws — reporting one of the two + would move the instant and give the same result for two different ones. +- A value before 1900 in a named zone throws. Before standard time a zone's offset is Local Mean + Time, which IANA records to the second (`+09:18:59` for `Asia/Tokyo`) while `TimeZoneInfo` may + round it to the minute, so the instant cannot be reproduced exactly. Store such a value in a + `'UTC'` column, which has no such offset. +- Dates at the far ends of the `DateTimeOffset` range do not survive, as described above. + +A column can also declare a fixed UTC offset instead of a named zone. ClickHouse spells this +`Fixed/UTC±HH:MM:SS`, with two digits in every field — the server rejects `Fixed/UTC+5:30:00` and +`Fixed/UTC+05:30`: + +```csharp +b.Property(e => e.RecordedAt).HasColumnType("DateTime64(7, 'Fixed/UTC+05:30:00')"); +``` + +None of the limits above applies here: the host needs no timezone data, a fixed offset is never +ambiguous, and it does not change before 1900. Two points of its own do: + +- `DateTimeOffset` holds an offset only within plus or minus 14 hours, and only in whole minutes, + while ClickHouse accepts more. Such a column still reads correctly — the instant is exact, because + the offset is known — but the value comes back at offset `+00:00` rather than the column's offset. + This mapping does not keep the offset in any case. +- ClickHouse does not hold the minutes and seconds fields to 59 — it carries the excess, so + `Fixed/UTC+05:60:00` is a legal name for `+06:00`. The driver does not read such a name, so the + read throws rather than depend on that. Declare the offset as `Fixed/UTC+06:00:00` instead. + +`DateTimeOffset` also composes into the collection types, so `DateTimeOffset[]`, +`List`, `Dictionary` and `Tuple` all +round trip. + +`DateTimeOffset` members such as `.Year` and `.UtcDateTime` do not translate to SQL yet. This +applies to `DateTime` as well — see [#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55). + ## Current Status This provider is in active development. It supports **LINQ queries**, **inserts**, **table engine configuration**, and **migrations** — you can define ClickHouse tables with engine-specific settings, create them via `dotnet ef migrations` or `EnsureCreated`, query with LINQ, and write data via `SaveChanges`. @@ -272,7 +371,7 @@ Configure ClickHouse table engines, ordering, partitioning, and more via EF Core ```csharp modelBuilder.Entity(b => { - b.HasKey(e => e.Id); + b.HasKey(e => e.Id); // becomes ORDER BY (the ClickHouse primary key) when no explicit ORDER BY is set b.Property(e => e.Temperature).HasCodec("Delta, ZSTD"); b.Property(e => e.Location).HasColumnComment("Installation site"); b.HasIndex(e => e.Timestamp) @@ -298,6 +397,8 @@ modelBuilder.Entity(b => **Default behavior:** If no engine is configured, the provider defaults to `MergeTree` with the EF primary key as `ORDER BY`. +**Primary key vs sorting key:** In ClickHouse the `ORDER BY` (sorting key) *is* the primary key, so `HasKey` alone is sufficient — it becomes `ORDER BY`. Only use `.WithPrimaryKey(...)` when you need the primary index to differ from the sort order (e.g. a `SummingMergeTree`/`AggregatingMergeTree` rollup with a long `ORDER BY` but a narrow index). ClickHouse requires the primary key to be a prefix of the `ORDER BY` columns. + ### Migrations The provider supports `dotnet ef migrations` for creating and applying migrations: diff --git a/RELEASENOTES.md b/RELEASENOTES.md index 885476a..3496c89 100644 --- a/RELEASENOTES.md +++ b/RELEASENOTES.md @@ -3,8 +3,17 @@ v0.3.1 (Unreleased) ### Query translation * **`toStartOf*` date-time functions** are now translatable through `EF.Functions`, covering the full family: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with an optional ClickHouse week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general-purpose `ToStartOfInterval(source, value, unit)`. They compose in `GROUP BY` for time bucketing. Input and return types follow ClickHouse: the calendar buckets (`ToStartOfYear`/`Quarter`/`Month`/`Week`) return `Date`, `ToStartOfDay` and the hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly` (Date/Date32); `ToStartOfInterval` is the exception — older ClickHouse rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument`, while recent versions accept it, so prefer a `DateTime`/`DateTime64` column for interval bucketing. `ToStartOfInterval` uses a `ClickHouseInterval` enum for the unit and is emitted as `toStartOfInterval(source, toInterval(value))`. Because the default result types (`Date`/`DateTime`) only span 1970–2149/2106, ClickHouse narrows out-of-range values (pre-1970 clamps to the epoch or wraps around); enable `enable_extended_results_for_datetime_functions` in your session (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) to get range-preserving `Date32`/`DateTime64` results. +### Types +* **`DateTimeOffset` is now a supported property type**, and maps to `DateTime64(7, 'UTC')`. The provider previously had no mapping for it, so EF Core fell back to `DateTimeOffsetToStringConverter` and made a `String` column with no warning — which broke queries against a real `DateTime64` column with `TYPE_MISMATCH`, and stopped `SaveChanges` from writing the value at all. The store type pins the timezone to `'UTC'` so the instant does not depend on the server's `session_timezone`, and precision 7 is one .NET tick (100 ns), so the round trip is exact. `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin. Note that ClickHouse has no type that stores a UTC offset: `DateTime64` holds an instant, and a declared timezone only decides how that instant is rendered. The instant is kept and the offset is not, so a value read back carries the offset of the column's declared timezone. A non-UTC column such as `DateTime64(6, 'Asia/Tokyo')` reads correctly, as does a fixed-offset column such as `DateTime64(7, 'Fixed/UTC+05:30:00')`, and `DateTimeOffset` composes into `Array`, `Map` and `Tuple` columns. Where an instant cannot be reproduced exactly the read throws rather than returning a value that is quietly wrong: the repeated hour when clocks go back in a zone whose candidate offsets are both non-zero (`Europe/Paris`), and a value before 1900 in a named zone, where the offset is Local Mean Time to the second. Writing a value the store type cannot hold throws for the same reason — ClickHouse wraps it rather than reporting it, which matters for precision 8 and 9, whose ranges are narrower than `DateTimeOffset`. **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')` — add `HasConversion()` to keep the previous shape. See the [DateTimeOffset section of the README](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore#datetimeoffset) for the daylight-saving and `MinValue`/`MaxValue` limits. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53)) + ### Bug fixes * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. +* **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)` and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran, and any component whose CLR type differs from the driver's type threw `InvalidCastException`. This is not new with `DateTimeOffset` — `DateOnly[]`, `Dictionary` and `Tuple` were already affected, because `DateOnly` also arrives from the driver as a `DateTime`. The composite is now rebuilt component by component, with the same two steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. An `enum` component, a `List` component and a nested composite therefore all read correctly, and a component that needs no conversion keeps the direct cast. **Known limit:** *writing* a component that needs a `ValueConverter` still does not work, because the bulk insert path passes model values to the driver without applying converters ([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)) — an `enum` inside a composite is written as its raw ordinal. +* **`Array(Nullable(T))` DDL is no longer double-wrapped.** For a value-type element the store type came out as `Array(Nullable(Nullable(T)))`, which ClickHouse rejects with `Nested type Nullable(T) cannot be inside Nullable type`, so `EnsureCreated` and migrations both failed. The component mapping is resolved from a store type that already carries the wrapper, and `HasColumnType(...)` text is kept verbatim, so the nullable-element wrapper added a second one. It now adds the wrapper only when the inner store type does not already have one, including through `LowCardinality(Nullable(T))`. Reference-type elements were never affected. +* **Composite component mappings now honour the CLR component type.** One ClickHouse store type can serve more than one CLR type: `DateTime64` serves both `DateTime` and `DateTimeOffset`, and `Date32` serves both `DateTime` and `DateOnly`. Resolving a component from an explicit store type such as `HasColumnType("Array(Nullable(DateTime64(7, 'UTC')))")` always picked the default CLR type, which gave the composite the wrong element type and broke change tracking. Array, Map and Tuple now pass the component CLR type from the model when they have one. +* **Composite components now resolve to the CLR type the model declares.** Three separate causes gave a composite the wrong element type, and the query then failed to compile with a coercion error naming a type the user never wrote. A converter on the component erased the `Nullable<>` — EF Core takes a mapping's CLR type from its converter's model type — so `Colour?[]` over `Array(Nullable(Enum8(...)))` resolved as `Colour[]`. A component resolved from an explicit store type kept that store type's default CLR type, so `DateTime[]` over `Array(Date32)` resolved as `DateOnly[]`. And the store-type parsers disagreed about whitespace, so `Array( Nullable ( Date32 ) )` — which ClickHouse accepts and normalizes — lost its nullable element. The parsers are now one shared helper. +* **Reading a `Tuple(...)` column no longer allocates per row.** The converter array was built inside the materializer, so every row allocated a fresh array of the same delegates, including rows the fast path returned untouched. It is now built once. `Array` and `Map` were never affected. +* **A nested composite whose innermost component converts is no longer returned untouched.** The pass-through check looked only at the immediate component mapping, and a composite component carries no converter of its own, so an `Array(Array(T))` could skip the conversion of its innermost values. The check now recurses. * Summing a `double` or `float` column (`.SumAsync(x => x.Value)`) no longer throws `InvalidCastException`. Two ClickHouse-specific mismatches were biting: EF Core hands the float literal generator a boxed `Int32` `0` as the empty-result fallback, and ClickHouse widens `sum(Float32)` to `Float64` so the driver couldn't read it back as a `float`. Both the literal generation and the `Float32` read path now convert instead of hard-casting. ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46)) (Thanks to @HotTotem!) * **SummingMergeTree with multiple sum columns** now produces valid DDL. Configuring more than one sum column (`HasSummingMergeTreeEngine("A", "B")`) previously emitted `SummingMergeTree(A, B)`, which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH` — the engine takes a single optional parameter that must be a tuple of columns. Multiple columns are now wrapped in a tuple (`SummingMergeTree((A, B))`); single-column and no-column usage are unchanged. diff --git a/src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs b/src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs index 5b57490..85d90f3 100644 --- a/src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs +++ b/src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs @@ -16,6 +16,10 @@ protected ClickHouseEngineBuilder(IMutableEntityType entityType, string engineNa entityType.SetEngine(engineName); } + /// + /// Sets the table's sorting key (ORDER BY). In ClickHouse the sorting key also serves as the + /// primary key unless an explicit one is set via . + /// public ClickHouseEngineBuilder WithOrderBy(params string[] columns) { ArgumentNullException.ThrowIfNull(columns); @@ -30,6 +34,11 @@ public ClickHouseEngineBuilder WithPartitionBy(params string[] columns) return this; } + /// + /// Sets an explicit primary key (PRIMARY KEY) distinct from the sorting key. Only needed when the + /// primary index should differ from ORDER BY; otherwise the sorting key is used as the primary key. + /// ClickHouse requires these columns to be a prefix of the columns. + /// public ClickHouseEngineBuilder WithPrimaryKey(params string[] columns) { ArgumentNullException.ThrowIfNull(columns); diff --git a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseStoreTypeName.cs b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseStoreTypeName.cs new file mode 100644 index 0000000..2d91743 --- /dev/null +++ b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseStoreTypeName.cs @@ -0,0 +1,87 @@ +namespace ClickHouse.EntityFrameworkCore.Storage.Internal; + +/// +/// Reads the wrapper structure of a ClickHouse store type name. +/// +/// +/// +/// Several parts of the provider ask the same questions of a store type: does it wrap with +/// Nullable(...), and what is inside a given wrapper. They used to answer with their own +/// string checks, which disagreed — one accepted Nullable ( Date32 ) and another did not, +/// so a column ClickHouse accepts lost its nullable element type. One helper keeps them consistent. +/// +/// +/// ClickHouse tolerates whitespace between a type name and its argument list, and normalizes it +/// away: Array( Nullable ( Date32 ) ) and Array(Nullable(Date32)) name the same type. +/// So does a difference of case. Both are accepted here. +/// +/// +internal static class ClickHouseStoreTypeName +{ + /// + /// Removes a layer, if is exactly that + /// wrapper applied to one inner type. + /// + public static bool TryUnwrap(string storeType, string wrapper, out string inner) + { + inner = storeType; + + var s = storeType.AsSpan().Trim(); + if (!s.StartsWith(wrapper, StringComparison.OrdinalIgnoreCase)) + return false; + + var rest = s[wrapper.Length..].TrimStart(); + if (rest.Length < 2 || rest[0] != '(') + return false; + + // Walk to the parenthesis that closes the one just found. It must be the last character, + // otherwise the wrapper is not applied to the whole of the store type. + var depth = 0; + for (var i = 0; i < rest.Length; i++) + { + if (rest[i] == '(') + { + depth++; + } + else if (rest[i] == ')') + { + depth--; + if (depth != 0) + continue; + + if (i != rest.Length - 1) + return false; + + inner = rest[1..i].Trim().ToString(); + return true; + } + } + + return false; + } + + /// + /// Reports whether denotes a value that can be NULL. + /// + /// + /// LowCardinality is a storage-only encoding, and is the one wrapper ClickHouse allows + /// outside Nullable — it rejects Nullable(LowCardinality(T)) and accepts + /// LowCardinality(Nullable(T)). So the nullable marker is not always the outermost one. + /// + public static bool IsNullable(string storeType) + { + var current = storeType; + while (true) + { + if (TryUnwrap(current, "Nullable", out _)) + return true; + if (TryUnwrap(current, "LowCardinality", out var inner)) + { + current = inner; + continue; + } + + return false; + } + } +} diff --git a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs index 22e9e08..8aaf602 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs @@ -28,6 +28,7 @@ public class ClickHouseTypeMappingSource : RelationalTypeMappingSource private static readonly RelationalTypeMapping Float64Mapping = new ClickHouseDoubleTypeMapping(); private static readonly RelationalTypeMapping DateTimeMapping = new ClickHouseDateTimeTypeMapping(); private static readonly RelationalTypeMapping DateTime64Mapping = new ClickHouseDateTime64TypeMapping(); + private static readonly RelationalTypeMapping DateTimeOffsetMapping = new ClickHouseDateTimeOffsetTypeMapping(); private static readonly RelationalTypeMapping DateOnlyMapping = new ClickHouseDateOnlyTypeMapping(); private static readonly RelationalTypeMapping GuidMapping = new ClickHouseGuidTypeMapping(); private static readonly RelationalTypeMapping IPv4Mapping = new ClickHouseIPAddressTypeMapping("IPv4"); @@ -80,6 +81,7 @@ public class ClickHouseTypeMappingSource : RelationalTypeMappingSource { typeof(float), Float32Mapping }, { typeof(double), Float64Mapping }, { typeof(DateTime), DateTimeMapping }, + { typeof(DateTimeOffset), DateTimeOffsetMapping }, { typeof(DateOnly), DateOnlyMapping }, { typeof(Guid), GuidMapping }, { typeof(char), StringMapping }, @@ -144,6 +146,9 @@ public class ClickHouseTypeMappingSource : RelationalTypeMappingSource // Matches a single-quoted string like 'UTC' or 'Asia/Tokyo' private static readonly Regex TimezoneRegex = new(@"'([^']+)'", RegexOptions.Compiled); + /// ClickHouse reads a bare DateTime64 with no argument as precision 3. + private const int BareDateTime64Precision = 3; + public ClickHouseTypeMappingSource( TypeMappingSourceDependencies dependencies, RelationalTypeMappingSourceDependencies relationalDependencies) @@ -283,6 +288,7 @@ public ClickHouseTypeMappingSource( // Call base so plugin/extension type mappings can intercept before our defaults. var mapping = base.FindMapping(in mappingInfo) + ?? FindDateTimeOffsetMapping(mappingInfo) ?? FindDateTime64Mapping(mappingInfo) ?? FindDateTimeMapping(mappingInfo) ?? FindFixedStringMapping(mappingInfo) @@ -340,6 +346,56 @@ private static bool IsCollectionClrType(Type? clrType) || def == typeof(IReadOnlyCollection<>); } + /// + /// Resolves properties. This runs before the + /// DateTime64/DateTime resolvers and before the store-type aliases, because those + /// all produce a CLR type. Without it, EF Core would find no mapping and + /// fall back to DateTimeOffsetToStringConverter, which silently makes a + /// String column (issue #53). + /// + private static RelationalTypeMapping? FindDateTimeOffsetMapping(in RelationalTypeMappingInfo mappingInfo) + { + if (mappingInfo.ClrType != typeof(DateTimeOffset)) + return null; + + var baseName = mappingInfo.StoreTypeNameBase; + var storeTypeName = mappingInfo.StoreTypeName; + + // No store type configured — use the UTC-pinned default, but respect HasPrecision(n). + if (string.IsNullOrWhiteSpace(baseName) && string.IsNullOrWhiteSpace(storeTypeName)) + { + return mappingInfo.Precision is null + ? DateTimeOffsetMapping + : new ClickHouseDateTimeOffsetTypeMapping( + mappingInfo.Precision, + ClickHouseDateTimeOffsetTypeMapping.DefaultTimezone); + } + + if (string.Equals(baseName, "DateTime64", StringComparison.OrdinalIgnoreCase)) + { + // A bare DateTime64 with no argument is precision 3 in ClickHouse, which is what + // FindDateTime64Mapping assumes as well. Our own default of 7 applies only when the + // model configures no store type at all. + return new ClickHouseDateTimeOffsetTypeMapping( + mappingInfo.Precision ?? BareDateTime64Precision, + storeTypeName is null ? null : ExtractTimezone(storeTypeName)); + } + + if (string.Equals(baseName, "DateTime", StringComparison.OrdinalIgnoreCase)) + { + return new ClickHouseDateTimeOffsetTypeMapping( + precision: null, + storeTypeName is null ? null : ExtractTimezone(storeTypeName)); + } + + // Any other explicit store type falls through to the resolvers below, which key off the + // store type rather than the CLR type. Pointing a DateTimeOffset property at an unrelated + // store type such as String therefore gives that store type's mapping with no converter, + // and the CLR type will not agree with the property. Use HasConversion() to store + // the value as text. + return null; + } + private RelationalTypeMapping? FindDateTime64Mapping(in RelationalTypeMappingInfo mappingInfo) { if (!string.Equals(mappingInfo.StoreTypeNameBase, "DateTime64", StringComparison.OrdinalIgnoreCase)) @@ -350,7 +406,7 @@ private static bool IsCollectionClrType(Type? clrType) if (storeTypeName is null || !storeTypeName.Contains('(')) return null; - var precision = mappingInfo.Precision ?? 3; + var precision = mappingInfo.Precision ?? BareDateTime64Precision; var timezone = ExtractTimezone(storeTypeName); return new ClickHouseDateTime64TypeMapping(precision, timezone); } @@ -391,6 +447,7 @@ private static bool IsCollectionClrType(Type? clrType) private RelationalTypeMapping? FindArrayMapping(in RelationalTypeMappingInfo mappingInfo) { RelationalTypeMapping? elementMapping = null; + var elementClrTypeHint = GetCollectionElementType(mappingInfo.ClrType); // Resolve element mapping from store type: Array(X). When the user wrote // HasColumnType("Array(...)"), prefer parsing the inner type from the store @@ -406,7 +463,7 @@ private static bool IsCollectionClrType(Type? clrType) if (innerType is null) return null; - elementMapping = FindComponentMapping(innerType); + elementMapping = FindComponentMapping(innerType, elementClrTypeHint); } // Fall back to the pre-resolved element type mapping from EF Core (used by @@ -414,7 +471,7 @@ private static bool IsCollectionClrType(Type? clrType) elementMapping ??= mappingInfo.ElementTypeMapping as RelationalTypeMapping; var clrType = mappingInfo.ClrType; - var elementClrType = GetCollectionElementType(clrType); + var elementClrType = elementClrTypeHint; // Resolve element mapping from CLR type if not already resolved if (elementMapping is null && elementClrType is not null) @@ -461,14 +518,32 @@ private static bool IsCollectionClrType(Type? clrType) /// /// EF Core's scalar nullability lives on , /// which is why strips Nullable(...) and - /// FindMapping returns the unwrapped scalar mapping — correct for scalar columns - /// where the property/column annotation carries the nullability separately, but - /// insufficient for composites whose element nullability has no annotation channel. + /// FindMapping returns the unwrapped scalar mapping. That is correct for a scalar column, + /// where the property annotation carries nullability separately, but a composite needs the + /// element nullability in the element mapping's CLR type. + /// + /// Note that EF Core does model this for a primitive collection, on + /// , which this + /// resolver does not yet consult. It has no equivalent for a Map value or one Tuple + /// position, so the store type stays the only channel for those. + /// /// /// - private RelationalTypeMapping? FindComponentMapping(string innerStoreType) + /// + /// The component's CLR type, where the model supplies one. Several CLR types share a single + /// ClickHouse store type — DateTime64 serves both and + /// , and Date32 serves both and + /// — so resolving from the store type alone would always pick the + /// default CLR type and give the composite the wrong element type. + /// + private RelationalTypeMapping? FindComponentMapping(string innerStoreType, Type? clrTypeHint = null) { - var inner = FindMapping(innerStoreType); + // Element nullability rides on the store type, so strip Nullable<> from the hint and let + // the wrapper below re-apply it. + var hint = clrTypeHint is null ? null : Nullable.GetUnderlyingType(clrTypeHint) ?? clrTypeHint; + + var inner = hint is null ? FindMapping(innerStoreType) : FindComponentMappingForHint(hint, innerStoreType); + if (inner is null) return null; @@ -485,33 +560,49 @@ private static bool IsCollectionClrType(Type? clrType) } /// - /// Returns true when directly or indirectly wraps with - /// Nullable(...). LowCardinality is a storage-only wrapper, but composes with - /// Nullable (LowCardinality(Nullable(T))) so we strip it to check the inner. + /// Resolves a component mapping for an explicit store type, for a component whose CLR type the + /// model states. /// - private static bool HasNullableElementWrapper(string storeType) + /// + /// Asking for the store type and the CLR type together is not enough on its own. A resolver + /// keyed on the store type may answer with its own default CLR type and ignore the one asked + /// for — Date32 answers whether or not the property is a + /// . The composite would then be built from the wrong element type, and + /// the query would fail to compile with a coercion error naming a type the user never wrote. + /// So the answer is checked, and a mapping that did not honour the request is not used. + /// + private RelationalTypeMapping? FindComponentMappingForHint(Type hint, string innerStoreType) { - var s = storeType.AsSpan().TrimStart(); - while (true) + var withHint = FindMapping(hint, innerStoreType); + if (withHint is not null && withHint.ClrType == hint) + return withHint; + + // The store type did not yield the requested CLR type. Ask for the CLR type alone, then + // keep the store type the model asked for, which is what the column actually is. + var byClrType = FindMapping(hint); + if (byClrType is not null) { - if (s.StartsWith("Nullable(", StringComparison.OrdinalIgnoreCase)) - return true; - if (s.StartsWith("LowCardinality(", StringComparison.OrdinalIgnoreCase)) - { - // Drop the LowCardinality( and matching ) and look at the inner. - var openParen = s.IndexOf('('); - if (openParen < 0) - return false; - s = s[(openParen + 1)..]; - // Trim the trailing matching paren (no need to find the exact match — any - // Nullable( inside will be detected by the StartsWith check on the next loop). - s = s.TrimStart(); - continue; - } - return false; + if (string.Equals(byClrType.StoreType, innerStoreType, StringComparison.Ordinal)) + return byClrType; + + RelationalTypeMappingInfo? cloneInfo = new RelationalTypeMappingInfo( + type: hint, + storeTypeName: innerStoreType, + storeTypeNameBase: null); + return byClrType.Clone(in cloneInfo, storeTypePostfix: StoreTypePostfix.None); } + + return withHint ?? FindMapping(innerStoreType); } + /// + /// Returns true when directly or indirectly wraps with + /// Nullable(...). LowCardinality is a storage-only wrapper, but composes with + /// Nullable (LowCardinality(Nullable(T))) so we strip it to check the inner. + /// + private static bool HasNullableElementWrapper(string storeType) + => ClickHouseStoreTypeName.IsNullable(storeType); + private static Type? GetCollectionElementType(Type? clrType) { if (clrType is null) @@ -546,8 +637,14 @@ private static bool HasNullableElementWrapper(string storeType) if (innerTypes is null) return null; - var keyMapping = FindComponentMapping(innerTypes[0]); - var valueMapping = FindComponentMapping(innerTypes[1]); + // Dictionary supplies the component CLR types when the model has one. + var dictionaryArgs = mappingInfo.ClrType is { IsGenericType: true } dictType + && dictType.GetGenericTypeDefinition() == typeof(Dictionary<,>) + ? dictType.GetGenericArguments() + : null; + + var keyMapping = FindComponentMapping(innerTypes[0], dictionaryArgs?[0]); + var valueMapping = FindComponentMapping(innerTypes[1], dictionaryArgs?[1]); if (keyMapping is null || valueMapping is null) return null; @@ -581,10 +678,18 @@ private static bool HasNullableElementWrapper(string storeType) if (innerTypes is null || innerTypes.Count == 0) return null; + // A tuple CLR type supplies the component CLR types, provided the arity agrees. + var tupleArgs = mappingInfo.ClrType is { IsGenericType: true } tupleType + && ClassifyTupleType(tupleType).IsTuple + && tupleType.GetGenericArguments() is { } args + && args.Length == innerTypes.Count + ? args + : null; + var elementMappings = new List(); - foreach (var innerType in innerTypes) + for (var i = 0; i < innerTypes.Count; i++) { - var mapping = FindComponentMapping(innerType); + var mapping = FindComponentMapping(innerTypes[i], tupleArgs?[i]); if (mapping is null) return null; elementMappings.Add(mapping); @@ -775,38 +880,7 @@ private static string UnwrapStoreType(string storeTypeName) } private static bool TryUnwrapPrefix(string s, string prefix, out string inner) - { - inner = s; - if (s.Length <= prefix.Length + 2 // need at least prefix + "(X)" - || !s.StartsWith(prefix, StringComparison.OrdinalIgnoreCase) - || s[prefix.Length] != '(') - return false; - - // Find matching close paren for the one at prefix.Length - var depth = 0; - for (var i = prefix.Length; i < s.Length; i++) - { - if (s[i] == '(') - depth++; - else if (s[i] == ')') - { - depth--; - if (depth == 0) - { - // Only unwrap if this closing paren is the last character - if (i == s.Length - 1) - { - inner = s[(prefix.Length + 1)..i].Trim(); - return true; - } - - return false; - } - } - } - - return false; - } + => ClickHouseStoreTypeName.TryUnwrap(s, prefix, out inner); /// /// Extracts the single inner type from a parameterized store type like Array(Int32). diff --git a/src/EFCore.ClickHouse/Storage/Internal/IClickHouseWriteValidatingTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/IClickHouseWriteValidatingTypeMapping.cs new file mode 100644 index 0000000..f94214e --- /dev/null +++ b/src/EFCore.ClickHouse/Storage/Internal/IClickHouseWriteValidatingTypeMapping.cs @@ -0,0 +1,21 @@ +namespace ClickHouse.EntityFrameworkCore.Storage.Internal; + +/// +/// Implemented by a type mapping whose store type cannot hold every value of its CLR type, so that +/// a value outside the store type's range is reported rather than written wrong. +/// +/// +/// The bulk insert path gives the driver the model values directly and does not consult the type +/// mapping, so a mapping cannot guard its own writes there. The batch calls this instead. ClickHouse +/// stores a DateTime64(P) as an count of 10^-P seconds, which wraps rather +/// than reports when the value does not fit, so the check has to happen on the client. +/// +internal interface IClickHouseWriteValidatingTypeMapping +{ + /// + /// Throws when cannot be written to this store type. + /// + /// The model value about to be written. Never . + /// The column being written, for the error message. + void ValidateWriteValue(object value, string? columnName); +} diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs index 99d8294..3b56ef3 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs @@ -14,6 +14,9 @@ public class ClickHouseArrayTypeMapping : RelationalTypeMapping private static readonly MethodInfo GetValueMethod = typeof(DbDataReader).GetRuntimeMethod(nameof(DbDataReader.GetValue), [typeof(int)])!; + private static readonly MethodInfo ConvertArrayMethod = + typeof(ClickHouseArrayTypeMapping).GetMethod(nameof(ConvertArray), BindingFlags.Static | BindingFlags.NonPublic)!; + public RelationalTypeMapping ElementMapping { get; } /// @@ -26,8 +29,8 @@ public ClickHouseArrayTypeMapping(RelationalTypeMapping elementMapping) : base( new RelationalTypeMappingParameters( new CoreTypeMappingParameters( - elementMapping.ClrType.MakeArrayType(), - comparer: CreateArrayComparer(elementMapping.ClrType), + ClickHouseNullableElementMapping.ComponentClrType(elementMapping).MakeArrayType(), + comparer: CreateArrayComparer(ClickHouseNullableElementMapping.ComponentClrType(elementMapping)), elementMapping: ExposableElementMapping(elementMapping)), $"Array({elementMapping.StoreType})", dbType: System.Data.DbType.Object)) @@ -81,7 +84,49 @@ public override Expression CustomizeDataReaderExpression(Expression expression) // When there's a ValueConverter (e.g. List ↔ T[]), the data reader must produce // the provider type (T[]). EF Core applies the converter afterward. var targetType = Converter?.ProviderClrType ?? ClrType; - return Expression.Convert(expression, targetType); + + // An element whose CLR type differs from what the driver produces (DateTimeOffset and + // DateOnly both arrive as DateTime) needs the array rebuilt element by element. Casting + // the whole array would throw InvalidCastException. Otherwise cast directly, which is + // both correct and cheaper. + if (!ClickHouseComponentConversion.NeedsConversion(ElementMapping)) + return Expression.Convert(expression, targetType); + + var elementType = ClickHouseNullableElementMapping.ComponentClrType(ElementMapping); + Expression converted = Expression.Call( + ConvertArrayMethod.MakeGenericMethod(elementType), + expression, + ClickHouseComponentConversion.CreateConverter(ElementMapping, elementType), + Expression.Constant(ClickHouseComponentConversion.CanPassThrough(ElementMapping))); + + return converted.Type == targetType ? converted : Expression.Convert(converted, targetType); + } + + /// + /// Rebuilds the driver's array as TElement[], converting each element. Nested composites + /// compose through this: an Array(Array(DateTime64)) element mapping is itself a + /// , so its own conversion runs per element. + /// + private static TElement[] ConvertArray( + object value, + Func convertElement, + bool canPassThrough) + { + // The driver often already produces the target type, for example Array(Int32) -> int[]. + // See ClickHouseComponentConversion.CanPassThrough for when that proves there is no work + // left to do. + if (canPassThrough && value is TElement[] alreadyTyped) + return alreadyTyped; + + var source = (Array)value; + var result = new TElement[source.Length]; + for (var i = 0; i < source.Length; i++) + { + var element = source.GetValue(i); + result[i] = element is null or DBNull ? default! : convertElement(element); + } + + return result; } protected override string GenerateNonNullSqlLiteral(object value) diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseComponentConversion.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseComponentConversion.cs new file mode 100644 index 0000000..68d5dc7 --- /dev/null +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseComponentConversion.cs @@ -0,0 +1,184 @@ +using System.Collections.Concurrent; +using System.Linq.Expressions; +using Microsoft.EntityFrameworkCore.Storage; + +namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; + +/// +/// Lets a composite mapping (Array, Map, Tuple) apply its component mappings' own read conversions +/// to each component of a value that the driver returned. +/// +/// +/// +/// A composite mapping reads its whole column through GetValue, so the component mappings' +/// read pipeline never runs. Where a component's CLR type differs from the type the driver produces, +/// casting the whole composite throws . The composite must instead +/// be rebuilt component by component, reusing each component mapping's existing conversion rather +/// than repeating it here. +/// +/// +/// A component read has the same two steps EF Core applies to a scalar column: +/// +/// +/// turns the raw driver value +/// into the provider CLR type — this is where and +/// are built from the the driver returns. +/// +/// +/// The mapping's turns the provider CLR type into +/// the model CLR type — this is where an Enum8 component becomes a C# enum and an +/// Array(T) component becomes a List<T>. +/// +/// +/// Both steps are needed. Applying only the first would leave a component mapping that carries a +/// converter readable as its provider type but not as the type the property declares. +/// +/// +internal static class ClickHouseComponentConversion +{ + // Keyed by (mapping, target type). The compiled converter is embedded in the materializer as a + // constant, so it is built once per mapping rather than once per row. + private static readonly ConcurrentDictionary<(RelationalTypeMapping Mapping, Type Target), Delegate> ConverterCache = new(); + + private static readonly ConcurrentDictionary NeedsConversionCache = new(); + + /// + /// Reports whether reading changes the value the driver produced. + /// + /// + /// Cached, because a nested composite's answer depends on its own components' answers and the + /// probe below builds a throw-away sub-tree to find out. + /// + public static bool NeedsConversion(RelationalTypeMapping mapping) + => NeedsConversionCache.GetOrAdd(mapping, static m => + { + // A value converter always changes the value. + if (m.Converter is not null) + return true; + + // CustomizeDataReaderExpression returns its argument unchanged when a mapping needs no + // conversion, so reference equality against a probe is a reliable test. + var probe = Expression.Parameter(typeof(object), "component"); + return !ReferenceEquals(m.CustomizeDataReaderExpression(probe), probe); + }); + + /// + /// Reports whether a composite that the driver already produced at the target CLR type may be + /// returned unchanged, instead of being rebuilt component by component. + /// + /// + /// + /// Rebuilding is only needed where reading a component changes its value. The composite + /// mappings therefore keep a fast path for a driver value that already has the target type — + /// which earns its keep for a nested composite such as Array(Array(Int32)), where the + /// inner mapping's read is a cast and nothing more. + /// + /// + /// That fast path is only sound where matching CLR types prove there is nothing left to do. + /// It holds for : every + /// component mapping that uses it either changes the CLR type — + /// and are both built from a — or coerces a + /// numeric type the driver may return too wide, which is a no-op once the type already matches. + /// It does not hold for a , which is free to + /// change the value while keeping the CLR type. So a component that carries one is always + /// rebuilt. + /// + /// + /// + /// + /// The check recurses. A composite component carries no converter of its own — an + /// Array(T) read as T[] does not — so looking only at the immediate mapping would + /// report that an Array(Array(T)) element needs nothing done, and the fast path would + /// return the driver's value with the innermost components left unconverted. + /// + /// + public static bool CanPassThrough(RelationalTypeMapping mapping) + { + if (mapping.Converter is not null) + return false; + + return mapping switch + { + ClickHouseNullableElementMapping nullable => CanPassThrough(nullable.Inner), + ClickHouseArrayTypeMapping array => CanPassThrough(array.ElementMapping), + ClickHouseMapTypeMapping map => CanPassThrough(map.KeyMapping) && CanPassThrough(map.ValueMapping), + ClickHouseTupleTypeMapping tuple => tuple.ElementMappings.All(CanPassThrough), + _ => true + }; + } + + /// + /// Returns an expression of type Func<object, TComponent> that reads one component, + /// where TComponent is . + /// + /// + /// The converter is compiled once per mapping and embedded as a constant. An inline + /// would instead be rebuilt on + /// every materialization, allocating a delegate per row for every composite column. The trade-off + /// is that a constant holding a delegate cannot be quoted, so composite columns whose components + /// convert are not usable from precompiled queries. + /// + public static Expression CreateConverter(RelationalTypeMapping mapping, Type componentType) + => Expression.Constant( + GetConverter(mapping, componentType), + typeof(Func<,>).MakeGenericType(typeof(object), componentType)); + + /// + /// Returns the compiled reader for one component as a Func<object, object?>, for a + /// caller that holds the readers in an array rather than embedding each one separately. + /// + public static Func CreateComponentReader(RelationalTypeMapping mapping) + => (Func)GetConverter(mapping, typeof(object)); + + private static Delegate GetConverter(RelationalTypeMapping mapping, Type componentType) + => ConverterCache.GetOrAdd( + (mapping, componentType), + static key => Compile(key.Mapping, key.Target)); + + private static Delegate Compile(RelationalTypeMapping mapping, Type componentType) + { + var parameter = Expression.Parameter(typeof(object), "component"); + var body = mapping.CustomizeDataReaderExpression(parameter); + + if (mapping.Converter is { } valueConverter) + { + // Step 1 produces the provider CLR type, which is what the converter accepts. + body = Coerce(body, valueConverter.ProviderClrType, mapping, componentType); + body = Expression.Invoke(valueConverter.ConvertFromProviderExpression, body); + } + + body = Coerce(body, componentType, mapping, componentType); + + return Expression.Lambda( + typeof(Func<,>).MakeGenericType(typeof(object), componentType), + body, + parameter) + .Compile(); + } + + private static Expression Coerce( + Expression expression, + Type targetType, + RelationalTypeMapping mapping, + Type componentType) + { + if (expression.Type == targetType) + return expression; + + try + { + return Expression.Convert(expression, targetType); + } + catch (InvalidOperationException ex) + { + // Without this the user sees EF Core's bare "No coercion operator is defined between + // types ...", which names two CLR types they never wrote. + throw new NotSupportedException( + $"The ClickHouse provider cannot read a component of store type '{mapping.StoreType}' " + + $"as '{componentType}'. Reading it produces '{expression.Type}', and no conversion " + + $"to '{targetType}' exists. Change the property type to match the column, or set an " + + $"explicit column type with HasColumnType(...).", + ex); + } + } +} diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs new file mode 100644 index 0000000..945ef02 --- /dev/null +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs @@ -0,0 +1,395 @@ +using System.Collections.Concurrent; +using System.Data.Common; +using System.Globalization; +using System.Linq; +using System.Linq.Expressions; +using System.Reflection; +using System.Text.RegularExpressions; +using Microsoft.EntityFrameworkCore.Storage; + +namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; + +/// +/// Maps to DateTime64 (or DateTime). +/// +/// ClickHouse has no type that stores a UTC offset: DateTime64 holds an instant, and any +/// declared timezone only decides how that instant is rendered. So the instant is preserved and +/// the original offset is not. A value read back carries the offset of the column's declared +/// timezone, which is +00:00 for the default store type. +/// +/// The default store type pins the timezone to 'UTC' on purpose. For a timezone-less +/// parameter type such as DateTime64(7), the driver sends a UTC wall clock and the server +/// then reads it in session_timezone, which moves the instant when that setting is not UTC. +/// +/// No value converter is used. The driver accepts a directly on both +/// the query parameter path and the bulk insert path, and converts it to the correct instant. +/// +/// A column may declare a fixed UTC offset rather than a named zone, which ClickHouse spells +/// Fixed/UTC±HH:MM:SS. reads those, because .NET has no +/// timezone of that name. +/// +/// A column that declares a timezone with daylight saving needs care on read. The driver gives a +/// wall clock in that timezone and drops the offset, so the repeated hour when clocks go back is +/// ambiguous. recovers it when the zone's standard offset is zero, for +/// example Europe/London, because a zero candidate can then be discarded. Where both candidates are +/// not zero, such as Europe/Paris, the instant cannot be recovered and the read throws — reporting +/// standard time would move the instant and map two distinct instants onto one. The default +/// 'UTC' store type has no daylight saving and is not affected, and neither is a fixed +/// offset, which by definition never changes. +/// +/// Two more reads throw rather than return a value that is quietly wrong: a value before +/// in a named zone, where the offset is Local Mean Time to the +/// second and may round it to the minute; and a fixed-offset name the +/// driver does not apply. An offset outside what can hold is not one of +/// them — the instant is still exact, so it is reported at offset zero. +/// +/// On write, refuses a value the store type cannot hold. ClickHouse +/// wraps such a value rather than reporting it. +/// +public class ClickHouseDateTimeOffsetTypeMapping : RelationalTypeMapping, IClickHouseWriteValidatingTypeMapping +{ + /// + /// One .NET tick is 100 ns, which is precision 7. This makes the round trip exact, so a + /// stored value never comes back truncated. + /// + public const int DefaultPrecision = 7; + + public const string DefaultTimezone = "UTC"; + + /// .NET cannot render more than 7 fractional digits, because a tick is its smallest unit. + private const int MaxFractionalDigits = 7; + + /// + /// The first year for which a named timezone has a standard offset in whole minutes. Before it, + /// IANA records Local Mean Time to the second and may round to the + /// minute. ClickHouse documents the same year as the start of the DateTime64 range. + /// + private const int FirstStandardTimeYear = 1900; + + /// + /// ClickHouse spells a fixed-offset timezone Fixed/UTC±HH:MM:SS. See + /// for why the pattern is this strict. + /// + private static readonly Regex FixedOffsetRegex = new( + @"^Fixed/UTC([+-])(\d{2}):(\d{2}):(\d{2})$", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + // DateTimeOffset holds an offset only within ±14 hours, and only in whole minutes. ClickHouse + // accepts both a larger magnitude and a finer granularity, for example 'Fixed/UTC+00:00:42'. + private static readonly TimeSpan MaxRepresentableOffset = TimeSpan.FromHours(14); + private static readonly TimeSpan MinRepresentableOffset = TimeSpan.FromHours(-14); + + private static readonly MethodInfo GetValueMethod = + typeof(DbDataReader).GetRuntimeMethod(nameof(DbDataReader.GetValue), [typeof(int)])!; + + private static readonly MethodInfo ConvertToDateTimeOffsetMethod = + typeof(ClickHouseDateTimeOffsetTypeMapping).GetMethod( + nameof(ConvertToDateTimeOffset), + BindingFlags.Public | BindingFlags.Static)!; + + // Resolved per row during materialization, so the lookup is cached. + private static readonly ConcurrentDictionary TimeZoneCache = new(); + + /// + /// The timezone declared by the store type, or when the store type + /// declares none. A timezone-less column is read as a UTC wall clock by the driver. + /// + public string? Timezone { get; } + + public ClickHouseDateTimeOffsetTypeMapping() + : this(DefaultPrecision, DefaultTimezone) + { + } + + /// + /// The DateTime64 precision, or for the second-precision + /// DateTime store type. + /// + /// The declared timezone, or for none. + public ClickHouseDateTimeOffsetTypeMapping(int? precision, string? timezone) + : base( + new RelationalTypeMappingParameters( + new CoreTypeMappingParameters(typeof(DateTimeOffset)), + FormatStoreType(precision, timezone), + StoreTypePostfix.None, + System.Data.DbType.DateTimeOffset, + precision: precision)) + { + Timezone = timezone; + } + + protected ClickHouseDateTimeOffsetTypeMapping(RelationalTypeMappingParameters parameters, string? timezone) + : base(parameters) + { + Timezone = timezone; + } + + protected override RelationalTypeMapping Clone(RelationalTypeMappingParameters parameters) + => new ClickHouseDateTimeOffsetTypeMapping(parameters, Timezone); + + // The driver returns DateTime for DateTime64 columns, never DateTimeOffset — + // GetFieldValue throws InvalidCastException. Read the raw value + // and attach the offset of the column's declared timezone. + public override MethodInfo GetDataReaderMethod() + => GetValueMethod; + + public override Expression CustomizeDataReaderExpression(Expression expression) + => Expression.Call( + ConvertToDateTimeOffsetMethod, + expression, + Expression.Constant(Timezone, typeof(string))); + + /// + /// Converts a value read from a ClickHouse date/time column into a . + /// + /// + /// The driver gives when the column's timezone is offset zero at + /// that instant, and a wall clock in the column's + /// timezone in all other cases. A column with no declared timezone is read as a UTC wall clock. + /// + public static DateTimeOffset ConvertToDateTimeOffset(object value, string? timezone) + { + if (value is DateTimeOffset dateTimeOffset) + return dateTimeOffset; + + var dateTime = (DateTime)value; + + // The driver already resolved the instant for us. + if (dateTime.Kind == DateTimeKind.Utc) + return new DateTimeOffset(dateTime); + + // The column declares no timezone, so the driver's wall clock is already UTC. + if (timezone is null) + return new DateTimeOffset(DateTime.SpecifyKind(dateTime, DateTimeKind.Utc)); + + // A fixed offset resolves without the host's timezone data. This must come before the + // lookup below, which cannot resolve such a name. + if (TryParseFixedOffset(timezone, out var fixedOffset)) + { + var fixedWallClock = DateTime.SpecifyKind(dateTime, DateTimeKind.Unspecified); + + // DateTimeOffset caps an offset at plus or minus 14 hours and holds only whole minutes, + // while ClickHouse accepts more. The instant is still exact — subtract the offset from + // the wall clock — so report it at offset zero rather than refusing to read the column. + // This mapping already does not keep the offset, so nothing more is lost here. + return IsRepresentableOffset(fixedOffset) + ? new DateTimeOffset(fixedWallClock, fixedOffset) + : new DateTimeOffset(DateTime.SpecifyKind(fixedWallClock - fixedOffset, DateTimeKind.Utc)); + } + + var zone = FindTimeZone(timezone) + ?? throw new InvalidOperationException( + $"Cannot read the DateTimeOffset column because this machine does not know the " + + $"timezone '{timezone}' that the column declares. The driver gives a wall clock in " + + $"that timezone, so the offset cannot be found without it. Install the operating " + + $"system timezone data (the 'tzdata' package on a minimal Linux image), or declare " + + $"the column as DateTime64(P, 'UTC')."); + + var wallClock = DateTime.SpecifyKind(dateTime, DateTimeKind.Unspecified); + + // Before standard time, a zone's offset is Local Mean Time, which IANA records to the + // second — Asia/Tokyo is +09:18:59. TimeZoneInfo rounds that to whole minutes on some + // hosts, so the instant would come back quietly shifted by up to a minute. ClickHouse + // documents DateTime64 as valid from 1900 for the same reason, so refuse rather than guess. + if (wallClock.Year < FirstStandardTimeYear) + throw new InvalidOperationException( + $"Cannot read the DateTimeOffset column because the value {wallClock:yyyy-MM-dd HH:mm:ss} " + + $"predates standard time in the timezone '{timezone}' that the column declares. Before " + + $"{FirstStandardTimeYear} a zone's offset is Local Mean Time, which is recorded to the " + + $"second, and TimeZoneInfo rounds it to whole minutes — so the instant cannot be " + + $"reproduced exactly. Declare the column as DateTime64(P, 'UTC') to store such a value."); + + return new DateTimeOffset(wallClock, ResolveOffset(zone, wallClock, timezone)); + } + + /// + /// Reports whether can hold : it caps the + /// magnitude at 14 hours and accepts only whole minutes. + /// + private static bool IsRepresentableOffset(TimeSpan offset) + => offset >= MinRepresentableOffset + && offset <= MaxRepresentableOffset + && offset.Ticks % TimeSpan.TicksPerMinute == 0; + + /// + /// Reads a ClickHouse fixed-offset timezone name into its offset. + /// + /// + /// + /// ClickHouse lets a column declare a fixed UTC offset instead of a named zone, and spells it + /// Fixed/UTC±HH:MM:SS — for example DateTime64(7, 'Fixed/UTC+05:30:00'). Each + /// field must have exactly two digits, and the server rejects Fixed/UTC+5:30:00, + /// Fixed/UTC+05:30 and any change of case. Such a name is not in the IANA database, so + /// cannot resolve it however complete the + /// host's timezone data is. It needs no daylight-saving logic either, because the offset is + /// fixed by definition, so the reading is never ambiguous. + /// + /// + /// The minutes and seconds fields are not held to 59. ClickHouse carries the excess, so + /// Fixed/UTC+05:60:00 is a legal name for the offset +06:00, and the server + /// accepts any name up to a total of 24 hours. The whole shape is matched here so that such a + /// name is diagnosed rather than left to the unresolvable-timezone error, but the offset is + /// only returned for the spelling the driver also reads. See the throw below. + /// + /// + private static bool TryParseFixedOffset(string timezone, out TimeSpan offset) + { + offset = default; + + var match = FixedOffsetRegex.Match(timezone); + if (!match.Success) + return false; + + var magnitude = new TimeSpan( + int.Parse(match.Groups[2].Value, CultureInfo.InvariantCulture), + int.Parse(match.Groups[3].Value, CultureInfo.InvariantCulture), + int.Parse(match.Groups[4].Value, CultureInfo.InvariantCulture)); + + var sign = match.Groups[1].Value == "-" ? -1 : 1; + var candidate = sign * magnitude; + + // An offset that DateTimeOffset cannot hold is no longer refused. The caller reports the + // instant at offset zero instead — see IsRepresentableOffset and its use above. + + // ClickHouse carries minutes and seconds above 59, so 'Fixed/UTC+05:60:00' is a legal name + // for the offset +06:00. The driver does not read those, and returns a UTC wall clock + // instead of one in the column's timezone, so the offset here cannot be attached to it — + // that would move the instant by the whole offset and report nothing. Only the spelling the + // driver agrees with can be read. + if (magnitude.Minutes != int.Parse(match.Groups[3].Value, CultureInfo.InvariantCulture) + || magnitude.Seconds != int.Parse(match.Groups[4].Value, CultureInfo.InvariantCulture)) + { + throw new InvalidOperationException( + $"Cannot read the DateTimeOffset column because the ClickHouse driver does not " + + $"support the timezone '{timezone}' that the column declares. ClickHouse reads it " + + $"as the offset {candidate}, but only spells that offset in a form the driver " + + $"accepts when the minutes and seconds are below 60. Declare the column as " + + $"DateTime64(P, '{FormatFixedOffset(candidate)}') instead."); + } + + offset = candidate; + return true; + } + + /// Spells an offset the way ClickHouse names a fixed-offset timezone. + private static string FormatFixedOffset(TimeSpan offset) + => string.Create( + CultureInfo.InvariantCulture, + $"Fixed/UTC{(offset < TimeSpan.Zero ? '-' : '+')}{offset.Duration():hh\\:mm\\:ss}"); + + private static TimeSpan ResolveOffset(TimeZoneInfo zone, DateTime wallClock, string timezone) + { + // GetUtcOffset reads an Unspecified value as a local time in the given zone. + if (!zone.IsAmbiguousTime(wallClock)) + return zone.GetUtcOffset(wallClock); + + // An ambiguous wall clock — the hour that repeats when clocks go back — has two candidate + // offsets, and GetUtcOffset would pick the standard-time one. We know more than it does: + // the driver only gives Kind=Unspecified when the true offset is not zero, so a zero + // candidate can be discarded. That recovers the exact instant for every zone whose + // standard offset is zero, such as Europe/London. + var standardOffset = zone.GetUtcOffset(wallClock); + if (standardOffset == TimeSpan.Zero) + { + return zone.GetAmbiguousTimeOffsets(wallClock) + .FirstOrDefault(candidate => candidate != TimeSpan.Zero, standardOffset); + } + + // Where both candidates are non-zero (Europe/Paris, America/New_York) the offset the driver + // dropped cannot be recovered. Picking one would silently move the instant of every value in + // the repeated hour, and would map two distinct instants onto the same result, so refuse. + var candidates = zone.GetAmbiguousTimeOffsets(wallClock); + throw new InvalidOperationException( + $"Cannot read the DateTimeOffset column because the wall clock " + + $"{wallClock:yyyy-MM-dd HH:mm:ss} is ambiguous in the timezone '{timezone}' that the " + + $"column declares. It is the hour that repeats when clocks go back, so it means either " + + $"{string.Join(" or ", candidates.Select(FormatOffset))}, and the ClickHouse driver " + + $"gives a wall clock without the offset. Two distinct instants would read back as one. " + + $"Declare the column as DateTime64(P, 'UTC') to store an unambiguous instant."); + } + + private static string FormatOffset(TimeSpan offset) + => string.Create(CultureInfo.InvariantCulture, $"{(offset < TimeSpan.Zero ? '-' : '+')}{offset.Duration():hh\\:mm}"); + + private static TimeZoneInfo? FindTimeZone(string timezone) + => TimeZoneCache.GetOrAdd(timezone, static id => + { + try + { + return TimeZoneInfo.FindSystemTimeZoneById(id); + } + catch (Exception ex) when (ex is TimeZoneNotFoundException or InvalidTimeZoneException) + { + return null; + } + }); + + /// + /// ClickHouse holds a DateTime64(P) as an count of 10^-P seconds since + /// the epoch, and a DateTime as a count of seconds. Neither reports a + /// value that does not fit — the count wraps, and the row comes back with a different date + /// entirely. Precision 7 spans about 29 000 years and so covers every + /// , but a finer precision does not: precision 9 reaches only + /// 1678–2262. So the value is checked here rather than left to wrap silently. + /// + public void ValidateWriteValue(object value, string? columnName) + { + if (value is not DateTimeOffset dateTimeOffset) + return; + + var seconds = dateTimeOffset.ToUnixTimeSeconds(); + var (min, max) = RepresentableSecondsRange(Precision); + if (seconds >= min && seconds <= max) + return; + + var column = columnName is null ? "a DateTimeOffset column" : $"column '{columnName}'"; + throw new InvalidOperationException( + $"Cannot write {dateTimeOffset:yyyy-MM-dd HH:mm:ssK} to {column}, because the store type " + + $"'{StoreType}' holds only {DateTimeOffset.FromUnixTimeSeconds(min):yyyy-MM-dd} to " + + $"{DateTimeOffset.FromUnixTimeSeconds(max):yyyy-MM-dd}. ClickHouse would wrap the value " + + $"rather than report it, and the row would read back with a different date. Use a " + + $"coarser precision — DateTime64(7) covers the whole DateTimeOffset range — or store a " + + $"value inside the range."); + } + + /// + /// The instants a store type of the given precision can hold, in seconds from the epoch. + /// + private static (long Min, long Max) RepresentableSecondsRange(int? precision) + { + // No precision means the second-resolution DateTime store type, an unsigned 32-bit count. + if (precision is null) + return (0, uint.MaxValue); + + var ticksPerSecond = 1L; + for (var i = 0; i < precision.Value; i++) + ticksPerSecond *= 10; + + var magnitude = long.MaxValue / ticksPerSecond; + + // Never report a range wider than DateTimeOffset itself, so the message stays meaningful. + return ( + Math.Max(-magnitude, DateTimeOffset.MinValue.ToUnixTimeSeconds()), + Math.Min(magnitude, DateTimeOffset.MaxValue.ToUnixTimeSeconds())); + } + + // An ISO-8601 literal that carries the offset is instant-exact whatever timezone the target + // column declares. A bare wall clock is not: the server reads it in the column's timezone. + protected override string GenerateNonNullSqlLiteral(object value) + { + ValidateWriteValue(value, columnName: null); + var dateTimeOffset = (DateTimeOffset)value; + var digits = Math.Min(Precision ?? 0, MaxFractionalDigits); + var fraction = digits == 0 ? string.Empty : "." + new string('f', digits); + return $"'{dateTimeOffset.ToString($"yyyy-MM-dd HH:mm:ss{fraction}zzz", CultureInfo.InvariantCulture)}'"; + } + + private static string FormatStoreType(int? precision, string? timezone) + => (precision, timezone) switch + { + (null, null) => "DateTime", + (null, _) => $"DateTime('{timezone}')", + (_, null) => $"DateTime64({precision})", + _ => $"DateTime64({precision}, '{timezone}')" + }; +} diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseMapTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseMapTypeMapping.cs index 980b3a3..8563112 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseMapTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseMapTypeMapping.cs @@ -13,15 +13,27 @@ public class ClickHouseMapTypeMapping : RelationalTypeMapping private static readonly MethodInfo GetValueMethod = typeof(DbDataReader).GetRuntimeMethod(nameof(DbDataReader.GetValue), [typeof(int)])!; + private static readonly MethodInfo ConvertMapMethod = + typeof(ClickHouseMapTypeMapping).GetMethod(nameof(ConvertMap), BindingFlags.Static | BindingFlags.NonPublic)!; + public RelationalTypeMapping KeyMapping { get; } public RelationalTypeMapping ValueMapping { get; } + // The CLR types this Map is built from. See ClickHouseNullableElementMapping.ComponentClrType + // for why the component mapping's own ClrType is not enough. + private Type KeyComponentClrType => ClickHouseNullableElementMapping.ComponentClrType(KeyMapping); + private Type ValueComponentClrType => ClickHouseNullableElementMapping.ComponentClrType(ValueMapping); + public ClickHouseMapTypeMapping(RelationalTypeMapping keyMapping, RelationalTypeMapping valueMapping) : base( new RelationalTypeMappingParameters( new CoreTypeMappingParameters( - typeof(Dictionary<,>).MakeGenericType(keyMapping.ClrType, valueMapping.ClrType), - comparer: CreateDictionaryComparer(keyMapping.ClrType, valueMapping.ClrType)), + typeof(Dictionary<,>).MakeGenericType( + ClickHouseNullableElementMapping.ComponentClrType(keyMapping), + ClickHouseNullableElementMapping.ComponentClrType(valueMapping)), + comparer: CreateDictionaryComparer( + ClickHouseNullableElementMapping.ComponentClrType(keyMapping), + ClickHouseNullableElementMapping.ComponentClrType(valueMapping))), $"Map({keyMapping.StoreType}, {valueMapping.StoreType})", dbType: System.Data.DbType.Object)) { @@ -46,7 +58,50 @@ public override MethodInfo GetDataReaderMethod() => GetValueMethod; public override Expression CustomizeDataReaderExpression(Expression expression) - => Expression.Convert(expression, ClrType); + { + // A key or value whose CLR type differs from what the driver produces (DateTimeOffset and + // DateOnly both arrive as DateTime) needs the dictionary rebuilt entry by entry. Casting + // the whole dictionary would throw InvalidCastException. + if (!ClickHouseComponentConversion.NeedsConversion(KeyMapping) + && !ClickHouseComponentConversion.NeedsConversion(ValueMapping)) + { + return Expression.Convert(expression, ClrType); + } + + Expression converted = Expression.Call( + ConvertMapMethod.MakeGenericMethod(KeyComponentClrType, ValueComponentClrType), + expression, + ClickHouseComponentConversion.CreateConverter(KeyMapping, KeyComponentClrType), + ClickHouseComponentConversion.CreateConverter(ValueMapping, ValueComponentClrType), + Expression.Constant( + ClickHouseComponentConversion.CanPassThrough(KeyMapping) + && ClickHouseComponentConversion.CanPassThrough(ValueMapping))); + + return converted.Type == ClrType ? converted : Expression.Convert(converted, ClrType); + } + + private static Dictionary ConvertMap( + object value, + Func convertKey, + Func convertValue, + bool canPassThrough) + where TKey : notnull + { + // See ClickHouseComponentConversion.CanPassThrough for when a dictionary the driver already + // typed needs no rebuilding. + if (canPassThrough && value is Dictionary alreadyTyped) + return alreadyTyped; + + var source = (IDictionary)value; + var result = new Dictionary(source.Count); + foreach (DictionaryEntry entry in source) + { + result[convertKey(entry.Key)] = + entry.Value is null or DBNull ? default! : convertValue(entry.Value); + } + + return result; + } protected override string GenerateNonNullSqlLiteral(object value) { diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseNullableElementMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseNullableElementMapping.cs index 2ed5ffa..eaab1a5 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseNullableElementMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseNullableElementMapping.cs @@ -1,3 +1,4 @@ +using System.Linq.Expressions; using System.Reflection; using Microsoft.EntityFrameworkCore.Storage; @@ -14,10 +15,14 @@ namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; /// so 's FindMapping strips /// Nullable(...) wrappers in ParseStoreTypeName and returns the unwrapped scalar /// mapping. That convention works for scalar columns but breaks composites: an -/// Array(Nullable(Int32)) property is int?[] at the CLR level, and there is no -/// per-element IsNullable annotation channel for the composite to consult. The only -/// way to surface element-level nullability is through the element mapping's -/// . +/// Array(Nullable(Int32)) property is int?[] at the CLR level, so the composite must +/// take element nullability from the element mapping's . +/// +/// For a primitive collection, EF Core does model this on +/// , and the +/// resolver should prefer that channel. It has no equivalent for a Map value or a single +/// Tuple position, so this wrapper stays necessary for those. +/// /// /// /// This wrapper exists for that single purpose: report Nullable<T> as the CLR @@ -31,6 +36,30 @@ public sealed class ClickHouseNullableElementMapping : RelationalTypeMapping { public RelationalTypeMapping Inner { get; } + /// + /// The CLR type a composite should give this component: Nullable<T>. + /// + /// + /// cannot be trusted for this. When the inner + /// mapping carries a + /// — an Enum8 component does — EF Core takes the mapping's CLR type from the converter's + /// model type, which is the non-nullable T, and the Nullable<T> asked for in + /// the constructor is discarded. A composite built from ClrType alone would then be + /// T[] where the property is T?[], and the query would fail to compile. + /// + public Type NullableClrType + => Nullable.GetUnderlyingType(ClrType) is not null + ? ClrType + : typeof(Nullable<>).MakeGenericType(ClrType); + + /// + /// The CLR type contributes as a component of a composite. Prefer + /// this over wherever an Array, Map, + /// Tuple or Variant builds its own CLR type from its components. + /// + public static Type ComponentClrType(RelationalTypeMapping mapping) + => mapping is ClickHouseNullableElementMapping wrapper ? wrapper.NullableClrType : mapping.ClrType; + public ClickHouseNullableElementMapping(RelationalTypeMapping inner) : base(BuildParameters(inner)) { @@ -60,7 +89,7 @@ private static RelationalTypeMappingParameters BuildParameters(RelationalTypeMap valueGeneratorFactory: null, elementMapping: inner.ElementTypeMapping, jsonValueReaderWriter: inner.JsonValueReaderWriter), - $"Nullable({inner.StoreType})", + FormatStoreType(inner.StoreType), inner.StoreTypePostfix, inner.DbType, inner.IsUnicode, @@ -70,10 +99,30 @@ private static RelationalTypeMappingParameters BuildParameters(RelationalTypeMap inner.Scale); } + /// + /// Adds the Nullable(...) wrapper unless the inner store type already carries one. + /// + /// + /// The inner mapping is resolved from the component store type, which still holds the + /// Nullable(...) text, and PreserveExplicitStoreType keeps that text verbatim. + /// Wrapping it again would give Nullable(Nullable(T)), which ClickHouse rejects with + /// Nested type Nullable(T) cannot be inside Nullable type. + /// + private static string FormatStoreType(string innerStoreType) + => ClickHouseStoreTypeName.IsNullable(innerStoreType) + ? innerStoreType + : $"Nullable({innerStoreType})"; + protected override RelationalTypeMapping Clone(RelationalTypeMappingParameters parameters) => new ClickHouseNullableElementMapping(parameters, Inner); public override MethodInfo GetDataReaderMethod() => Inner.GetDataReaderMethod(); + // Delegate the read conversion as well, so a composite over Nullable(DateTime64) or + // Nullable(Date32) still converts each element. Callers handle the null case before this + // runs, so the inner non-nullable conversion is safe here. + public override Expression CustomizeDataReaderExpression(Expression expression) + => Inner.CustomizeDataReaderExpression(expression); + protected override string GenerateNonNullSqlLiteral(object value) => Inner.GenerateSqlLiteral(value); } diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseTupleTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseTupleTypeMapping.cs index b91cd96..595a4d9 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseTupleTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseTupleTypeMapping.cs @@ -14,10 +14,11 @@ public class ClickHouseTupleTypeMapping : RelationalTypeMapping typeof(DbDataReader).GetRuntimeMethod(nameof(DbDataReader.GetValue), [typeof(int)])!; private static readonly MethodInfo ConvertMethod = - typeof(ClickHouseTupleTypeMapping).GetMethod(nameof(ConvertToValueTuple), BindingFlags.Static | BindingFlags.NonPublic)!; + typeof(ClickHouseTupleTypeMapping).GetMethod(nameof(ConvertTuple), BindingFlags.Static | BindingFlags.NonPublic)!; - // Cache compiled constructors per ValueTuple type to avoid Activator.CreateInstance per row - private static readonly ConcurrentDictionary ConstructorCache = new(); + // Cache the compiled constructor and its component types per tuple type, to avoid + // Activator.CreateInstance and reflection per row. + private static readonly ConcurrentDictionary ConstructorCache = new(); public IReadOnlyList ElementMappings { get; } @@ -25,7 +26,9 @@ public ClickHouseTupleTypeMapping(IReadOnlyList elementMa : base( new RelationalTypeMappingParameters( new CoreTypeMappingParameters( - MakeTupleType(elementMappings.Select(m => m.ClrType).ToArray(), useValueTuple)), + MakeTupleType( + elementMappings.Select(ClickHouseNullableElementMapping.ComponentClrType).ToArray(), + useValueTuple)), FormatStoreType(elementMappings), dbType: System.Data.DbType.Object)) { @@ -48,48 +51,95 @@ public override MethodInfo GetDataReaderMethod() public override Expression CustomizeDataReaderExpression(Expression expression) { - // The driver returns System.Tuple<>, but C# value tuples are ValueTuple<>. - // Use a conversion helper that handles both cases. - if (ClrType.IsValueType) - return Expression.Call(ConvertMethod.MakeGenericMethod(ClrType), expression); - - return Expression.Convert(expression, ClrType); + // Two reasons to rebuild: the driver returns System.Tuple<> even where the CLR type is a + // ValueTuple<>, and a component whose CLR type differs from what the driver produces + // (DateTimeOffset and DateOnly both arrive as DateTime) cannot be cast in place. + var needsComponentConversion = ElementMappings.Any(ClickHouseComponentConversion.NeedsConversion); + + if (!ClrType.IsValueType && !needsComponentConversion) + return Expression.Convert(expression, ClrType); + + // Build the delegate array once and embed it as a constant. Expression.NewArrayInit would + // instead make the allocation part of the materializer, so every row read would allocate a + // fresh array of the same delegates — about 40 bytes per row on a two-component tuple. + var componentConverters = Expression.Constant( + ElementMappings.Select(ClickHouseComponentConversion.CreateComponentReader).ToArray(), + typeof(Func[])); + + return Expression.Call( + ConvertMethod.MakeGenericMethod(ClrType), + expression, + componentConverters, + Expression.Constant(ElementMappings.All(ClickHouseComponentConversion.CanPassThrough))); } - // Converts the driver's Tuple<> to ValueTuple<> (or passes through if already correct type) - private static T ConvertToValueTuple(object value) where T : struct + // Rebuilds the driver's tuple as T, converting each component. Handles both ValueTuple<> and + // System.Tuple<> targets, since both expose a constructor taking every component. + private static T ConvertTuple( + object value, + Func[] convertComponents, + bool canPassThrough) { - if (value is T t) - return t; + // A ValueTuple target never takes this path, because the driver returns System.Tuple<>. + // See ClickHouseComponentConversion.CanPassThrough for the component condition. + if (canPassThrough && value is T alreadyTyped) + return alreadyTyped; + + if (value is not ITuple tuple) + throw new InvalidCastException($"Cannot convert {value.GetType()} to {typeof(T)}"); + + if (tuple.Length != convertComponents.Length) + throw new InvalidCastException( + $"Cannot convert {value.GetType()} to {typeof(T)}: the value has {tuple.Length} " + + $"components but the mapping expects {convertComponents.Length}."); - // Driver returns System.Tuple<>, need to create ValueTuple<> from its elements - if (value is ITuple tuple) + var (factory, componentTypes) = ConstructorCache.GetOrAdd(typeof(T), static type => { - var args = new object?[tuple.Length]; - for (var i = 0; i < tuple.Length; i++) - args[i] = tuple[i]; + var constructor = type.GetConstructors()[0]; + var ctorParams = constructor.GetParameters(); + var argsParam = Expression.Parameter(typeof(object[]), "args"); + var bodyArgs = new Expression[ctorParams.Length]; - var factory = ConstructorCache.GetOrAdd(typeof(T), static type => + for (var j = 0; j < ctorParams.Length; j++) { - var ctorParams = type.GetConstructors()[0].GetParameters(); - var argsParam = Expression.Parameter(typeof(object[]), "args"); - var bodyArgs = new Expression[ctorParams.Length]; + bodyArgs[j] = Expression.Convert( + Expression.ArrayIndex(argsParam, Expression.Constant(j)), + ctorParams[j].ParameterType); + } + + return ( + (Delegate)Expression.Lambda>( + Expression.New(constructor, bodyArgs), argsParam).Compile(), + Array.ConvertAll(ctorParams, p => p.ParameterType)); + }); - for (var j = 0; j < ctorParams.Length; j++) + var args = new object?[tuple.Length]; + for (var i = 0; i < tuple.Length; i++) + { + var component = tuple[i]; + if (component is null or DBNull) + { + // The array and map helpers substitute default(T) for a null component. A tuple slot + // cannot: the constructor takes it positionally, so a null on a non-nullable + // value-type slot would fail inside the compiled factory as a NullReferenceException. + // ClickHouse only returns NULL for a Nullable(...) slot, so reaching this means the + // column and the CLR tuple disagree — say so plainly. + if (componentTypes[i].IsValueType && Nullable.GetUnderlyingType(componentTypes[i]) is null) { - bodyArgs[j] = Expression.Convert( - Expression.ArrayIndex(argsParam, Expression.Constant(j)), - ctorParams[j].ParameterType); + throw new InvalidCastException( + $"Cannot convert {value.GetType()} to {typeof(T)}: component {i} is NULL but " + + $"'{componentTypes[i]}' is not nullable. Declare that tuple component as " + + $"'{componentTypes[i]}?', or make the column non-nullable."); } - var body = Expression.New(type.GetConstructors()[0], bodyArgs); - return Expression.Lambda>(body, argsParam).Compile(); - }); + args[i] = null; + continue; + } - return ((Func)factory)(args!); + args[i] = convertComponents[i](component); } - throw new InvalidCastException($"Cannot convert {value.GetType()} to {typeof(T)}"); + return ((Func)factory)(args!); } protected override string GenerateNonNullSqlLiteral(object value) diff --git a/src/EFCore.ClickHouse/Update/Internal/ClickHouseModificationCommandBatch.cs b/src/EFCore.ClickHouse/Update/Internal/ClickHouseModificationCommandBatch.cs index 870b77b..70a5d1e 100644 --- a/src/EFCore.ClickHouse/Update/Internal/ClickHouseModificationCommandBatch.cs +++ b/src/EFCore.ClickHouse/Update/Internal/ClickHouseModificationCommandBatch.cs @@ -101,7 +101,19 @@ public override async Task ExecuteAsync( var row = new object[writeColumns.Count]; for (var i = 0; i < writeColumns.Count; i++) { - row[i] = writeColumns[i].Value ?? DBNull.Value; + var modification = writeColumns[i]; + var value = modification.Value; + + // This path gives the driver the model value directly, so a mapping cannot guard + // its own writes. Ask the ones whose store type has a narrower range than the CLR + // type, or the value would wrap and the row would read back wrong. + if (value is not null + && modification.TypeMapping is IClickHouseWriteValidatingTypeMapping validating) + { + validating.ValidateWriteValue(value, modification.ColumnName); + } + + row[i] = value ?? DBNull.Value; } return row; }); diff --git a/test/EFCore.ClickHouse.Tests/CompositeComponentClrTypeTests.cs b/test/EFCore.ClickHouse.Tests/CompositeComponentClrTypeTests.cs new file mode 100644 index 0000000..2c68ac4 --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/CompositeComponentClrTypeTests.cs @@ -0,0 +1,130 @@ +using ClickHouse.EntityFrameworkCore.Extensions; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +public enum ComponentColour +{ + Red = 0, + Green = 1 +} + +/// +/// Components whose CLR type a composite must take from the model rather than from the store type's +/// default. Each property here resolved to the wrong element type before, and the query then failed +/// to compile with a coercion error naming a type the user never wrote. +/// +public class ComponentClrTypeEntity +{ + public long Id { get; set; } + + /// A converter-backed component: the enum mapping's converter used to erase the + /// Nullable<>, so this resolved as ComponentColour[]. + public ComponentColour?[] Colours { get; set; } = []; + + /// Date32 serves both and , and the + /// store type's default won. + public DateTime[] Timestamps { get; set; } = []; + + /// ClickHouse accepts and normalizes this spelling; the provider's parser did not. + public DateOnly?[] Spaced { get; set; } = []; +} + +public class ComponentClrTypeDbContext(string connectionString) : DbContext +{ + public DbSet Entities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + => modelBuilder.Entity(e => + { + e.ToTable("component_clr_types", t => t.HasMergeTreeEngine().WithOrderBy("id")); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Colours).HasColumnName("colours") + .HasColumnType("Array(Nullable(Enum8('Red' = 0, 'Green' = 1)))"); + e.Property(x => x.Timestamps).HasColumnName("timestamps").HasColumnType("Array(Date32)"); + e.Property(x => x.Spaced).HasColumnName("spaced").HasColumnType("Array( Nullable ( Date32 ) )"); + }); +} + +public class ComponentClrTypeFixture : IAsyncLifetime +{ + public string ConnectionString { get; private set; } = string.Empty; + + public async Task InitializeAsync() + { + ConnectionString = await SharedContainer.GetConnectionStringAsync(); + using var ctx = new ComponentClrTypeDbContext(ConnectionString); + await ctx.Database.EnsureCreatedAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; +} + +public class CompositeComponentClrTypeTests(ComponentClrTypeFixture fixture) + : IClassFixture +{ + private Type MappedClrTypeOf(string propertyName) + { + using var ctx = new ComponentClrTypeDbContext(fixture.ConnectionString); + return ctx.Model.FindEntityType(typeof(ComponentClrTypeEntity))! + .FindProperty(propertyName)! + .FindRelationalTypeMapping()! + .ClrType; + } + + /// + /// A converter on the component mapping made EF Core take the mapping's CLR type from the + /// converter's model type, discarding the Nullable<> the wrapper asked for. + /// + [Fact] + public void A_converter_backed_component_keeps_its_nullable_clr_type() + => Assert.Equal(typeof(ComponentColour?[]), MappedClrTypeOf(nameof(ComponentClrTypeEntity.Colours))); + + /// One store type serves several CLR types, so the model's choice has to win. + [Fact] + public void A_component_resolves_as_the_clr_type_the_model_declares() + => Assert.Equal(typeof(DateTime[]), MappedClrTypeOf(nameof(ComponentClrTypeEntity.Timestamps))); + + /// + /// ClickHouse tolerates whitespace before an argument list and normalizes it away, so the + /// provider's parsers must agree with each other on such a spelling. + /// + [Fact] + public void Whitespace_in_a_store_type_still_yields_a_nullable_element() + => Assert.Equal(typeof(DateOnly?[]), MappedClrTypeOf(nameof(ComponentClrTypeEntity.Spaced))); + + /// Every property above must also survive query compilation and materialization. + [Fact] + public async Task Components_round_trip_through_the_database() + { + var timestamps = new[] { new DateTime(2026, 1, 15), new DateTime(2026, 6, 30) }; + var spaced = new DateOnly?[] { new DateOnly(2026, 2, 1), null }; + + using (var write = new ComponentClrTypeDbContext(fixture.ConnectionString)) + { + write.Entities.Add(new ComponentClrTypeEntity + { + Id = 1, + // An enum inside a composite is still written as its raw ordinal (#54), so the + // round trip asserted here is on the components that need no converter. + Colours = [ComponentColour.Red, null], + Timestamps = timestamps, + Spaced = spaced + }); + await write.SaveChangesAsync(); + } + + using var read = new ComponentClrTypeDbContext(fixture.ConnectionString); + var row = await read.Entities.SingleAsync(x => x.Id == 1); + + Assert.Equal(timestamps, row.Timestamps); + Assert.Equal(spaced, row.Spaced); + } + +} diff --git a/test/EFCore.ClickHouse.Tests/CompositeElementConversionTests.cs b/test/EFCore.ClickHouse.Tests/CompositeElementConversionTests.cs new file mode 100644 index 0000000..2901c12 --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/CompositeElementConversionTests.cs @@ -0,0 +1,663 @@ +using System.Linq.Expressions; +using ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Storage; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +/// +/// A composite mapping reads its whole column with GetValue, so the component mappings' +/// read conversions have to be applied per component. Both and +/// arrive from the driver as , so without that the +/// whole composite fails to cast. +/// +public class CompositeElementEntity +{ + public long Id { get; set; } + public DateTimeOffset[] Offsets { get; set; } = []; + public DateOnly[] Dates { get; set; } = []; + public List OffsetList { get; set; } = []; + public DateTimeOffset[][] NestedOffsets { get; set; } = []; + public Dictionary OffsetsByName { get; set; } = []; + public Dictionary NamesByDate { get; set; } = []; + public Tuple? OffsetTuple { get; set; } + public (DateTimeOffset When, int Count) OffsetValueTuple { get; set; } + public int[] Ints { get; set; } = []; + public Dictionary Counts { get; set; } = []; +} + +public class NullableCompositeElementEntity +{ + public long Id { get; set; } + public DateTimeOffset?[] Offsets { get; set; } = []; + public DateOnly?[] Dates { get; set; } = []; +} + +public enum CompositeColour +{ + Red, + Green, + Blue +} + +/// +/// Components that carry a ValueConverter rather than a data-reader conversion: an enum +/// converts through EnumToStringConverter, and a List<T> component through +/// ListToArrayConverter. The composite has to apply both, or the column becomes writable but +/// unreadable. +/// +public class ConvertedCompositeEntity +{ + public long Id { get; set; } + public CompositeColour[] Colours { get; set; } = []; + public Tuple? ColourTuple { get; set; } + public Dictionary ColourByName { get; set; } = []; + public Dictionary> Buckets { get; set; } = []; + public List> Nested { get; set; } = []; + public List[] ListArray { get; set; } = []; + public IList Interfaced { get; set; } = new List(); + public IReadOnlyList ReadOnlyDates { get; set; } = []; + + // Components that need no conversion at all — these must keep the direct cast. + public string[] Names { get; set; } = []; + public double[] Ratios { get; set; } = []; + public Dictionary Labels { get; set; } = []; +} + +public class CompositeElementDbContext : DbContext +{ + private readonly string _connectionString; + + public CompositeElementDbContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + public DbSet NullableEntities => Set(); + public DbSet ConvertedEntities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(e => + { + e.ToTable("composite_elements"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Offsets).HasColumnName("offsets"); + e.Property(x => x.Dates).HasColumnName("dates"); + e.Property(x => x.OffsetList).HasColumnName("offset_list"); + e.Property(x => x.NestedOffsets).HasColumnName("nested_offsets"); + e.Property(x => x.OffsetsByName).HasColumnName("offsets_by_name"); + e.Property(x => x.NamesByDate).HasColumnName("names_by_date"); + e.Property(x => x.OffsetTuple).HasColumnName("offset_tuple"); + e.Property(x => x.OffsetValueTuple).HasColumnName("offset_value_tuple"); + e.Property(x => x.Ints).HasColumnName("ints"); + e.Property(x => x.Counts).HasColumnName("counts"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("nullable_composite_elements"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + // The resolver takes element nullability from the store type. EF Core also models it on + // IElementType.IsNullable for a primitive collection, which the resolver does not use yet. + e.Property(x => x.Offsets).HasColumnName("offsets") + .HasColumnType("Array(Nullable(DateTime64(7, 'UTC')))"); + e.Property(x => x.Dates).HasColumnName("dates") + .HasColumnType("Array(Nullable(Date32))"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("converted_composites"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Colours).HasColumnName("colours") + .HasColumnType("Array(Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3))"); + e.Property(x => x.ColourTuple).HasColumnName("colour_tuple") + .HasColumnType("Tuple(Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3), Int32)"); + e.Property(x => x.ColourByName).HasColumnName("colour_by_name") + .HasColumnType("Map(String, Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3))"); + e.Property(x => x.Buckets).HasColumnName("buckets"); + e.Property(x => x.Nested).HasColumnName("nested"); + e.Property(x => x.ListArray).HasColumnName("list_array"); + e.Property(x => x.Interfaced).HasColumnName("interfaced"); + e.Property(x => x.ReadOnlyDates).HasColumnName("readonly_dates"); + e.Property(x => x.Names).HasColumnName("names"); + e.Property(x => x.Ratios).HasColumnName("ratios"); + e.Property(x => x.Labels).HasColumnName("labels"); + }); + } +} + +public class ElementConverterEntity +{ + public long Id { get; set; } + public string[] Tags { get; set; } = []; +} + +/// +/// An element converter that keeps the CLR type, set through EF Core's public +/// ElementType().HasConversion(...) API. +/// +public class ElementConverterDbContext : DbContext +{ + private readonly string _connectionString; + + public ElementConverterDbContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + => modelBuilder.Entity(e => + { + e.ToTable("element_converted"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.PrimitiveCollection(x => x.Tags).HasColumnName("tags") + .ElementType(el => el.HasConversion( + new ValueConverter(v => v, v => v + "!"))); + }); +} + +public class CompositeElementConversionFixture : IAsyncLifetime +{ + public string ConnectionString { get; private set; } = string.Empty; + + public async Task InitializeAsync() + { + ConnectionString = await SharedContainer.GetConnectionStringAsync(); + using var ctx = new CompositeElementDbContext(ConnectionString); + await ctx.Database.EnsureCreatedAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; +} + +public class CompositeElementConversionTests : IClassFixture +{ + private readonly CompositeElementConversionFixture _fixture; + + public CompositeElementConversionTests(CompositeElementConversionFixture fixture) + => _fixture = fixture; + + private static readonly DateTimeOffset Instant = new(2026, 1, 15, 5, 0, 0, TimeSpan.Zero); + + private static CompositeElementEntity NewRow(long id) => new() + { + Id = id, + Offsets = [Instant, Instant.AddDays(1)], + Dates = [new DateOnly(2026, 1, 15), new DateOnly(2026, 2, 20)], + OffsetList = [Instant, Instant.AddHours(3)], + NestedOffsets = [[Instant], [Instant.AddDays(2), Instant.AddDays(3)]], + OffsetsByName = new Dictionary { ["start"] = Instant, ["end"] = Instant.AddDays(5) }, + NamesByDate = new Dictionary { [new DateOnly(2026, 3, 1)] = "march" }, + OffsetTuple = Tuple.Create(Instant, "reference"), + OffsetValueTuple = (Instant.AddDays(7), 42), + Ints = [1, 2, 3], + Counts = new Dictionary { ["a"] = 1, ["b"] = 2 }, + }; + + [Fact] + public async Task Store_types_are_the_expected_composites() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "DESCRIBE TABLE composite_elements"; + + var columns = new Dictionary(); + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + columns[(string)reader.GetValue(0)] = (string)reader.GetValue(1); + + Assert.Equal("Array(DateTime64(7, 'UTC'))", columns["offsets"]); + Assert.Equal("Array(Date32)", columns["dates"]); + Assert.Equal("Array(Array(DateTime64(7, 'UTC')))", columns["nested_offsets"]); + Assert.Equal("Map(String, DateTime64(7, 'UTC'))", columns["offsets_by_name"]); + Assert.Equal("Map(Date32, String)", columns["names_by_date"]); + Assert.Equal("Tuple(DateTime64(7, 'UTC'), String)", columns["offset_tuple"]); + } + + [Fact] + public async Task Every_composite_shape_round_trips() + { + using var writeContext = new CompositeElementDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(NewRow(1)); + await writeContext.SaveChangesAsync(); + + using var readContext = new CompositeElementDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 1); + var expected = NewRow(1); + + Assert.Equal(expected.Offsets, row.Offsets); + Assert.Equal(expected.Dates, row.Dates); + Assert.Equal(expected.OffsetList, row.OffsetList); + Assert.Equal(expected.OffsetsByName, row.OffsetsByName); + Assert.Equal(expected.NamesByDate, row.NamesByDate); + Assert.Equal(expected.OffsetTuple, row.OffsetTuple); + Assert.Equal(expected.OffsetValueTuple, row.OffsetValueTuple); + Assert.Equal(expected.Ints, row.Ints); + Assert.Equal(expected.Counts, row.Counts); + + // Nested arrays compose: the element mapping is itself an array mapping. + Assert.Equal(expected.NestedOffsets.Length, row.NestedOffsets.Length); + for (var i = 0; i < expected.NestedOffsets.Length; i++) + Assert.Equal(expected.NestedOffsets[i], row.NestedOffsets[i]); + } + + /// Projecting the column alone exercises the mapping without entity materialization. + [Fact] + public async Task Projecting_a_composite_column_converts_its_elements() + { + using var writeContext = new CompositeElementDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(NewRow(2)); + await writeContext.SaveChangesAsync(); + + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + + Assert.Equal([Instant, Instant.AddDays(1)], await ctx.Entities.Where(e => e.Id == 2).Select(e => e.Offsets).SingleAsync()); + Assert.Equal([new DateOnly(2026, 1, 15), new DateOnly(2026, 2, 20)], await ctx.Entities.Where(e => e.Id == 2).Select(e => e.Dates).SingleAsync()); + Assert.Equal(Tuple.Create(Instant, "reference"), await ctx.Entities.Where(e => e.Id == 2).Select(e => e.OffsetTuple).SingleAsync()); + } + + [Fact] + public async Task Empty_composites_round_trip() + { + using var writeContext = new CompositeElementDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(new CompositeElementEntity + { + Id = 3, + Offsets = [], + Dates = [], + OffsetList = [], + NestedOffsets = [], + OffsetsByName = [], + NamesByDate = [], + OffsetTuple = Tuple.Create(Instant, "only"), + OffsetValueTuple = (Instant, 0), + Ints = [], + Counts = [], + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new CompositeElementDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 3); + + Assert.Empty(row.Offsets); + Assert.Empty(row.Dates); + Assert.Empty(row.OffsetList); + Assert.Empty(row.NestedOffsets); + Assert.Empty(row.OffsetsByName); + } + + /// + /// The nullable wrapper must add exactly one Nullable(...). The inner mapping is resolved + /// from a store type that already carries the wrapper, and that text is preserved verbatim, so + /// wrapping again gave Array(Nullable(Nullable(T))) — DDL that ClickHouse rejects. + /// + [Fact] + public async Task Nullable_element_store_type_is_not_double_wrapped() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var entityType = ctx.Model.FindEntityType(typeof(NullableCompositeElementEntity))!; + + Assert.Equal( + "Array(Nullable(DateTime64(7, 'UTC')))", + entityType.FindProperty(nameof(NullableCompositeElementEntity.Offsets))!.GetColumnType()); + Assert.Equal( + "Array(Nullable(Date32))", + entityType.FindProperty(nameof(NullableCompositeElementEntity.Dates))!.GetColumnType()); + + // And the table really exists with that shape, so EnsureCreated accepted the DDL. + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "DESCRIBE TABLE nullable_composite_elements"; + + var columns = new Dictionary(); + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + columns[(string)reader.GetValue(0)] = (string)reader.GetValue(1); + + Assert.Equal("Array(Nullable(DateTime64(7, 'UTC')))", columns["offsets"]); + Assert.Equal("Array(Nullable(Date32))", columns["dates"]); + } + + /// + /// Array(Nullable(T)) goes through ClickHouseNullableElementMapping, which must + /// delegate the read conversion to the inner mapping while nulls pass straight through. + /// + [Fact] + public async Task Nullable_elements_round_trip_with_nulls() + { + using var writeContext = new CompositeElementDbContext(_fixture.ConnectionString); + writeContext.NullableEntities.Add(new NullableCompositeElementEntity + { + Id = 1, + Offsets = [Instant, null, Instant.AddDays(1)], + Dates = [new DateOnly(2026, 1, 15), null], + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new CompositeElementDbContext(_fixture.ConnectionString); + var row = await readContext.NullableEntities.SingleAsync(e => e.Id == 1); + + Assert.Equal([Instant, null, Instant.AddDays(1)], row.Offsets); + Assert.Equal([new DateOnly(2026, 1, 15), null], row.Dates); + } + + /// + /// A component that needs no conversion must keep the direct cast, so the common case pays + /// nothing. Asserted on the shape of the read expression rather than on a round trip, because a + /// round trip passes either way. String and Float64 convert nothing; note that the + /// integer mappings do (they widen with Convert.ToInt32 for aggregates), so + /// int[] is not a valid example here. + /// + [Theory] + [InlineData(nameof(ConvertedCompositeEntity.Names))] + [InlineData(nameof(ConvertedCompositeEntity.Ratios))] + [InlineData(nameof(ConvertedCompositeEntity.Labels))] + public void Components_needing_no_conversion_keep_the_direct_cast(string propertyName) + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var mapping = ctx.Model + .FindEntityType(typeof(ConvertedCompositeEntity))! + .FindProperty(propertyName)! + .GetRelationalTypeMapping(); + + var read = mapping.CustomizeDataReaderExpression(Expression.Parameter(typeof(object), "v")); + + // A direct cast is a UnaryExpression; the rebuild path emits a Call to a Convert* helper. + Assert.IsAssignableFrom(read); + } + + /// The counterpart: a converting component must take the rebuild path. + [Theory] + [InlineData(nameof(CompositeElementEntity.Offsets))] + [InlineData(nameof(CompositeElementEntity.Dates))] + [InlineData(nameof(CompositeElementEntity.OffsetsByName))] + [InlineData(nameof(CompositeElementEntity.Ints))] + public void Converting_components_take_the_rebuild_path(string propertyName) + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var mapping = ctx.Model + .FindEntityType(typeof(CompositeElementEntity))! + .FindProperty(propertyName)! + .GetRelationalTypeMapping(); + + var read = mapping.CustomizeDataReaderExpression(Expression.Parameter(typeof(object), "v")); + + Assert.IsAssignableFrom(read); + } + + /// + /// The per-component converter must be embedded as a constant, not as an inline lambda. An inline + /// lambda is rebuilt on every materialization, which allocates a delegate per row for every + /// composite column that converts — including ones where the runtime fast path then returns the + /// driver's array untouched. + /// + [Theory] + [InlineData(nameof(CompositeElementEntity.Offsets))] + [InlineData(nameof(CompositeElementEntity.Ints))] + [InlineData(nameof(CompositeElementEntity.OffsetsByName))] + public void Component_converters_are_embedded_as_constants(string propertyName) + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var mapping = ctx.Model + .FindEntityType(typeof(CompositeElementEntity))! + .FindProperty(propertyName)! + .GetRelationalTypeMapping(); + + var read = (MethodCallExpression)mapping.CustomizeDataReaderExpression( + Expression.Parameter(typeof(object), "v")); + + // Argument 0 is the raw value; every argument after it is a component converter. + Assert.All( + read.Arguments.Skip(1), + argument => Assert.Equal(ExpressionType.Constant, argument.NodeType)); + } + + // --- components carrying a ValueConverter ------------------------------- + + private static ConvertedCompositeEntity NewConvertedRow(long id) => new() + { + Id = id, + Colours = [CompositeColour.Red, CompositeColour.Blue], + ColourTuple = Tuple.Create(CompositeColour.Blue, 7), + ColourByName = new Dictionary { ["primary"] = CompositeColour.Green }, + Buckets = new Dictionary> { ["low"] = [1, 2], ["high"] = [9] }, + Nested = [[1, 2], [3]], + ListArray = [[4, 5], []], + Interfaced = new List { Instant, Instant.AddDays(1) }, + ReadOnlyDates = [new DateOnly(2026, 4, 1)], + Names = ["a", "b"], + Ratios = [1.5, 2.5], + Labels = new Dictionary { ["k"] = "v" }, + }; + + /// + /// A component mapping can convert through a ValueConverter instead of a data-reader + /// conversion. The composite must apply that too — otherwise the column is writable but not + /// readable, which is worse than refusing the model up front. + /// + /// + /// Seeded with raw SQL on purpose. Writing a converter-bearing component through + /// SaveChanges does not work yet: the bulk insert path passes model values straight to the + /// driver without applying the converter, so an enum component is written as its raw ordinal + /// (issue #54). This test covers the read direction, which is what the composite conversion + /// fixes. + /// + [Fact] + public async Task Components_with_a_value_converter_read_correctly() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var insert = connection.CreateCommand(); + insert.CommandText = """ + INSERT INTO converted_composites + (id, colours, colour_tuple, colour_by_name, buckets, nested, list_array, + interfaced, readonly_dates, names, ratios, labels) + VALUES + (1, ['Red', 'Blue'], ('Blue', 7), {'primary': 'Green'}, + {'low': [1, 2], 'high': [9]}, [[1, 2], [3]], [[4, 5], []], + ['2026-01-15 05:00:00.0000000', '2026-01-16 05:00:00.0000000'], + ['2026-04-01'], ['a', 'b'], [1.5, 2.5], {'k': 'v'}) + """; + await insert.ExecuteNonQueryAsync(); + + using var readContext = new CompositeElementDbContext(_fixture.ConnectionString); + var row = await readContext.ConvertedEntities.SingleAsync(e => e.Id == 1); + var expected = NewConvertedRow(1); + + // EnumToStringConverter components. + Assert.Equal(expected.Colours, row.Colours); + Assert.Equal(expected.ColourTuple, row.ColourTuple); + Assert.Equal(expected.ColourByName, row.ColourByName); + + // ListToArrayConverter components, including inside a Map and nested one level. + Assert.Equal(expected.Buckets, row.Buckets); + Assert.Equal(expected.Nested, row.Nested); + Assert.Equal(expected.ListArray.Length, row.ListArray.Length); + for (var i = 0; i < expected.ListArray.Length; i++) + Assert.Equal(expected.ListArray[i], row.ListArray[i]); + + // Collection-interface components go through EnumerableToArrayConverter. + Assert.Equal(expected.Interfaced, row.Interfaced); + Assert.Equal(expected.ReadOnlyDates, row.ReadOnlyDates); + + // And the no-conversion components still work. + Assert.Equal(expected.Names, row.Names); + Assert.Equal(expected.Ratios, row.Ratios); + Assert.Equal(expected.Labels, row.Labels); + } + + [Fact] + public async Task Enum_component_store_types_are_as_configured() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "DESCRIBE TABLE converted_composites"; + + var columns = new Dictionary(); + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + columns[(string)reader.GetValue(0)] = (string)reader.GetValue(1); + + Assert.Equal("Array(Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3))", columns["colours"]); + Assert.Equal("Map(String, Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3))", columns["colour_by_name"]); + Assert.Equal("Map(String, Array(Int32))", columns["buckets"]); + Assert.Equal("Array(Array(Int32))", columns["nested"]); + } + + // --- the already-typed pass-through ------------------------------------- + + /// + /// Reading a composite keeps a fast path for a driver value that already has the target CLR + /// type. That is sound only where matching types prove there is no work left, which a + /// ValueConverter can break: it may change the value and keep the CLR type, so the + /// driver's array is already string[] while ConvertFromProvider still has to run. + /// + /// + /// No mapping the provider resolves on its own is shaped this way — every converter it uses also + /// changes the CLR type — but the shape is reachable from the public API through + /// ElementType().HasConversion(...), which the model test below covers. This one drives + /// the mapping directly so the fast path is exercised without a model. + /// + [Fact] + public void A_same_clr_type_component_converter_is_not_skipped_by_the_fast_path() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + var stringMapping = source.FindMapping(typeof(string), "String")!; + + // A converter that keeps the CLR type but changes the value. + var elementMapping = (RelationalTypeMapping)stringMapping.WithComposedConverter( + new ValueConverter(v => v, v => v + "!")); + var arrayMapping = new ClickHouseArrayTypeMapping(elementMapping); + + Assert.Equal(typeof(string[]), arrayMapping.ClrType); + Assert.NotNull(elementMapping.Converter); + + // The driver hands back string[], which is already the target type. + var read = Read(arrayMapping, new[] { "a", "b" }); + + Assert.Equal(["a!", "b!"], read); + } + + /// + /// The fast path must survive for the case it exists to serve: a component whose read is a cast + /// and nothing more, as in a nested array. Here the driver's value is returned as it stands. + /// + [Fact] + public void An_already_typed_component_with_no_converter_passes_straight_through() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + var innerArray = source.FindMapping(typeof(int[]), "Array(Int32)")!; + var outerArray = new ClickHouseArrayTypeMapping(innerArray); + + // No converter, so a matching CLR type is proof enough that nothing is left to do. + Assert.Null(innerArray.Converter); + + var driverValue = new[] { new[] { 1, 2 }, new[] { 3 } }; + var read = Read(outerArray, driverValue); + + Assert.Same(driverValue, read); + } + + /// + /// The same hazard for a Map. Both the key and the value mapping must be checked, so this + /// puts the converter on the value and leaves the key alone. + /// + [Fact] + public void A_same_clr_type_map_value_converter_is_not_skipped_by_the_fast_path() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + var stringMapping = source.FindMapping(typeof(string), "String")!; + + var valueMapping = (RelationalTypeMapping)stringMapping.WithComposedConverter( + new ValueConverter(v => v, v => v + "!")); + var mapMapping = new ClickHouseMapTypeMapping(stringMapping, valueMapping); + + // The driver hands back the target dictionary type already. + var read = Read>( + mapMapping, + new Dictionary { ["k"] = "a" }); + + Assert.Equal("a!", read["k"]); + } + + /// + /// The same hazard for a Tuple. A reference tuple is used, because a ValueTuple + /// target never reaches the fast path — the driver returns System.Tuple<>. + /// + [Fact] + public void A_same_clr_type_tuple_component_converter_is_not_skipped_by_the_fast_path() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + var stringMapping = source.FindMapping(typeof(string), "String")!; + + var componentMapping = (RelationalTypeMapping)stringMapping.WithComposedConverter( + new ValueConverter(v => v, v => v + "!")); + var tupleMapping = new ClickHouseTupleTypeMapping( + [componentMapping, componentMapping], + useValueTuple: false); + + Assert.Equal(typeof(Tuple), tupleMapping.ClrType); + + var read = Read>(tupleMapping, Tuple.Create("a", "b")); + + Assert.Equal(Tuple.Create("a!", "b!"), read); + } + + /// + /// The reachable route to the same shape: an element converter set through the public API. This + /// is why the gate matters rather than being defence against a shape nobody can build. + /// + [Fact] + public async Task An_element_converter_set_on_the_model_is_applied_on_read() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + + // The fixture already created the database, so EnsureCreated would add nothing. Written + // outside EF anyway, because SaveChanges does not apply converters yet (#54). + using var create = connection.CreateCommand(); + create.CommandText = + "CREATE TABLE IF NOT EXISTS element_converted (id Int64, tags Array(String)) " + + "ENGINE = MergeTree ORDER BY id"; + await create.ExecuteNonQueryAsync(); + + using var command = connection.CreateCommand(); + command.CommandText = "INSERT INTO element_converted VALUES (1, ['a', 'b'])"; + await command.ExecuteNonQueryAsync(); + + using var readContext = new ElementConverterDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 1); + + // Without the gate the driver's raw string[] would come straight through as "a", "b". + Assert.Equal(["a!", "b!"], row.Tags); + } + + /// Compiles and runs a mapping's data-reader expression over one driver value. + private static T Read(RelationalTypeMapping mapping, object driverValue) + { + var parameter = Expression.Parameter(typeof(object), "value"); + var body = mapping.CustomizeDataReaderExpression(parameter); + + return Expression.Lambda>(Expression.Convert(body, typeof(T)), parameter) + .Compile()(driverValue); + } +} diff --git a/test/EFCore.ClickHouse.Tests/DateTimeOffsetMappingTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeOffsetMappingTests.cs new file mode 100644 index 0000000..2737be8 --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/DateTimeOffsetMappingTests.cs @@ -0,0 +1,1129 @@ +using ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +public class DateTimeOffsetEntity +{ + public long Id { get; set; } + public DateTimeOffset Default { get; set; } + public DateTimeOffset? Nullable { get; set; } + public double Value { get; set; } +} + +/// Points DateTimeOffset properties at columns with differing declared timezones. +public class DateTimeOffsetTzEntity +{ + public long Id { get; set; } + public DateTimeOffset Utc { get; set; } + public DateTimeOffset Naive { get; set; } + public DateTimeOffset Tokyo { get; set; } + public DateTimeOffset Seconds { get; set; } +} + +/// A zone with daylight saving, for the ambiguous-wall-clock case. +public class DateTimeOffsetDstEntity +{ + public long Id { get; set; } + public DateTimeOffset London { get; set; } +} + +/// Precision set through HasPrecision rather than HasColumnType. +public class DateTimeOffsetPrecisionEntity +{ + public long Id { get; set; } + public DateTimeOffset Millis { get; set; } + + /// Precision 9 reaches only 1678-2262, so it cannot hold every DateTimeOffset. + public DateTimeOffset Nanos { get; set; } +} + +/// +/// Columns that declare a fixed UTC offset instead of a named zone. ClickHouse spells these +/// Fixed/UTC±HH:MM:SS, and .NET has no timezone of that name. +/// +public class DateTimeOffsetFixedEntity +{ + public long Id { get; set; } + public DateTimeOffset Half { get; set; } + public DateTimeOffset Negative { get; set; } + public DateTimeOffset Quarter { get; set; } + public DateTimeOffset Zero { get; set; } + public DateTimeOffset Seconds { get; set; } +} + +/// +/// A column whose fixed-offset name carries minutes above 59. ClickHouse accepts the name, the +/// driver cannot read it, so the read must report that rather than guess an offset. +/// +public class DateTimeOffsetCarriedEntity +{ + public long Id { get; set; } + public DateTimeOffset Carried { get; set; } +} + +public class DateTimeOffsetDbContext : DbContext +{ + private readonly string _connectionString; + + public DateTimeOffsetDbContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + public DbSet TzEntities => Set(); + public DbSet DstEntities => Set(); + public DbSet PrecisionEntities => Set(); + public DbSet FixedEntities => Set(); + public DbSet CarriedEntities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(e => + { + e.ToTable("dto_default"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Default).HasColumnName("dt"); + e.Property(x => x.Nullable).HasColumnName("dt_null"); + e.Property(x => x.Value).HasColumnName("value"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_tz"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Utc).HasColumnName("utc").HasColumnType("DateTime64(6, 'UTC')"); + // A timezone-less column only round trips here because the test container's + // session_timezone is UTC. This is the hazard the default UTC pin avoids. + e.Property(x => x.Naive).HasColumnName("naive").HasColumnType("DateTime64(6)"); + e.Property(x => x.Tokyo).HasColumnName("tokyo").HasColumnType("DateTime64(6, 'Asia/Tokyo')"); + e.Property(x => x.Seconds).HasColumnName("seconds").HasColumnType("DateTime('UTC')"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_dst"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.London).HasColumnName("london").HasColumnType("DateTime64(7, 'Europe/London')"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_precision"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + // Precision only — no HasColumnType, so the UTC pin must be kept. + e.Property(x => x.Millis).HasColumnName("millis").HasPrecision(3); + e.Property(x => x.Nanos).HasColumnName("nanos").HasColumnType("DateTime64(9, 'UTC')"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_fixed"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Half).HasColumnName("half") + .HasColumnType("DateTime64(7, 'Fixed/UTC+05:30:00')"); + e.Property(x => x.Negative).HasColumnName("negative") + .HasColumnType("DateTime64(7, 'Fixed/UTC-07:00:00')"); + // A quarter-hour offset, which no whole-hour shortcut would handle. + e.Property(x => x.Quarter).HasColumnName("quarter") + .HasColumnType("DateTime64(7, 'Fixed/UTC+05:45:00')"); + // Offset zero: the driver reports Kind=Utc here rather than a wall clock. + e.Property(x => x.Zero).HasColumnName("zero") + .HasColumnType("DateTime64(7, 'Fixed/UTC+00:00:00')"); + // A whole-minute offset written with the seconds field the name always carries. + e.Property(x => x.Seconds).HasColumnName("seconds") + .HasColumnType("DateTime64(7, 'Fixed/UTC+00:01:00')"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_carried"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + // ClickHouse reads this as +06:00. The driver does not read it at all. + e.Property(x => x.Carried).HasColumnName("carried") + .HasColumnType("DateTime64(7, 'Fixed/UTC+05:60:00')"); + }); + } +} + +public class StringDateTimeOffsetEntity +{ + public long Id { get; set; } + public DateTimeOffset ViaConversion { get; set; } + public DateTimeOffset ViaColumnType { get; set; } +} + +public class StringDateTimeOffsetDbContext : DbContext +{ + private readonly string _connectionString; + + public StringDateTimeOffsetDbContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(e => + { + e.ToTable("dto_string"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.ViaConversion).HasColumnName("via_conversion").HasConversion(); + e.Property(x => x.ViaColumnType).HasColumnName("via_column_type").HasColumnType("String"); + }); + } +} + +public class DateTimeOffsetMappingFixture : IAsyncLifetime +{ + public string ConnectionString { get; private set; } = string.Empty; + + public async Task InitializeAsync() + { + ConnectionString = await SharedContainer.GetConnectionStringAsync(); + using var ctx = new DateTimeOffsetDbContext(ConnectionString); + await ctx.Database.EnsureCreatedAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; +} + +public class DateTimeOffsetMappingTests : IClassFixture +{ + private readonly DateTimeOffsetMappingFixture _fixture; + + public DateTimeOffsetMappingTests(DateTimeOffsetMappingFixture fixture) => _fixture = fixture; + + private static long ToMicros(DateTimeOffset value) => value.ToUnixTimeMilliseconds() * 1000L; + + // --- mapping resolution ------------------------------------------------- + + [Fact] + public void Default_mapping_is_utc_pinned_datetime64() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetEntity))! + .FindProperty(nameof(DateTimeOffsetEntity.Default))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.IsType(mapping); + Assert.Equal(typeof(DateTimeOffset), mapping.ClrType); + Assert.Equal("DateTime64(7, 'UTC')", mapping.StoreType); + // The old behaviour resolved a String column through DateTimeOffsetToStringConverter. + Assert.Null(mapping.Converter); + } + + [Fact] + public void Nullable_property_resolves_the_same_mapping() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetEntity))! + .FindProperty(nameof(DateTimeOffsetEntity.Nullable))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.IsType(mapping); + Assert.Equal("DateTime64(7, 'UTC')", mapping.StoreType); + } + + [Theory] + [InlineData(nameof(DateTimeOffsetTzEntity.Utc), "DateTime64(6, 'UTC')")] + [InlineData(nameof(DateTimeOffsetTzEntity.Naive), "DateTime64(6)")] + [InlineData(nameof(DateTimeOffsetTzEntity.Tokyo), "DateTime64(6, 'Asia/Tokyo')")] + [InlineData(nameof(DateTimeOffsetTzEntity.Seconds), "DateTime('UTC')")] + public void Explicit_store_type_is_preserved(string propertyName, string expectedStoreType) + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetTzEntity))! + .FindProperty(propertyName)!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.IsType(mapping); + Assert.Equal(typeof(DateTimeOffset), mapping.ClrType); + Assert.Equal(expectedStoreType, property.GetColumnType()); + } + + [Fact] + public async Task EnsureCreated_makes_a_datetime64_column_not_a_string() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "DESCRIBE TABLE dto_default"; + + var columns = new Dictionary(); + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + columns[(string)reader.GetValue(0)] = (string)reader.GetValue(1); + + Assert.Equal("DateTime64(7, 'UTC')", columns["dt"]); + Assert.Equal("Nullable(DateTime64(7, 'UTC'))", columns["dt_null"]); + } + + /// + /// HasPrecision(n) with no HasColumnType must change the precision and keep the + /// UTC pin. Dropping the facet would silently give the default precision 7 instead. + /// + [Fact] + public void HasPrecision_is_honoured_and_keeps_the_utc_pin() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetPrecisionEntity))! + .FindProperty(nameof(DateTimeOffsetPrecisionEntity.Millis))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.Equal("DateTime64(3, 'UTC')", mapping.StoreType); + Assert.Equal("DateTime64(3, 'UTC')", property.GetColumnType()); + Assert.Equal(3, mapping.Precision); + } + + /// A bare DateTime64 with no argument is precision 3 in ClickHouse, not 7. + [Fact] + public void Bare_datetime64_store_type_uses_clickhouse_default_precision() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + + var mapping = source.FindMapping(typeof(DateTimeOffset), "DateTime64"); + + Assert.Equal(3, mapping!.Precision); + } + + [Fact] + public void Bare_datetime_store_type_maps_to_seconds_precision() + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision: null, timezone: null); + + Assert.Equal("DateTime", mapping.StoreType); + Assert.Null(mapping.Precision); + } + + /// + /// When the configured text differs from the mapping's canonical store type, + /// PreserveExplicitStoreType clones the mapping to keep that text. The clone must carry + /// Timezone over, or the read path would lose the offset of the declared zone. + /// + [Theory] + [InlineData("DateTime64(6,'Asia/Tokyo')", "Asia/Tokyo")] + [InlineData("Nullable(DateTime64(6, 'Asia/Tokyo'))", "Asia/Tokyo")] + [InlineData("LowCardinality(DateTime64(6, 'Asia/Tokyo'))", "Asia/Tokyo")] + [InlineData("datetime64(6, 'Asia/Tokyo')", "Asia/Tokyo")] + public void Clone_for_a_preserved_store_type_keeps_the_timezone(string columnType, string expectedTimezone) + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + + var mapping = source.FindMapping(typeof(DateTimeOffset), columnType); + + var typed = Assert.IsType(mapping); + Assert.Equal(typeof(DateTimeOffset), typed.ClrType); + // The user's text survives verbatim... + Assert.Equal(columnType, typed.StoreType); + // ...and the timezone needed by the read path survives the clone. + Assert.Equal(expectedTimezone, typed.Timezone); + } + + // --- SQL literals ------------------------------------------------------- + + [Theory] + [InlineData(7, "UTC", "DateTime64(7, 'UTC')", "'2026-01-15 10:00:00.1234567+05:00'")] + [InlineData(6, "UTC", "DateTime64(6, 'UTC')", "'2026-01-15 10:00:00.123456+05:00'")] + [InlineData(3, null, "DateTime64(3)", "'2026-01-15 10:00:00.123+05:00'")] + [InlineData(0, "UTC", "DateTime64(0, 'UTC')", "'2026-01-15 10:00:00+05:00'")] + public void Literal_carries_the_offset(int precision, string? timezone, string expectedStoreType, string expectedLiteral) + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision, timezone); + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)).AddTicks(1234567); + + Assert.Equal(expectedStoreType, mapping.StoreType); + Assert.Equal(expectedLiteral, mapping.GenerateSqlLiteral(value)); + } + + [Fact] + public void Seconds_precision_literal_has_no_fractional_digits() + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision: null, timezone: "UTC"); + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)); + + Assert.Equal("DateTime('UTC')", mapping.StoreType); + Assert.Equal("'2026-01-15 10:00:00+05:00'", mapping.GenerateSqlLiteral(value)); + } + + /// + /// A literal that carries its offset must land on the same instant whatever timezone the + /// target column declares. A bare wall clock does not, which is why the offset is emitted. + /// + [Theory] + [InlineData("DateTime64(6)")] + [InlineData("DateTime64(6, 'UTC')")] + [InlineData("DateTime64(6, 'Asia/Tokyo')")] + [InlineData("DateTime64(6, 'Fixed/UTC+05:30:00')")] + [InlineData("DateTime")] + [InlineData("DateTime('Asia/Tokyo')")] + public async Task Literal_is_instant_exact_for_any_column_timezone(string columnType) + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(6, "UTC"); + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)); + + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = + $"SELECT toUnixTimestamp64Micro(toDateTime64(CAST({mapping.GenerateSqlLiteral(value)} AS {columnType}), 6))"; + + Assert.Equal(ToMicros(value), Convert.ToInt64(await command.ExecuteScalarAsync())); + } + + // --- round trip --------------------------------------------------------- + + [Fact] + public async Task Insert_and_read_back_preserves_the_instant_at_tick_precision() + { + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)).AddTicks(1234567); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(new DateTimeOffsetEntity { Id = 1, Default = value, Nullable = value, Value = 1.5 }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 1); + + // The instant survives exactly; the offset becomes the column's, which is UTC. + Assert.Equal(value.ToUniversalTime(), row.Default); + Assert.Equal(TimeSpan.Zero, row.Default.Offset); + Assert.Equal(value.ToUniversalTime(), row.Nullable); + } + + /// + /// The pattern in issue #53 uses DateTimeOffset.MinValue/MaxValue as open-ended + /// range sentinels, so both ends of the CLR range must survive. Precision 7 keeps this working: + /// 100 ns units in an Int64 span about 29,000 years, which covers all of + /// . A higher precision would not — DateTime64(9) overflows + /// well before year 9999. + /// + [Fact] + public async Task Min_and_max_values_round_trip_and_work_as_range_sentinels() + { + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.Entities.AddRange( + new DateTimeOffsetEntity { Id = 40, Default = DateTimeOffset.MinValue, Value = 1 }, + new DateTimeOffsetEntity { Id = 41, Default = DateTimeOffset.MaxValue, Value = 2 }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var min = await readContext.Entities.SingleAsync(e => e.Id == 40); + var max = await readContext.Entities.SingleAsync(e => e.Id == 41); + + Assert.Equal(DateTimeOffset.MinValue, min.Default); + Assert.Equal(DateTimeOffset.MaxValue, max.Default); + + // Used as sentinels, they must bracket everything. + DateTimeOffset? startDate = null; + DateTimeOffset? endDate = null; + startDate ??= DateTimeOffset.MinValue; + endDate ??= DateTimeOffset.MaxValue; + + var total = await readContext.Entities + .Where(e => e.Id == 40 || e.Id == 41) + .Where(e => e.Default >= startDate) + .Where(e => e.Default <= endDate) + .CountAsync(); + + Assert.Equal(2, total); + } + + /// + /// The sentinel pattern above only works on a column whose timezone offset is zero. Both ends of + /// the range sit at the edge of 's range, and + /// the driver has to build a wall clock in the column's timezone to return one. Any non-zero + /// offset pushes one end outside , and the driver throws while doing so — + /// before the provider sees the value, so this cannot be reported any better from here. + /// + /// + /// This is not specific to a fixed offset; a named zone such as Asia/Tokyo behaves the + /// same way. Reading the low end of the range from a named zone is worse than an error: zones + /// carry a Local Mean Time offset for year 1 (+09:18:59 for Tokyo), so the value comes + /// back quietly shifted. + /// + [Fact] + public async Task Max_value_does_not_read_back_from_a_non_zero_offset_column() + { + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.TzEntities.Add(new DateTimeOffsetTzEntity + { + Id = 50, + Utc = DateTimeOffset.MaxValue, + Naive = DateTimeOffset.MaxValue, + Tokyo = DateTimeOffset.MaxValue, + Seconds = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero) + }); + + // The write itself is fine: an instant needs no wall clock. + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + await Assert.ThrowsAsync( + () => readContext.TzEntities.SingleAsync(e => e.Id == 50)); + + // Reading the same row through the zero-offset column alone is fine, which is why the + // sentinel pattern still works on the default store type. (These columns are precision 6, + // so the value truncates to microseconds; Min_and_max_values_round_trip_and_work_as_range + // _sentinels covers the exact round trip on a precision 7 column.) + using var projectingContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var utcOnly = await projectingContext.TzEntities + .Where(e => e.Id == 50) + .Select(e => e.Utc) + .SingleAsync(); + + Assert.Equal(DateTimeOffset.MaxValue.ToUnixTimeMilliseconds(), utcOnly.ToUnixTimeMilliseconds()); + } + + [Fact] + public async Task Null_round_trips() + { + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(new DateTimeOffsetEntity + { + Id = 2, + Default = new DateTimeOffset(2026, 2, 1, 0, 0, 0, TimeSpan.Zero), + Nullable = null, + Value = 1 + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 2); + + Assert.Null(row.Nullable); + } + + /// + /// Reading a column that declares a non-UTC timezone must still give the right instant. The + /// driver returns a wall clock in the column's timezone, so the mapping attaches that offset. + /// + [Fact] + public async Task Read_attaches_the_offset_of_the_declared_timezone() + { + var value = new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.TzEntities.Add(new DateTimeOffsetTzEntity + { + Id = 1, + Utc = value, + Naive = value, + Tokyo = value, + Seconds = value + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var row = await readContext.TzEntities.SingleAsync(e => e.Id == 1); + + Assert.Equal(value, row.Utc); + Assert.Equal(value, row.Naive); + Assert.Equal(value, row.Tokyo); + Assert.Equal(value, row.Seconds); + + // Same instant, rendered in the column's zone. + Assert.Equal(TimeSpan.Zero, row.Utc.Offset); + Assert.Equal(TimeSpan.FromHours(9), row.Tokyo.Offset); + } + + // --- queries ------------------------------------------------------------ + + /// Reproduces issue #53 directly: a range filter plus an aggregate. + [Fact] + public async Task Range_filter_with_parameters_and_aggregate() + { + using var seedContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + seedContext.Entities.AddRange( + new DateTimeOffsetEntity { Id = 10, Default = new DateTimeOffset(2026, 1, 15, 0, 0, 0, TimeSpan.Zero), Value = 1.5 }, + new DateTimeOffsetEntity { Id = 11, Default = new DateTimeOffset(2026, 2, 15, 0, 0, 0, TimeSpan.Zero), Value = 2.5 }, + new DateTimeOffsetEntity { Id = 12, Default = new DateTimeOffset(2026, 5, 15, 0, 0, 0, TimeSpan.Zero), Value = 4.0 }); + await seedContext.SaveChangesAsync(); + + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + DateTimeOffset? startDate = null; + DateTimeOffset? endDate = null; + startDate ??= new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero); + endDate ??= new DateTimeOffset(2026, 3, 1, 0, 0, 0, TimeSpan.Zero); + + var query = ctx.Entities + .Where(e => e.Id >= 10 && e.Id <= 12) + .Where(e => e.Default >= startDate) + .Where(e => e.Default < endDate); + + // The parameter must be declared as the column type, not String. + Assert.Contains("{startDate:DateTime64(7, 'UTC')}", query.ToQueryString()); + + Assert.Equal(4.0, await query.SumAsync(e => e.Value)); + } + + [Fact] + public async Task Filter_with_a_non_zero_offset_parameter_matches_the_same_instant() + { + using var seedContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + seedContext.Entities.Add(new DateTimeOffsetEntity + { + Id = 20, + Default = new DateTimeOffset(2026, 6, 15, 5, 0, 0, TimeSpan.Zero), + Value = 7.0 + }); + await seedContext.SaveChangesAsync(); + + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + // The same instant written with a +05:00 offset. + var equivalent = new DateTimeOffset(2026, 6, 15, 10, 0, 0, TimeSpan.FromHours(5)); + + var found = await ctx.Entities.SingleOrDefaultAsync(e => e.Id == 20 && e.Default == equivalent); + + Assert.NotNull(found); + } + + [Fact] + public async Task Ordering_and_projection_work() + { + using var seedContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + seedContext.Entities.AddRange( + new DateTimeOffsetEntity { Id = 30, Default = new DateTimeOffset(2026, 9, 3, 0, 0, 0, TimeSpan.Zero), Value = 1 }, + new DateTimeOffsetEntity { Id = 31, Default = new DateTimeOffset(2026, 9, 1, 0, 0, 0, TimeSpan.Zero), Value = 2 }); + await seedContext.SaveChangesAsync(); + + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var ordered = await ctx.Entities + .Where(e => e.Id == 30 || e.Id == 31) + .OrderBy(e => e.Default) + .Select(e => e.Default) + .ToListAsync(); + + Assert.Equal([ + new DateTimeOffset(2026, 9, 1, 0, 0, 0, TimeSpan.Zero), + new DateTimeOffset(2026, 9, 3, 0, 0, 0, TimeSpan.Zero) + ], ordered); + } + + // --- opt out of the new default ----------------------------------------- + + /// + /// HasConversion<string>() is the documented way to keep the old String shape. + /// It composes a converter, so the CLR type stays . + /// Note that writing any converted property is a separate defect (#54). + /// + [Fact] + public void HasConversion_string_keeps_the_old_string_shape() + { + using var ctx = new StringDateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(StringDateTimeOffsetEntity))! + .FindProperty(nameof(StringDateTimeOffsetEntity.ViaConversion))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.Equal("String", mapping.StoreType); + Assert.Equal(typeof(DateTimeOffset), mapping.ClrType); + Assert.NotNull(mapping.Converter); + } + + /// + /// By contrast, a bare HasColumnType("String") gives the plain string mapping with no + /// converter, so the CLR type does not agree with the property. This is pre-existing behaviour + /// for any CLR type pointed at an unrelated store type, and is why the README tells users to + /// use HasConversion<string>() instead. + /// + [Fact] + public void HasColumnType_string_alone_does_not_compose_a_converter() + { + using var ctx = new StringDateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(StringDateTimeOffsetEntity))! + .FindProperty(nameof(StringDateTimeOffsetEntity.ViaColumnType))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.Equal("String", mapping.StoreType); + Assert.Equal(typeof(string), mapping.ClrType); + Assert.Null(mapping.Converter); + } + + // --- daylight saving ---------------------------------------------------- + + /// + /// The hour that repeats when clocks go back is ambiguous, because the driver gives a wall + /// clock and drops the offset. For a zone whose standard offset is zero the instant is still + /// recoverable: the driver only returns when the true + /// offset is not zero, so the zero candidate can be discarded. + /// + [Fact] + public async Task Ambiguous_wall_clock_round_trips_in_a_zero_standard_offset_zone() + { + // On 2026-10-25 the UK goes back from +01:00 to +00:00 at 02:00 local. + // These two distinct instants share the wall clock 01:30 in Europe/London. + var duringDaylightSaving = new DateTimeOffset(2026, 10, 25, 0, 30, 0, TimeSpan.Zero); + var afterDaylightSaving = new DateTimeOffset(2026, 10, 25, 1, 30, 0, TimeSpan.Zero); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.DstEntities.AddRange( + new DateTimeOffsetDstEntity { Id = 1, London = duringDaylightSaving }, + new DateTimeOffsetDstEntity { Id = 2, London = afterDaylightSaving }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var first = await readContext.DstEntities.SingleAsync(e => e.Id == 1); + var second = await readContext.DstEntities.SingleAsync(e => e.Id == 2); + + // Both instants survive, and they stay distinct. + Assert.Equal(duringDaylightSaving, first.London.ToUniversalTime()); + Assert.Equal(afterDaylightSaving, second.London.ToUniversalTime()); + Assert.NotEqual(first.London.ToUniversalTime(), second.London.ToUniversalTime()); + + // The offsets show which side of the change each one is on. + Assert.Equal(TimeSpan.FromHours(1), first.London.Offset); + Assert.Equal(TimeSpan.Zero, second.London.Offset); + } + + [Fact] + public void Ambiguous_wall_clock_picks_the_non_zero_offset_when_standard_is_zero() + { + // 01:30 on 2026-10-25 is ambiguous in Europe/London: +01:00 or +00:00. + var wallClock = new DateTime(2026, 10, 25, 1, 30, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, "Europe/London"); + + // Kind=Unspecified means the driver found a non-zero offset, so it must be the daylight one. + Assert.Equal(TimeSpan.FromHours(1), result.Offset); + Assert.Equal(new DateTimeOffset(2026, 10, 25, 0, 30, 0, TimeSpan.Zero), result.ToUniversalTime()); + } + + /// + /// In a zone where both candidate offsets are non-zero, the offset the driver dropped cannot be + /// recovered. Picking standard time silently moved the instant and mapped two distinct instants + /// onto one, so the read now throws instead. Europe/Paris goes back from +02:00 to +01:00. + /// + [Fact] + public void Ambiguous_wall_clock_throws_when_both_offsets_are_non_zero() + { + // 02:30 on 2026-10-25 is ambiguous in Europe/Paris: +02:00 or +01:00. + var wallClock = new DateTime(2026, 10, 25, 2, 30, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, "Europe/Paris")); + + Assert.Contains("Europe/Paris", ex.Message); + Assert.Contains("ambiguous", ex.Message); + Assert.Contains("+02:00", ex.Message); + Assert.Contains("+01:00", ex.Message); + } + + /// + /// Before standard time a zone's offset is Local Mean Time, recorded to the second, which + /// TimeZoneInfo may round to whole minutes. Asia/Tokyo is +09:18:59. The instant would come back + /// shifted by up to a minute, so such a value is refused rather than read wrong. + /// + [Fact] + public void A_value_before_standard_time_throws_rather_than_drifting() + { + var wallClock = new DateTime(2, 1, 2, 9, 18, 59, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, "Asia/Tokyo")); + + Assert.Contains("Asia/Tokyo", ex.Message); + Assert.Contains("Local Mean Time", ex.Message); + } + + [Fact] + public void Unresolvable_declared_timezone_throws_rather_than_guessing() + { + var wallClock = new DateTime(2026, 1, 15, 5, 0, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, "Not/AZone")); + + Assert.Contains("Not/AZone", ex.Message); + } + + // --- fixed-offset timezones --------------------------------------------- + + /// + /// ClickHouse lets a column declare a fixed UTC offset, spelled Fixed/UTC±HH:MM:SS. + /// No such .NET timezone exists, so TimeZoneInfo.FindSystemTimeZoneById cannot resolve + /// the name however complete the host's timezone data is. Reading it must still work. + /// + [Fact] + public void A_fixed_offset_name_is_not_a_dotnet_timezone() + => Assert.Throws( + () => TimeZoneInfo.FindSystemTimeZoneById("Fixed/UTC+05:30:00")); + + [Theory] + [InlineData("Fixed/UTC+05:30:00", 5, 30)] + [InlineData("Fixed/UTC-07:00:00", -7, 0)] + [InlineData("Fixed/UTC+05:45:00", 5, 45)] + [InlineData("Fixed/UTC-09:30:00", -9, -30)] + [InlineData("Fixed/UTC+00:01:00", 0, 1)] + [InlineData("Fixed/UTC+14:00:00", 14, 0)] + [InlineData("Fixed/UTC-14:00:00", -14, 0)] + public void ConvertToDateTimeOffset_reads_a_fixed_offset_timezone( + string timezone, int expectedHours, int expectedMinutes) + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, timezone); + + var expectedOffset = new TimeSpan(expectedHours, expectedMinutes, 0); + Assert.Equal(expectedOffset, result.Offset); + // The wall clock is kept as given; only the offset is attached. + Assert.Equal(wallClock, result.DateTime); + Assert.Equal(wallClock - expectedOffset, result.UtcDateTime); + } + + /// + /// A fixed offset never changes, so the daylight-saving ambiguity that affects a named zone + /// cannot arise. The same wall clock therefore always gives the same instant. + /// + [Fact] + public void A_fixed_offset_is_never_ambiguous() + { + // In Europe/London this wall clock is the repeated hour when clocks go back. + var ambiguousElsewhere = new DateTime(2026, 10, 25, 1, 30, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset( + ambiguousElsewhere, "Fixed/UTC+01:00:00"); + + Assert.Equal(TimeSpan.FromHours(1), result.Offset); + Assert.Equal(new DateTimeOffset(2026, 10, 25, 0, 30, 0, TimeSpan.Zero), result.ToUniversalTime()); + } + + /// + /// ClickHouse accepts offsets that cannot hold: it caps the + /// magnitude at 14 hours and requires whole minutes. The instant is still exact, because the + /// offset is known and fixed, so the value is reported at offset zero rather than refused. + /// This mapping does not keep the offset in any case. + /// + /// The offset the timezone name spells, which the server applied + /// to produce the wall clock. + [Theory] + [InlineData("Fixed/UTC+15:00:00", 15 * 60)] + [InlineData("Fixed/UTC-15:00:00", -15 * 60)] + [InlineData("Fixed/UTC+24:00:00", 24 * 60)] + [InlineData("Fixed/UTC+05:30:30", 5 * 60 + 30)] + public void An_unrepresentable_fixed_offset_still_reports_the_exact_instant( + string timezone, int offsetFromName) + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + // Fixed/UTC+05:30:30 carries 30 seconds that the whole-minute InlineData cannot express. + var extraSeconds = timezone == "Fixed/UTC+05:30:30" ? 30 : 0; + var appliedOffset = TimeSpan.FromMinutes(offsetFromName) + TimeSpan.FromSeconds(extraSeconds); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, timezone); + + // The instant is recovered exactly: wall clock minus the offset the server applied. + Assert.Equal(wallClock - appliedOffset, result.UtcDateTime); + Assert.Equal(TimeSpan.Zero, result.Offset); + } + + /// + /// ClickHouse carries minutes and seconds above 59, but the driver does not read such a name and + /// returns a UTC wall clock instead of one in the column's timezone. Recovering the instant would + /// depend on that driver quirk, so this spelling is still refused. + /// + [Fact] + public void A_fixed_offset_the_driver_cannot_read_still_throws() + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, "Fixed/UTC+09:99:99")); + + Assert.Contains("Fixed/UTC+09:99:99", ex.Message); + Assert.Contains("does not support", ex.Message); + } + + /// + /// A name that only looks like a fixed offset is not one. ClickHouse rejects each of these, so + /// no such column can exist, and the unresolvable-timezone error is the right answer. + /// + [Theory] + [InlineData("Fixed/UTC+05:30")] + [InlineData("fixed/utc+05:30:00")] + [InlineData("Fixed/UTC05:30:00")] + [InlineData("Fixed/UTC+5:30:00")] + public void A_name_that_only_looks_like_a_fixed_offset_is_not_guessed_at(string timezone) + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, timezone)); + + Assert.Contains(timezone, ex.Message); + Assert.Contains("does not know the timezone", ex.Message); + } + + /// + /// ClickHouse does not hold the minutes and seconds fields to 59 — it carries the excess, so + /// Fixed/UTC+05:60:00 is a legal name for +06:00. The driver does not read those + /// names, and gives a UTC wall clock rather than one in the column's timezone, so attaching the + /// offset would move the instant by the whole offset and report nothing. Such a column must + /// report the driver's limit and the spelling to use instead. + /// + [Theory] + [InlineData("Fixed/UTC+05:60:00", "Fixed/UTC+06:00:00")] + [InlineData("Fixed/UTC+05:00:60", "Fixed/UTC+05:01:00")] + [InlineData("Fixed/UTC+00:99:00", "Fixed/UTC+01:39:00")] + [InlineData("Fixed/UTC-05:60:00", "Fixed/UTC-06:00:00")] + public void A_carried_fixed_offset_name_reports_the_driver_limit(string timezone, string suggested) + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, timezone)); + + Assert.Contains(timezone, ex.Message); + Assert.Contains("driver does not support", ex.Message); + // The canonical spelling of the same offset, which the driver does read. + Assert.Contains(suggested, ex.Message); + // It must not be reported as missing host timezone data, which was the old wrong answer. + Assert.DoesNotContain("tzdata", ex.Message); + } + + [Theory] + [InlineData(nameof(DateTimeOffsetFixedEntity.Half), "DateTime64(7, 'Fixed/UTC+05:30:00')", "Fixed/UTC+05:30:00")] + [InlineData(nameof(DateTimeOffsetFixedEntity.Negative), "DateTime64(7, 'Fixed/UTC-07:00:00')", "Fixed/UTC-07:00:00")] + [InlineData(nameof(DateTimeOffsetFixedEntity.Zero), "DateTime64(7, 'Fixed/UTC+00:00:00')", "Fixed/UTC+00:00:00")] + public void A_fixed_offset_store_type_resolves_and_keeps_its_timezone( + string propertyName, string expectedStoreType, string expectedTimezone) + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetFixedEntity))! + .FindProperty(propertyName)!; + + var mapping = property.GetRelationalTypeMapping(); + + var typed = Assert.IsType(mapping); + Assert.Equal(expectedStoreType, property.GetColumnType()); + Assert.Equal(expectedTimezone, typed.Timezone); + } + + /// + /// The end-to-end case from the review. Each column declares a different fixed offset, and + /// every instant must survive with the offset the column declares. + /// + [Fact] + public async Task Fixed_offset_columns_round_trip_the_instant_and_report_their_offset() + { + var value = new DateTimeOffset(2026, 3, 21, 14, 25, 36, TimeSpan.Zero).AddTicks(1234567); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.FixedEntities.Add(new DateTimeOffsetFixedEntity + { + Id = 1, + Half = value, + Negative = value, + Quarter = value, + Zero = value, + Seconds = value + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var row = await readContext.FixedEntities.SingleAsync(e => e.Id == 1); + + // The instant survives exactly at tick precision through every declared offset. + Assert.Equal(value, row.Half.ToUniversalTime()); + Assert.Equal(value, row.Negative.ToUniversalTime()); + Assert.Equal(value, row.Quarter.ToUniversalTime()); + Assert.Equal(value, row.Zero.ToUniversalTime()); + Assert.Equal(value, row.Seconds.ToUniversalTime()); + + // Each value is rendered at the offset its column declares. + Assert.Equal(new TimeSpan(5, 30, 0), row.Half.Offset); + Assert.Equal(new TimeSpan(-7, 0, 0), row.Negative.Offset); + Assert.Equal(new TimeSpan(5, 45, 0), row.Quarter.Offset); + Assert.Equal(TimeSpan.Zero, row.Zero.Offset); + Assert.Equal(new TimeSpan(0, 1, 0), row.Seconds.Offset); + } + + /// + /// The same case end to end. ClickHouse accepts Fixed/UTC+05:60:00 and creates the + /// column, so the write succeeds; the read must then say what is actually wrong. Reporting + /// missing host timezone data, as it did before, sends the user to install tzdata that + /// would never help. + /// + [Fact] + public async Task A_carried_fixed_offset_column_reports_the_driver_limit_on_read() + { + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.CarriedEntities.Add(new DateTimeOffsetCarriedEntity + { + Id = 1, + Carried = new DateTimeOffset(2026, 3, 21, 14, 25, 36, TimeSpan.Zero) + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var ex = await Assert.ThrowsAsync( + () => readContext.CarriedEntities.SingleAsync(e => e.Id == 1)); + + Assert.Contains("Fixed/UTC+05:60:00", ex.Message); + Assert.Contains("Fixed/UTC+06:00:00", ex.Message); + Assert.DoesNotContain("tzdata", ex.Message); + } + + /// A filter must still match on the instant, whatever offset the column declares. + [Fact] + public async Task Fixed_offset_column_filters_on_the_instant() + { + var value = new DateTimeOffset(2026, 4, 2, 8, 15, 0, TimeSpan.Zero); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.FixedEntities.Add(new DateTimeOffsetFixedEntity + { + Id = 2, + Half = value, + Negative = value, + Quarter = value, + Zero = value, + Seconds = value + }); + await writeContext.SaveChangesAsync(); + + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + // The same instant written at a third offset again. + var equivalent = new DateTimeOffset(2026, 4, 2, 13, 45, 0, TimeSpan.FromHours(5.5)); + + var found = await ctx.FixedEntities.SingleOrDefaultAsync(e => e.Id == 2 && e.Half == equivalent); + + Assert.NotNull(found); + Assert.Contains("Fixed/UTC+05:30:00", ctx.FixedEntities.Where(e => e.Half == equivalent).ToQueryString()); + } + + // --- conversion helper -------------------------------------------------- + + [Fact] + public void ConvertToDateTimeOffset_reads_utc_kind_as_offset_zero() + { + var value = new DateTime(2026, 1, 15, 5, 0, 0, DateTimeKind.Utc); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(value, "UTC"); + + Assert.Equal(new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero), result); + } + + [Fact] + public void ConvertToDateTimeOffset_reads_a_timezone_less_wall_clock_as_utc() + { + var value = new DateTime(2026, 1, 15, 5, 0, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(value, null); + + Assert.Equal(new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero), result); + } + + [Fact] + public void ConvertToDateTimeOffset_attaches_a_declared_zone_offset() + { + // A Tokyo wall clock of 14:00 is the instant 05:00Z. + var value = new DateTime(2026, 1, 15, 14, 0, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(value, "Asia/Tokyo"); + + Assert.Equal(TimeSpan.FromHours(9), result.Offset); + Assert.Equal(new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero), result.ToUniversalTime()); + } + + [Fact] + public void ConvertToDateTimeOffset_passes_through_a_datetimeoffset() + { + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(value, "UTC"); + + Assert.Equal(value, result); + } + + // --- write range --------------------------------------------------------- + + /// + /// A DateTime64(P) is an Int64 count of 10^-P seconds. A value that does not fit wraps rather + /// than reporting, so the row would read back with an unrelated date. Precision 7 spans about + /// 29 000 years and holds every DateTimeOffset; a finer precision does not. + /// + [Theory] + [InlineData(9)] + [InlineData(8)] + public void A_value_the_precision_cannot_hold_is_refused_rather_than_wrapped(int precision) + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision, "UTC"); + + var ex = Assert.Throws( + () => mapping.GenerateSqlLiteral(DateTimeOffset.MaxValue)); + + Assert.Contains($"DateTime64({precision}, 'UTC')", ex.Message); + Assert.Contains("wrap", ex.Message); + } + + /// The finer precisions stay usable for the range they can hold. + [Theory] + [InlineData(7)] + [InlineData(8)] + [InlineData(9)] + public void A_value_inside_the_precision_range_is_accepted(int precision) + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision, "UTC"); + + var literal = mapping.GenerateSqlLiteral(new DateTimeOffset(2026, 1, 15, 10, 30, 45, TimeSpan.Zero)); + + Assert.Contains("2026-01-15 10:30:45", literal); + } + + /// Precision 7 covers the whole DateTimeOffset range, which is why it is the default. + [Theory] + [InlineData(7)] + public void The_default_precision_holds_the_whole_datetimeoffset_range(int precision) + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision, "UTC"); + + Assert.NotNull(mapping.GenerateSqlLiteral(DateTimeOffset.MinValue)); + Assert.NotNull(mapping.GenerateSqlLiteral(DateTimeOffset.MaxValue)); + } + + /// + /// The bulk insert path gives the driver model values without consulting the type mapping, so it + /// asks the mapping to check the range itself. Without that, SaveChanges wrote a wrapped date. + /// + [Fact] + public async Task SaveChanges_refuses_a_value_the_column_precision_cannot_hold() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + ctx.PrecisionEntities.Add(new DateTimeOffsetPrecisionEntity + { + Id = 9001, + Millis = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.Zero), + Nanos = DateTimeOffset.MaxValue + }); + + var ex = await Assert.ThrowsAsync(() => ctx.SaveChangesAsync()); + + Assert.Contains("wrap", ex.Message); + } +}