Map DateTimeOffset to DateTime64 and fix composite component reads (#53)

CHANGELOG.md

@@ -3,8 +3,17 @@ v0.3.1 (Unreleased)
 ### Query translation
 * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval<unit>(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results.
 
+### Types
+* **`DateTimeOffset` is now a supported property type**, and maps to `DateTime64(7, 'UTC')`. The provider previously had no mapping for it, so EF Core fell back to `DateTimeOffsetToStringConverter` and made a `String` column with no warning — which broke queries against a real `DateTime64` column with `TYPE_MISMATCH`, and stopped `SaveChanges` from writing the value at all. The store type pins the timezone to `'UTC'` so the instant does not depend on the server's `session_timezone`, and precision 7 is one .NET tick (100 ns), so the round trip is exact. `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin. Note that ClickHouse has no type that stores a UTC offset: `DateTime64` holds an instant, and a declared timezone only decides how that instant is rendered. The instant is kept and the offset is not, so a value read back carries the offset of the column's declared timezone. A non-UTC column such as `DateTime64(6, 'Asia/Tokyo')` reads correctly, as does a fixed-offset column such as `DateTime64(7, 'Fixed/UTC+05:30:00')`, and `DateTimeOffset` composes into `Array`, `Map` and `Tuple` columns. Where an instant cannot be reproduced exactly the read throws rather than returning a value that is quietly wrong: the repeated hour when clocks go back in a zone whose candidate offsets are both non-zero (`Europe/Paris`), and a value before 1900 in a named zone, where the offset is Local Mean Time to the second. Writing a value the store type cannot hold throws for the same reason — ClickHouse wraps it rather than reporting it, which matters for precision 8 and 9, whose ranges are narrower than `DateTimeOffset`. **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')` — add `HasConversion<string>()` to keep the previous shape. See the [DateTimeOffset section of the README](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore#datetimeoffset) for the daylight-saving and `MinValue`/`MaxValue` limits. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53))
+
 ### Bug fixes
 * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported.
+* **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)` and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran, and any component whose CLR type differs from the driver's type threw `InvalidCastException`. This is not new with `DateTimeOffset` — `DateOnly[]`, `Dictionary<string, DateOnly>` and `Tuple<DateOnly, …>` were already affected, because `DateOnly` also arrives from the driver as a `DateTime`. The composite is now rebuilt component by component, with the same two steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. An `enum` component, a `List<T>` component and a nested composite therefore all read correctly, and a component that needs no conversion keeps the direct cast. **Known limit:** *writing* a component that needs a `ValueConverter` still does not work, because the bulk insert path passes model values to the driver without applying converters ([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)) — an `enum` inside a composite is written as its raw ordinal.
+* **`Array(Nullable(T))` DDL is no longer double-wrapped.** For a value-type element the store type came out as `Array(Nullable(Nullable(T)))`, which ClickHouse rejects with `Nested type Nullable(T) cannot be inside Nullable type`, so `EnsureCreated` and migrations both failed. The component mapping is resolved from a store type that already carries the wrapper, and `HasColumnType(...)` text is kept verbatim, so the nullable-element wrapper added a second one. It now adds the wrapper only when the inner store type does not already have one, including through `LowCardinality(Nullable(T))`. Reference-type elements were never affected.
+* **Composite component mappings now honour the CLR component type.** One ClickHouse store type can serve more than one CLR type: `DateTime64` serves both `DateTime` and `DateTimeOffset`, and `Date32` serves both `DateTime` and `DateOnly`. Resolving a component from an explicit store type such as `HasColumnType("Array(Nullable(DateTime64(7, 'UTC')))")` always picked the default CLR type, which gave the composite the wrong element type and broke change tracking. Array, Map and Tuple now pass the component CLR type from the model when they have one.
+* **Composite components now resolve to the CLR type the model declares.** Three separate causes gave a composite the wrong element type, and the query then failed to compile with a coercion error naming a type the user never wrote. A converter on the component erased the `Nullable<>` — EF Core takes a mapping's CLR type from its converter's model type — so `Colour?[]` over `Array(Nullable(Enum8(...)))` resolved as `Colour[]`. A component resolved from an explicit store type kept that store type's default CLR type, so `DateTime[]` over `Array(Date32)` resolved as `DateOnly[]`. And the store-type parsers disagreed about whitespace, so `Array( Nullable ( Date32 ) )` — which ClickHouse accepts and normalizes — lost its nullable element. The parsers are now one shared helper.
+* **Reading a `Tuple(...)` column no longer allocates per row.** The converter array was built inside the materializer, so every row allocated a fresh array of the same delegates, including rows the fast path returned untouched. It is now built once. `Array` and `Map` were never affected.
+* **A nested composite whose innermost component converts is no longer returned untouched.** The pass-through check looked only at the immediate component mapping, and a composite component carries no converter of its own, so an `Array(Array(T))` could skip the conversion of its innermost values. The check now recurses.
 * `Sum`/`SumAsync` over a `double` or `float` column no longer throws `InvalidCastException`. EF Core wraps a top-level aggregate so the empty case returns `0`, supplying that fallback as a boxed `Int32` carrying the `Float64`/`Float32` mapping; the literal generators now convert rather than unbox. The `Float32` read path also converts, since ClickHouse widens `sum(Float32)` to `Float64` (which the driver's `GetFloat()` refuses to downcast). ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46))
 * **SummingMergeTree with multiple sum columns**: `HasSummingMergeTreeEngine("A", "B")` now generates valid DDL (`SummingMergeTree((A, B))`). Previously it emitted a comma-separated argument list (`SummingMergeTree(A, B)`), which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH`. Single-column and no-column usage are unaffected.
 

README.md

@@ -67,7 +67,7 @@ public class PageView
 | **Bool** | `Bool` | `bool` |
 | **Strings** | `String`, `FixedString(N)` | `string` |
 | **Enums** | `Enum8(...)`, `Enum16(...)` | `string` or C# `enum` |
-| **Date/time** | `Date`, `Date32`, `DateTime`, `DateTime64(P, 'TZ')` | `DateOnly`, `DateTime` |
+| **Date/time** | `Date`, `Date32`, `DateTime`, `DateTime64(P, 'TZ')` | `DateOnly`, `DateTime`, `DateTimeOffset` (see [below](#datetimeoffset)) |
 | **Time** | `Time`, `Time64(N)` | `TimeSpan` |
 | **UUID** | `UUID` | `Guid` |
 | **Network** | `IPv4`, `IPv6` | `IPAddress` |
@@ -80,6 +80,105 @@ public class PageView
 | **Geographic** | `Point`, `Ring`, `LineString`, `Polygon`, `MultiLineString`, `MultiPolygon`, `Geometry` | `Tuple<double,double>` and arrays thereof; `object` for Geometry |
 | **Wrappers** | `Nullable(T)`, `LowCardinality(T)` | Unwrapped automatically |
 
+### DateTimeOffset
+
+A `DateTimeOffset` property maps to `DateTime64(7, 'UTC')` by default:
+
+```csharp
+public class Reading
+{
+    public long Id { get; set; }
+    public DateTimeOffset RecordedAt { get; set; }   // DateTime64(7, 'UTC')
+}
+```
+
+Three things to know:
+
+**The offset is not kept.** ClickHouse has no type that stores a UTC offset. `DateTime64` holds an
+instant, and a declared timezone only decides how that instant is rendered. A value written with
+any offset is stored as the correct instant, and a value read back carries the offset of the
+column's timezone — `+00:00` for the default store type. With that default store type, comparisons
+and ordering are instant-correct, so these two values match the same row:
+
+```csharp
+// The same instant, written two ways.
+var a = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5));
+var b = new DateTimeOffset(2026, 1, 15,  5, 0, 0, TimeSpan.Zero);
+```
+
+If you must keep the offset, store it yourself in a second column alongside a `DateTime`. To keep
+the whole value as text, ask for the conversion explicitly with `HasConversion<string>()` — note
+that `HasColumnType("String")` on its own is not enough, because it adds no converter. Such a
+column is read-only for now: `SaveChanges` cannot write any property that has a value converter
+([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)).
+
+**Precision 7 makes the round trip exact.** One .NET tick is 100 ns, which is precision 7, so a
+stored value never comes back truncated. Precision 7 also covers the full `DateTimeOffset` range,
+which lets you use `DateTimeOffset.MinValue` and `MaxValue` as open-ended range limits on the
+default store type. Keep those two sentinels to a column whose timezone offset is zero, such as the
+default `'UTC'`. Both sit at the edge of the `DateTime` range, and the driver has to build a wall
+clock in the column's timezone to return a value, so any non-zero offset pushes one end outside
+`DateTime`: reading `MaxValue` from a `DateTime64(7, 'Asia/Tokyo')` column throws. Choose a smaller
+precision if you prefer, but be aware that it discards the digits below it:
+
+```csharp
+b.Property(e => e.RecordedAt).HasColumnType("DateTime64(3, 'UTC')");   // milliseconds
+b.Property(e => e.RecordedAt).HasPrecision(3);                         // the same thing
+```
+
+Precision 8 and 9 are accepted, but they hold a narrower range of dates. ClickHouse stores a
+`DateTime64(P)` as an `Int64` count of 10^-P seconds, so precision 9 reaches only 1678 to 2262 and
+precision 8 about 1970 ± 2900 years. A value outside the range wraps rather than reporting, so the
+provider checks it and throws on write instead. Precision 7 has no such limit — it spans roughly
+29 000 years, which is why it is the default. Note also that .NET cannot represent more than 7
+fractional digits, so the extra precision stores no extra detail.
+
+**Keep `'UTC'` in the store type** unless you have a reason to change it. For a timezone-less type
+such as `DateTime64(7)`, the server reads the query parameter in its `session_timezone`, which moves
+the instant when that setting is not UTC.
+
+A column that declares a different timezone, for example `DateTime64(6, 'Asia/Tokyo')`, is read
+correctly and returns that zone's offset. Two limits apply to such a column:
+
+- The host operating system must know the timezone, or the read throws. Minimal Linux images may
+  need the `tzdata` package.
+- In a zone with daylight saving, the repeated hour when clocks go back is ambiguous, because the
+  driver gives a wall clock and drops the offset. The provider recovers the instant where the zone's
+  standard offset is zero, such as `Europe/London`. Where both candidate offsets are non-zero, such
+  as `Europe/Paris`, the instant cannot be recovered and the read throws — reporting one of the two
+  would move the instant and give the same result for two different ones.
+- A value before 1900 in a named zone throws. Before standard time a zone's offset is Local Mean
+  Time, which IANA records to the second (`+09:18:59` for `Asia/Tokyo`) while `TimeZoneInfo` may
+  round it to the minute, so the instant cannot be reproduced exactly. Store such a value in a
+  `'UTC'` column, which has no such offset.
+- Dates at the far ends of the `DateTimeOffset` range do not survive, as described above.
+
+A column can also declare a fixed UTC offset instead of a named zone. ClickHouse spells this
+`Fixed/UTC±HH:MM:SS`, with two digits in every field — the server rejects `Fixed/UTC+5:30:00` and
+`Fixed/UTC+05:30`:
+
+```csharp
+b.Property(e => e.RecordedAt).HasColumnType("DateTime64(7, 'Fixed/UTC+05:30:00')");
+```
+
+None of the limits above applies here: the host needs no timezone data, a fixed offset is never
+ambiguous, and it does not change before 1900. Two points of its own do:
+
+- `DateTimeOffset` holds an offset only within plus or minus 14 hours, and only in whole minutes,
+  while ClickHouse accepts more. Such a column still reads correctly — the instant is exact, because
+  the offset is known — but the value comes back at offset `+00:00` rather than the column's offset.
+  This mapping does not keep the offset in any case.
+- ClickHouse does not hold the minutes and seconds fields to 59 — it carries the excess, so
+  `Fixed/UTC+05:60:00` is a legal name for `+06:00`. The driver does not read such a name, so the
+  read throws rather than depend on that. Declare the offset as `Fixed/UTC+06:00:00` instead.
+
+`DateTimeOffset` also composes into the collection types, so `DateTimeOffset[]`,
+`List<DateTimeOffset>`, `Dictionary<string, DateTimeOffset>` and `Tuple<DateTimeOffset, …>` all
+round trip.
+
+`DateTimeOffset` members such as `.Year` and `.UtcDateTime` do not translate to SQL yet. This
+applies to `DateTime` as well — see [#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55).
+
 ## Current Status
 
 This provider is in active development. It supports **LINQ queries**, **inserts**, **table engine configuration**, and **migrations** — you can define ClickHouse tables with engine-specific settings, create them via `dotnet ef migrations` or `EnsureCreated`, query with LINQ, and write data via `SaveChanges`.
@@ -272,7 +371,7 @@ Configure ClickHouse table engines, ordering, partitioning, and more via EF Core
 ```csharp
 modelBuilder.Entity<SensorReading>(b =>
 {
-    b.HasKey(e => e.Id);
+    b.HasKey(e => e.Id); // becomes ORDER BY (the ClickHouse primary key) when no explicit ORDER BY is set
     b.Property(e => e.Temperature).HasCodec("Delta, ZSTD");
     b.Property(e => e.Location).HasColumnComment("Installation site");
     b.HasIndex(e => e.Timestamp)
@@ -298,6 +397,8 @@ modelBuilder.Entity<SensorReading>(b =>
 
 **Default behavior:** If no engine is configured, the provider defaults to `MergeTree` with the EF primary key as `ORDER BY`.
 
+**Primary key vs sorting key:** In ClickHouse the `ORDER BY` (sorting key) *is* the primary key, so `HasKey` alone is sufficient — it becomes `ORDER BY`. Only use `.WithPrimaryKey(...)` when you need the primary index to differ from the sort order (e.g. a `SummingMergeTree`/`AggregatingMergeTree` rollup with a long `ORDER BY` but a narrow index). ClickHouse requires the primary key to be a prefix of the `ORDER BY` columns.
+
 ### Migrations
 
 The provider supports `dotnet ef migrations` for creating and applying migrations: