Sqlite types fix by AndreiKingsley · Pull Request #2001 · Kotlin/dataframe

AndreiKingsley · 2026-07-27T18:41:34Z

Fixes #1013.
Fixes #1935.
Fixes #1747.

Significantly improved different types support for SQLite DbType.

Add out-of-box support for Boolean and date-time types. They don't have a their own storage classes inside SQLite, but they have an official specification, so we can detect them from column metadata name and try to extract from possible SQLite storage classes (primitives like Int, Long, String, etc., see full list in code).
Add a special DSL for specifying any other custom SQLite type , by providing converter from storage classes to expected Kotlin types. Column KTypes are infered automatically!

…s and add tests

Copilot

Pull request overview

This PR improves SQLite type handling in the dataframe-jdbc module by introducing built-in conversions for SQLite’s “declared type vs storage class” mismatch (notably booleans and temporal types) and adding a DSL to register per-type / per-column custom converters, addressing issues like #1013/#1935.

Changes:

Added Sqlite.withCustomConverters { ... } DSL for mapping declared SQLite types / columns to Kotlin KTypes and optional value conversion.
Implemented SQLite-specific preprocessing to convert storage classes into idiomatic Kotlin types for BOOLEAN, DATE, TIME, DATETIME, and TIMESTAMP.
Expanded SQLite test coverage to include boolean and temporal column conversions, and updated docs/samples to the new custom-converter approach.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/schemas/DataSchemasTroubleshooting.kt	Updates troubleshooting sample to use the new SQLite custom converter DSL.
docs/StardustDocs/topics/schemas/Data-Schemas-And-Extension-Properties-Troubleshooting.md	Updates documentation section explaining SQLite type affinity and custom converter DSL usage.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/sqliteTest.kt	Adds regression tests for SQLite boolean + date/time/timestamp conversions.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/sqliteCustomTypesTest.kt	Updates and expands tests demonstrating custom converter DSL behavior and precedence rules.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/db/jdbcTypesTest.kt	Updates SQLite type tests to assert nullability behavior of identity `forType<T>(...)` mappings.
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt	Implements new converter DSL, custom mapping precedence, and storage-class→Kotlin conversions for SQLite.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

zaleslaw · 2026-07-29T11:10:55Z

@AndreiKingsley please fix Copilot and Jolan review comments
After that I'll do a last round

Jolanrensen · 2026-07-29T11:24:42Z

Seems to have a lot of duplication with AdvancedDbType. Maybe the necessary changes (like converters based on column name) could be added there instead. No need to reinvent the wheel :)

zaleslaw

Mostly agree with the main thing which @Jolanrensen is mentioned plus a couple of minor notes and well documented workarounds and problems

zaleslaw · 2026-08-04T11:21:15Z

+        // 3) BOOLEAN / BIT — SQLite has no boolean storage class. Xerial reports `Types.BOOLEAN`
+        //    metadata but `getObject` returns Integer (0/1). Convert explicitly.
+        when (tableColumnMetadata.jdbcType) {
+            Types.BOOLEAN, Types.BIT ->


This branch keys off jdbcType, but Xerial derives jdbcType from the stored storage class, not the declared type — the same reason you match dates by name just above. Verified empirically: a BOOLEAN column stored as TEXT 'true' reports getColumnType == VARCHAR(12), not BOOLEAN(16), so it never reaches this branch and is read as String; the String case in convertToBoolean is then unreachable. Suggest matching by declared name for consistency: "BOOL" in declaredUpper || "BIT" in declaredUpper.

zaleslaw · 2026-08-04T11:29:10Z

+ */
+public data class SqliteCustomTypeConverter<T, R>(
+    val expectedType: KType,
+    val resultingType: KType,


MAYBE: resultingType is used as-is (typeOf()), without withNullability(isNullable) — the test pins this as intentional. Worth a KDoc warning: forType("X") on a column that contains NULLs yields a non-null String schema type and relies on downstream Infer.Nulls.

We trust user specified type OVER jdbc column metadata nullability here.

zaleslaw · 2026-08-04T11:30:35Z

+    /**
+     * Xerial SQLite JDBC driver seems to give identical metadata for `Int?` and `Long?` columns
+     */
+    class DynamicTypes {


Good that forColumn<Long?>() is a manual escape hatch, but Int-vs-Long is still unresolved (Xerial reports identical metadata; the converter is generated once per column while getObject may return Long per row). A doc note that this is a workaround, not auto-detection, and that #1747 stays open would set expectations.

The only way to auto-detect this is reflection, which is too expensive, so we decided not to use it (globally across the Kotlin DataFrame).
This is a bug in the JDBC driver (which I believe will not be fixed), so for the time being, the official recommendation is to explicitly specify the column types.

@Jolanrensen do you agree here?

Jolanrensen · 2026-08-04T18:11:36Z

 *     Use this to override every column that shares a declared type.
 *  3. The built-in SQLite conversion for BOOLEAN, DATE, DATETIME, TIME, TIMESTAMP, DECIMAL,
 *     NUMERIC.
+ *     **This conversion is also applied for types that have substring of these type names**.


good note :) maybe add the example of "LONGVARCHAR" -> "LONG" to show how stupid this rule is and everyone takes extra care about this rule

Jolanrensen · 2026-08-04T18:24:03Z

 }

+@DataSchema
+interface FlagSQLite {


FlagSqlite seems better CamelCased. (though our own renameToCamelCase() makes it FlagSqLite XD)

Jolanrensen · 2026-08-04T18:31:53Z

+        //    when the stored value is a Julian day / Unix seconds).
+        when {
+            "DATETIME" in declaredUpper ->
+                return jdbcToDfConverterFor<Any?>(expectedKType).withPreprocessor {


I think putting all .withPreprocessor calls on a new line helps with readability

Jolanrensen · 2026-08-04T18:36:50Z

+     */
+    private fun fallbackConverter(meta: TableColumnMetadata): AnyJdbcToDataFrameConverter {
+        val expectedJdbcTypeBase = fallback.getExpectedJdbcType(meta)
+        val preprocessedValueType = fallback.getPreprocessedValueType(meta, expectedJdbcTypeBase)


using the preprocessedValueType from fallback only makes sense if you're also using fallback.preprocessedValue() as valuePreprocessor. Otherwise you can get type mismatches

AndreiKingsley added 4 commits July 23, 2026 15:36

Handle SQLite BOOLEAN columns correctly and add tests

bf8ce52

Support idiomatic Kotlin type conversions for SQLite date/time column…

71ee483

…s and add tests

Add custom type and column mappings for SQLite

e4365b2

formatting

84d45d7

AndreiKingsley requested review from Jolanrensen and zaleslaw July 27, 2026 18:41

zaleslaw requested a review from Copilot July 28, 2026 09:57

Copilot started reviewing on behalf of zaleslaw July 28, 2026 09:58 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

api dump

36e07e8

Jolanrensen reviewed Jul 28, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt Outdated

Jolanrensen reviewed Jul 28, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt Outdated

Jolanrensen reviewed Jul 28, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt Outdated

Jolanrensen reviewed Jul 28, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt

AndreiKingsley added 3 commits August 3, 2026 22:11

rewrote sqlite implementation with AdvancedDbType

94baf3d

apiDump

acdf75c

reduces fallbackConverter comment

775534d

AndreiKingsley requested a review from Jolanrensen August 3, 2026 18:19

Jolanrensen reviewed Aug 4, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt Outdated

Jolanrensen reviewed Aug 4, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt Outdated

Jolanrensen reviewed Aug 4, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt Outdated

Jolanrensen reviewed Aug 4, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt Outdated

Jolanrensen reviewed Aug 4, 2026

View reviewed changes

Comment thread dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt

zaleslaw requested changes Aug 4, 2026

View reviewed changes

Refactor SQLite converter methods for clarity and consistency

430c2d5

AndreiKingsley requested review from Jolanrensen and zaleslaw August 4, 2026 14:07

Jolanrensen reviewed Aug 4, 2026

View reviewed changes

Uh oh!

Conversation

AndreiKingsley commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zaleslaw commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jolanrensen commented Jul 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zaleslaw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AndreiKingsley commented Jul 27, 2026 •

edited

Loading

zaleslaw commented Jul 29, 2026 •

edited

Loading