Fix lexer edge cases and string handling by JanJakes · Pull Request #505 · WordPress/sqlite-database-integration

JanJakes · 2026-09-08T07:51:38Z

Summary

This PR fixes lexer edge cases and related string handling:

String values: Preserve binary and single-byte charset payloads and correctly decode escaped newlines.
Function names: Exclude trailing whitespace from function-name token ranges under IGNORE_SPACE.
Version comments: Support six-digit versions from MySQL 8.1, preserve five-digit fallback parsing and unversioned numeric content, and recognize vertical tabs as whitespace.
WordPress validation: Mirror the parent wpdb::query() charset checks, retaining its binary-data and prevalidated-query exemptions.
LIKE BINARY patterns: Preserve raw bytes instead of producing NULL during GLOB conversion, and handle escaped newlines.

The lexer fixes cover both PHP packages and the native extension.

Why

The token decoder's UTF-8 regex modifier caused an accidental TypeError for legitimate binary data as well as malformed text. Decoding must preserve bytes; charset enforcement needs column types and SQL mode. Mirroring WordPress's checks restores its expected validation behavior.

Regression tests cover decoding, token ranges, version comments, and pattern conversion. CI expectations remove the test_strip_invalid_text cases fixed by decoding and the two invalid-query tests fixed by WordPress validation.

Remaining limitations

Direct MySQL-on-SQLite writes can still accept invalid text under strict mode; full charset enforcement remains driver work. WordPress's checks retain their exemptions, including tables containing binary columns.
SQLite GLOB can still treat distinct invalid UTF-8 sequences as equal. The LIKE BINARY fix preserves pattern bytes but does not provide fully byte-correct matching.
Legacy multibyte charsets such as Big5 and GBK remain unsupported by the decoder.

coderabbitai · 2026-09-08T07:52:02Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a4d89563-b551-4294-b3a4-b3460c8a5eab

📥 Commits

Reviewing files that changed from the base of the PR and between 4ade8df and 8cd2a07.

📒 Files selected for processing (11)

.github/workflows/wp-tests-phpunit-run.js
packages/mysql-on-sqlite/src/mysql/class-wp-mysql-lexer.php
packages/mysql-on-sqlite/src/mysql/class-wp-mysql-token.php
packages/mysql-on-sqlite/src/sqlite/class-wp-mysql-on-sqlite.php
packages/mysql-on-sqlite/src/sqlite/class-wp-sqlite-pdo-user-defined-functions.php
packages/mysql-on-sqlite/tests/WP_MySQL_On_SQLite_Tests.php
packages/mysql-parser/src/class-wp-mysql-lexer.php
packages/mysql-parser/src/class-wp-mysql-token.php
packages/mysql-parser/tests/WP_MySQL_Server_Suite_Parser_Tests.php
packages/php-ext-wp-mysql-parser/src/lib.rs
packages/plugin-sqlite-database-integration/wp-includes/sqlite/class-wp-sqlite-db.php

💤 Files with no reviewable changes (1)

.github/workflows/wp-tests-phpunit-run.js

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

📝 Walkthrough

Walkthrough

The change updates MySQL token handling, lexer parsing, LIKE pattern translation, and database query validation. It adds coverage for binary data, escaped newlines, vertical tabs, versioned comments, token ranges, LIKE patterns, and invalid query text.

Changes

MySQL compatibility updates

Layer / File(s)	Summary
Binary-safe token and LIKE values `packages/mysql-parser/src/class-wp-mysql-token.php`, `packages/mysql-on-sqlite/src/mysql/`, `packages/mysql-on-sqlite/src/sqlite/`, `packages/mysql-parser/tests/WP_MySQL_Token_Tests.php`, `packages/mysql-on-sqlite/tests/*`	Token and LIKE escape handling preserves non-UTF-8 bytes and unescapes escaped newlines. Tests cover raw bytes, multibyte values, wildcards, and GLOB metacharacters.
Lexer whitespace and comment parsing `packages/mysql-parser/src/class-wp-mysql-lexer.php`, `packages/mysql-on-sqlite/src/mysql/class-wp-mysql-lexer.php`, `packages/php-ext-wp-mysql-parser/src/`, `packages/mysql-parser/tests/`, `packages/mysql-on-sqlite/tests/mysql/*`, `.github/workflows/wp-tests-phpunit-run.js`	The lexers recognize vertical tabs, support six-digit MySQL version comments, and keep ignored whitespace out of function token ranges. Tests cover version boundaries, whitespace variants, token offsets, corpus configuration, and PHPUnit expectations.
Invalid query text validation `packages/plugin-sqlite-database-integration/wp-includes/sqlite/class-wp-sqlite-db.php`	`WP_SQLite_DB::query()` validates marked queries after charset discovery, records rejected query details, resets validation state, and updates query-count tracking.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~30 minutes

Merge Risk: ⚪ Minimal · up to 8cd2a

This updates MySQL-compatible lexing and binary string handling, fixes token ranges and version-comment parsing, and adds invalid-query validation. The supplied coverage and current implementation summaries show no remaining merge-blocking risk.

Sequence Diagram(s)

sequenceDiagram
  participant SQLInput as SQL input
  participant WP_MySQL_Lexer
  participant WP_MySQL_Token
  SQLInput->>WP_MySQL_Lexer: tokenize comments, whitespace, and function syntax
  WP_MySQL_Lexer->>WP_MySQL_Token: emit token bytes and ranges
  WP_MySQL_Token->>WP_MySQL_Token: unescape token value

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 36 functions across 15 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: fixes for lexer edge cases and string handling. It is concise and related to the pull request objectives.

Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch lexer-improvements

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-08T07:52:27Z

🤖 Lexer benchmark

Changes to lexer-related files were detected and triggered a benchmark:

Config	Base (QPS)	This PR (QPS)	Speedup
no JIT	70,197	70,109	1.00×
tracing JIT	158,144	157,337	0.99×

Note: Hosted runners are noisy, and absolute numbers vary. Treat the results with caution and verify them locally.

To reproduce locally:

cd packages/mysql-on-sqlite && composer run bench-lexer

mho22

Looks good to me!

mho22 · 2026-09-09T06:25:33Z

 			}
 			++$total;
-			$tokens = ( new WP_MySQL_Lexer( $query ) )->remaining_tokens();
+			$tokens = ( new WP_MySQL_Lexer( $query, 80038 ) )->remaining_tokens();


The only thing I found worth mentionning, that could maybe be removed. Since WP_MySQL_Lexer seems to default to 80038 when it is not defined.

It does currently default to 80038, but this test checks an exact set of expected failures that depends on the MySQL version. Passing it explicitly keeps the test stable if the lexer’s default changes. I’ve added a comment explaining why it is pinned.

The final backslash-stripping step used preg_replace() with the "u" (UTF-8) modifier. Invalid UTF-8 made it return null, causing a TypeError because get_value() must return a string. MySQL literals can legitimately contain non-UTF-8 bytes in binary or single-byte charset payloads. Switch the modifier to "s" (DOTALL). Byte-wise unescaping preserves valid UTF-8 and raw bytes, and also handles a backslash followed by a newline. Legacy multibyte charsets such as Big5 and GBK remain unsupported. Remove 14 obsolete WordPress test failure expectations. Reclassify Tests_DB_Charset::test_invalid_characters_in_query as an expected assertion failure: invalid text is accepted instead of crashing at this step. Charset enforcement remains separate from token decoding.

When resolving a function keyword (SYM_FN), the lexer peeks for a following "(" and, under SQL_MODE_IGNORE_SPACE, skips intervening whitespace first. It skipped by advancing bytes_already_read and never restored it. When no "(" followed, the keyword was emitted as an IDENTIFIER whose length — derived from bytes_already_read in produce() — now covered the trailing whitespace, so the extracted value was e.g. "COUNT " instead of "COUNT". Under this ANSI-style mode a column or table named after a function would resolve to the wrong identifier. Peek with a local index instead of mutating bytes_already_read, so the token's byte range ends at the keyword and the next scan consumes the whitespace.

Recognize six-digit version numbers followed by whitespace starting with MySQL 8.1. Preserve five-digit parsing for older server versions and when the sixth digit is not followed by whitespace. Keep fewer than five leading digits as SQL content. Accept all MySQL ASCII whitespace, including vertical tabs, in the PHP and native lexers. Test version boundaries, whitespace, fallback parsing, and unversioned numeric content. Keep the corpus expectations pinned to MySQL 8.0.38. https://dev.mysql.com/blog-archive/are-you-ready-for-mysql-10/ https://github.com/mysql/mysql-server/blob/mysql-8.1.0/sql/sql_lex.cc

Use wpdb's charset checks to reject invalid text before executing a query, while preserving binary data and prevalidated queries. Keep Query Monitor entries aligned when validation performs charset lookups. Remove the two obsolete failure expectations for the existing WordPress invalid-text tests. Document that full charset enforcement belongs in MySQL on SQLite so it protects all callers, including direct driver use.

Remove UTF-8 validation from LIKE-to-GLOB pattern conversion so raw bytes do not turn the pattern into NULL. Use DOTALL to also unescape backslashes followed by newlines. Cover byte preservation, malformed UTF-8, Unicode, wildcards, escapes, and NULL in helper and query tests. Document the remaining GLOB limitation: distinct invalid UTF-8 sequences can match as equal.

JanJakes force-pushed the lexer-improvements branch 2 times, most recently from f695a70 to 4ade8df Compare September 8, 2026 10:13

JanJakes marked this pull request as ready for review September 8, 2026 10:23

JanJakes requested a review from mho22 September 8, 2026 11:40

mho22 approved these changes Sep 9, 2026

View reviewed changes

JanJakes added 5 commits September 9, 2026 11:51

JanJakes force-pushed the lexer-improvements branch from 4ade8df to 8cd2a07 Compare September 9, 2026 09:51

JanJakes changed the title ~~Fix lexer string values, token ranges, and version comments~~ Fix lexer edge cases, WordPress validation, and LIKE BINARY Sep 9, 2026

JanJakes changed the title ~~Fix lexer edge cases, WordPress validation, and LIKE BINARY~~ Fix lexer edge cases and string handling Sep 9, 2026

Navigation Menu

Uh oh!

Fix lexer edge cases and string handling - #505

Fix lexer edge cases and string handling#505
JanJakes wants to merge 5 commits into
trunkfrom
lexer-improvements

JanJakes commented Sep 8, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 8, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Sep 8, 2026 •

edited

Loading

Uh oh!

mho22 left a comment

Uh oh!

mho22 Sep 9, 2026

Uh oh!

JanJakes Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

JanJakes commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Remaining limitations

Uh oh!

coderabbitai Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Lexer benchmark

Uh oh!

mho22 left a comment

Choose a reason for hiding this comment

Uh oh!

mho22 Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

JanJakes Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JanJakes commented Sep 8, 2026 •

edited

Loading

coderabbitai Bot commented Sep 8, 2026 •

edited

Loading

github-actions Bot commented Sep 8, 2026 •

edited

Loading