Remove the search/extras table split by roed-math · Pull Request #59 · roed314/psycodict

roed-math · 2026-07-19T21:38:28Z

Stacked on #58 — this branch is based on the CI branch so the new test suite guards the refactor; only the top commit (e6db842) is this PR's change. It will rebase to a one-commit diff once #58 merges.

Implements the pre-1.0 removal of the extras-table mechanism, per the removal plan: a psycodict table is now always exactly one postgres table. Two facts made this an easy call: create_extra_table has been broken on postgres 10+ for years (single-column parenthesized SET syntax — the strict xfail in the CI suite documented it), and update() on extras columns was NotImplementedError.

API changes (backward-incompatible, hence pre-1.0)

create_table loses extra_columns; reload/copy_from/copy_to/copy_to_example lose extrafile; add_column loses extra; PostgresTable.__init__ loses has_extras.
Deleted outright: create_extra_table, move_column_to_extra, move_column_to_search, _move_column, _get_table_clause, the extra_table/extra_cols attributes, and the unused EmptyContext.
_parse_projection returns a single tuple (was a (search_cols, extra_cols) pair) and _search_iterator drops extra_cols — note: seminars calls both of these private APIs (seminars/utils.py, seminar.py, talk.py) and needs a small paired PR before its next deploy.
Projection codes 2 and 3 are kept as historical aliases (2 ≡ 1, 3 = 1 + id), since lucky/lookup default to projection=2 throughout downstream code.
has_extras leaves the meta_tables model. The exported <name>_meta.txt format shrinks from 11 to 10 required fields — old exports need field 6 (has_extras) deleted before reloading (e.g. awk -F'|' 'BEGIN{OFS="|"} {for(i=6;i<NF;i++) $i=$(i+1); NF--; print}'). Existing databases keep a harmless stray column; dropping it is optional tidiness.
Defensive checks: connecting to a database whose meta_tables still has has_extras rows set raises with a migration pointer, and reload_all raises on unexpected _extras.txt files.

Not included (coordinated separately)

Production migration: devmirror still shows three live extras tables (maass_newforms, maass_rigor, test_xyz) — being merged manually before deployment.
lmfdb PR (six lines reading .extra_cols: lmfdb_database.py, knowledge/main.py, api/api.py + apischema.html) and the seminars PR above — both to land lockstep with this merging, since both install psycodict from an unpinned git URL.

Verification

Full suite on the CI base: 463 passed / 18 skipped / 42 xfailed (9 extras tests removed including the retired xfail, 6 updated), ruff clean.
A 35-step workload regression (DDL, inserts, full query language, jsonb paths, arrays, buffered iteration, update/upsert/delete, stats, copy_to/copy_from/reload) against the base branch: byte-identical output — the removal changes no behavior for single-table use.

Net −479 library lines.

🤖 Generated with Claude Code

roed-math · 2026-07-20T03:29:09Z

Status update now that #58 has merged:

Rebased onto master — this is a clean single-commit PR again.
The production migration is done. devmirror now reports no has_extras rows and no _extras tables, and test_xyz is gone, so the startup check this PR adds has nothing to trip over.
Verified against the real mirror: the full test_devmirror.py suite (21 tests) passes against devmirror with this branch installed, on top of the 463-passing local suite and the byte-identical workload regression.
Downstream PRs are open: LMFDB/lmfdb#7070 and roed314/seminars#974. Neither blocks this one — the LMFDB change only removes reads of .extra_cols (empty everywhere today), and the seminars change is deliberately version-tolerant, so all three can merge in any order.

A psycodict table could be backed by two postgres tables: a search table with the searchable columns and an "extras" table with the rest, joined by id. The complexity is no longer worth it (a separate psycodict table can play the same role), the LMFDB is retiring its last extras tables, and create_extra_table has been broken on postgres 10+ for years (bad single-column parenthesized SET syntax), so before 1.0 is the time for this backward-incompatible removal. API changes: - create_table loses extra_columns; reload, copy_from, copy_to and copy_to_example lose extrafile; add_column loses extra; PostgresTable.__init__ loses has_extras - deleted: create_extra_table, move_column_to_extra, move_column_to_search, _move_column, _get_table_clause, the extra_table/extra_cols attributes, and the (unused) EmptyContext - _parse_projection returns a single tuple of columns instead of a (search_cols, extra_cols) pair, and _search_iterator loses its extra_cols argument (note: seminars uses both private APIs) - projection codes 2 and 3 are kept as historical aliases (2 == 1, 3 == 1 + id): lucky and lookup default to projection=2 downstream - has_extras is removed from the meta_tables model (metafiles shrink from 11 to 10 required columns; old exports need the has_extras field deleted before reloading) - connecting to a database whose meta_tables still has rows with has_extras set raises with a pointer to the migration - reload_all raises on unexpected _extras.txt files Simplified along the way: upsert, insert_many, delete, update, update_from_file, rewrite, resort, analyze, lucky, search, join_search (which also loses its broken never-taken extras-join branches), random_sample, create_table_like, drop_table, rename_table, table_sizes, reload/swap/revert/cleanup suffix loops, constraint and lock handling. Net -479 library lines. Tests: 9 extras tests removed (including the strict xfail documenting that create_extra_table was broken), 6 updated; the suite passes 463/18 skipped/42 xfailed, and a 35-step workload regression against the base branch shows no behavioral change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The last LMFDB tables using the search/extras split (maass_newforms, maass_rigor) have had their extras columns merged into the search table, and psycodict is removing support for the mechanism entirely before its 1.0 release (roed314/psycodict#59). - lmfdb_database.py: column_description and drop_table no longer append self.extra_cols (empty for every table now); create_extra_table and move_column leave _nolog_changetypes, since neither operation exists - knowledge/main.py: the missing-column-knowls report drops extra_cols - api/api.py: stop building extra_schema for the collection and data pages, and stop passing it to the templates - api/templates/apischema.html: drop the extra-columns rows This is compatible with both the current psycodict and the version that removes extras support, since extra_cols is [] for every table today. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

lmfdb main reads table.extra_cols (api/api.py) and seminars master selects the has_extras column in its own meta_tables query, so both downstream jobs fail against this branch until their paired PRs land: LMFDB/lmfdb#7070 and roed314/seminars#974, both verified against this psycodict (api tests and the full 24-check seminars run). The downstream jobs and the full-suite workflow's lmfdb ref are pinned to those PR heads, marked with TRANSITION comments. Revert this commit once #7070 and #974 are merged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

psycodict is removing the ability to split a table into a search table and an extras table before its 1.0 release (roed314/psycodict#59), which changes three private APIs this code calls: _parse_projection returns a single tuple of columns instead of a (search_cols, extra_cols) pair, _search_iterator loses its extra_cols argument, and _get_table_clause is gone (it only ever added the join to the extras table). The three helpers added to utils.py -- parse_projection, search_iterator and plain_iterator -- work against psycodict on either side of that change, so this can be deployed before or after psycodict updates. They branch on a single probe, _psycodict_has_extras; once the psycodict requirement is pinned past the removal, the probe and its branches can be deleted and the helpers inlined. - utils.py: search_distinct and lucky_distinct now work with one flat tuple of columns; lucky_distinct builds its own FROM clause (what _get_table_clause returned with no extras columns); sanitized_table selects has_extras from meta_tables only when psycodict still has it, since the row is splatted positionally into PostgresSearchTable - seminar.py, talk.py: the object_iterator wrappers take (cur, cols, projection), and the objects=False case returns plain_iterator(table) rather than the raw _search_iterator, so every iterator handed to search_distinct has the same signature The 'more' pseudo-column, which search_distinct used to smuggle in via extra_cols, is now appended to the flat column tuple instead. Verified by exercising the new helpers against psycodict both before and after the removal: the probe flips and every other result -- projections by int/list/str/dict, iterator output, the 'more' column, and building a table from a meta_tables row -- is identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

roed-math · 2026-07-20T18:59:28Z

Rebased onto current master (over the 25 merged fix PRs) and CI-readied:

Conflict resolutions fold master's fixes into the simplified post-extras code paths: create_constraint keeps the corrected "NOT NULL" spelling in Remove the search/extras table split #59's flattened validation, and insert_many keeps the copy/id-stamp/jsonb-NULL behaviour from count() returns a stale total for default-settings tables #61/insert_many corrupts the caller's dictionaries #64/jsonb None is stored as the value 'null', not SQL NULL #65 in the extras-free branch. Everything else auto-merged and is covered by the suite.
Zero xfail markers remain repo-wide — the last one (the create_extra_table test) is deleted along with the machinery it pinned. Suite: 515 passed plain / 516 under Sage, ruff clean, devmirror 21/21.
Downstream CI is pinned to the paired PRs for the transition (separate commit, revert it after they merge): lmfdb main reads table.extra_cols (verified: 2 api-test failures) and seminars master selects has_extras in its own meta query (verified: 7 check failures on a fresh bootstrap). Against the paired heads, both pass — Stop referring to psycodict extras tables LMFDB/lmfdb#7070: 6/6 api tests, Stop depending on psycodict's extras-table APIs seminars#974: 24/24 checks, both run locally against this branch.

Merge order: this PR and the two paired PRs can still land in any order; the pin only exists so this branch's own CI is meaningful before they do.

Reverts the transition pin: LMFDB/lmfdb#7070 and roed314/seminars#974 are both merged, so lmfdb main and seminars master now contain exactly the code the pinned refs pointed at, and the downstream jobs go back to testing reality. This reverts commit 5cdca76a. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

roed314 · 2026-07-20T19:25:09Z

-                raise ValueError("Column %s repeated" % (C.most_common(1)[0][0]))
        description_columns = []
-        for col in itertools.chain(search_columns.values(), [] if extra_columns is None else extra_columns.values()):
+        for col in itertools.chain(search_columns.values()):


itertools.chain is no longer needed.

Right — the single-argument chain was the vestige of chaining the search and extras column dictionaries. Now plain iteration, and the itertools import went with it.

roed314 · 2026-07-20T19:35:47Z

-    assert meta_tables_row(db, name)["has_extras"] is True
-
-
-def test_create_table_validates_columns_label_and_sort(db, empty_table):


Why was this test removed?

Collateral, not intent: this branch's original commit predates the current test suite and deleted this along with the genuinely extras-dependent tests, and the rebase carried the deletion forward. Restored verbatim from master — it touches nothing extras-related.

roed314 · 2026-07-20T19:36:08Z

-        db.create_table(fresh_name(), [("n", "no_such_type")], label_col="n")
-
-
-def test_create_table_force_description(db, transient):


Also here: why removed?

Same story as above — restored verbatim. Of the six tests this branch deleted, only these two were extras-free; the other four (extra_columns creation, add_column(extra=True), the mixed-constraint rejection, create_extra_table itself) all exercise the removed machinery and stay gone.

test_create_table_validates_columns_label_and_sort and test_create_table_force_description touch nothing extras-related; they were deleted by this branch's original commit, which predates the current suite, and the rebase carried the deletion forward. Restored verbatim from master. (The other removed tests all exercise the extras machinery itself and stay removed.) itertools.chain over a single iterable was the vestige of chaining the search and extras column dictionaries; plain iteration replaces it and the itertools import goes with it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This was referenced Jul 20, 2026

Stop referring to psycodict extras tables LMFDB/lmfdb#7070

Merged

Stop depending on psycodict's extras-table APIs roed314/seminars#974

Merged

roed-math force-pushed the remove-extras branch from e6db842 to 9674571 Compare July 20, 2026 03:27

This was referenced Jul 20, 2026

Fix the bugs found by the CI suite #60

Closed

create_extra_table has never worked #81

Closed

roed-math force-pushed the remove-extras branch from 9674571 to 5cdca76 Compare July 20, 2026 18:59

roed314 reviewed Jul 20, 2026

View reviewed changes

roed314 merged commit 9e65584 into roed314:master Jul 20, 2026
16 checks passed

This was referenced Jul 20, 2026

Port from psycopg2 to psycopg3 #88

Open

Update psycodict past the extras removal; import SQL from psycodict AndrewVSutherland/psetpartners#578

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove the search/extras table split#59

Remove the search/extras table split#59
roed314 merged 4 commits into
roed314:masterfrom
roed-math:remove-extras

roed-math commented Jul 19, 2026

Uh oh!

roed-math commented Jul 20, 2026

Uh oh!

roed-math commented Jul 20, 2026

Uh oh!

roed314 Jul 20, 2026

Uh oh!

roed-math Jul 20, 2026

Uh oh!

roed314 Jul 20, 2026

Uh oh!

roed-math Jul 20, 2026

Uh oh!

roed314 Jul 20, 2026

Uh oh!

roed-math Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		assert meta_tables_row(db, name)["has_extras"] is True


		def test_create_table_validates_columns_label_and_sort(db, empty_table):

		db.create_table(fresh_name(), [("n", "no_such_type")], label_col="n")


		def test_create_table_force_description(db, transient):

Conversation

roed-math commented Jul 19, 2026

API changes (backward-incompatible, hence pre-1.0)

Not included (coordinated separately)

Verification

Uh oh!

roed-math commented Jul 20, 2026

Uh oh!

roed-math commented Jul 20, 2026

Uh oh!

roed314 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

roed-math Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

roed314 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

roed-math Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

roed314 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

roed-math Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants