Skip to content

perf: avoid full snapshot mapping in _resolve_table/_resolve_tables - #6068

Draft
mday-io wants to merge 5 commits into
SQLMesh:mainfrom
mday-io:fix/6017-resolve-table-mapping
Draft

perf: avoid full snapshot mapping in _resolve_table/_resolve_tables#6068
mday-io wants to merge 5 commits into
SQLMesh:mainfrom
mday-io:fix/6017-resolve-table-mapping

Conversation

@mday-io

@mday-io mday-io commented Sep 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Resolving one table during virtual-layer updates previously built and normalized a mapping for every snapshot. _resolve_table now maps only the matching snapshot on an exact lookup, retaining the full-mapping fallback for dialect-dependent name mismatches. _resolve_tables skips snapshot scans and expansion setup when an expression contains no table reference.

Explicit table_mapping entries retain their dialect-aware matching and precedence, including when an exact snapshot match exists or a later equivalent override follows an exact-key override. The fast path avoids scanning unrelated snapshots; processing explicit overrides remains proportional to the size of that mapping.

Fixes #6017.

Test Plan

  • Regression coverage for exact snapshot matches with equivalent override keys, including later-override precedence through a rendered macro.
  • Existing coverage checks snapshot scan counts, cross-dialect fallbacks, mapping without snapshots, expression inputs, expansion, table-less properties, and deployability.
  • make style: passed (Ruff, formatting, mypy, migration validation).
  • make fast-test: passed — 2,636 passed / 4 skipped in the main phase; isolated phases passed (5, 1, and 161 tests).
  • GitHub CI: DCO/Commits Check passed on e3cf6b7c; remaining jobs are running.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

mday-io and others added 3 commits September 11, 2026 10:50
@mday-io @claude
perf: avoid full snapshot mapping in _resolve_table/_resolve_tables
3cc4daa 
_resolve_table always merged the environment-wide snapshot->table-name
mapping and handed it to exp.replace_tables, which re-normalizes
(parses) every mapping key on every call, even to resolve a single
table. _resolve_tables did the same for property expressions
(virtual_properties, session_properties) that contain no table
reference at all. For an environment with N promoted views, this made
"Updating virtual layer" O(N^2) in pure Python.

_resolve_table now looks up only the one relevant snapshot/table_mapping
entry instead of building the full mapping (table_name arrives already
normalized to the same key format snapshots/table_mapping use, via
d.normalize_model_name at both call sites). _resolve_tables now skips
building the mapping and calling replace_tables entirely when the
expression has no exp.Table node to replace.

Fixes SQLMesh#6017 (one of three sub-issues split out of SQLMesh#6014).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyFdP5xRu9D368mjGcYDLn
Signed-off-by: mday-io <mdaytn@gmail.com>
@mday-io @claude
fix: preserve cross-dialect resolution in narrowed _resolve_table lookup
c0eecff 
The narrowed single-snapshot lookup added in the previous commit did a raw
snapshots.get(table_name) dict lookup. table_name is normalized under the
referencing renderer's own dialect, while a snapshots dict key is each
model's fqn, normalized under that model's own dialect. These can disagree
in casing when models use different dialects (e.g. a case-uppercasing
dialect like snowflake referenced from a case-insensitive one like duckdb),
causing the lookup to silently miss an existing snapshot and leave the
table name unmapped, even though the old full-mapping + exp.replace_tables
path (which reconciles casing per-dialect during matching) would have
resolved it correctly.

_resolve_table now falls back to building the full mapping only when the
narrowed lookup misses and the name isn't in table_mapping either, so the
common same-dialect case stays O(1) while the rare cross-dialect miss still
gets exp.replace_tables' dialect-aware reconciliation.

Also adds tests for: the cross-dialect regression itself, table_mapping-only
resolution with no snapshots, the non-string exp.Expr branch (otherwise
unreachable from any real call site), expand-then-find-Table ordering in
_resolve_tables, a table reference appearing only inside a string literal,
and deployability_index handling through the narrowed path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DzEtt434q32KhtoGDtD424
Signed-off-by: Michael Day <mdaytn@gmail.com>
@mday-io
Merge branch 'main' into fix/6017-resolve-table-mapping
a5ee072
@mday-io
mday-io force-pushed the fix/6017-resolve-table-mapping branch from ee4b41e to a5ee072 Compare September 15, 2026 00:15
@mday-io
fix: two more gaps in the narrowed resolve_table/resolve_tables paths
5dbdf2f 
An independent review of the previous two commits found two more correctness/
performance gaps in the same narrowed-lookup change:

1. _resolve_table's dialect-reconciling fallback only triggered when `snapshots`
   was non-empty (`if snapshot is None and snapshots and table_name not in
   table_mapping`). When `snapshots` is None/empty - e.g. sqlmesh test's
   render_query_or_raise(table_mapping=...) call, which passes a table_mapping
   built under the project's dialect with no snapshots at all - a table_mapping
   key differing only in casing/quoting from the resolved name silently missed,
   instead of falling back to exp.replace_tables' own normalization like the
   pre-existing snapshots case does. Fixed by dropping the `and snapshots`
   condition so the fallback covers a miss in either dict.

2. _resolve_tables' "skip the mapping build when there's no table to replace"
   check ran after building the `expand` set and `model_mapping`, both of which
   are themselves O(N) in the number of snapshots (the `expand` set comprehension
   scans every snapshot's `is_embedded` flag unconditionally). So the claimed
   O(1)/no-op behavior for table-less expressions (virtual_properties,
   session_properties) was not actually achieved when the environment had any
   embedded models - the mapping build was skipped, but the O(N) expand-set scan
   was not. Moved the `expression.find(exp.Table)` check to the top of the
   function, before expand is computed, since an expression with no table node
   can't be affected by expand either.

Adds test_resolve_table_table_mapping_only_dialect_mismatch (verifies fix 1 -
fails on the prior commit, passes here) and
test_resolve_tables_skips_expand_computation_without_table_refs (verifies fix 2
via a dict subclass that counts .items() calls on `snapshots`, asserting it's
never called for a table-less expression even with an embedded snapshot
present - also fails on the prior commit, passes here).

Signed-off-by: Michael Day <mdaytn@gmail.com>
mday-io
mday-io commented Sep 15, 2026
View reviewed changes

@mday-io mday-io left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings: override-precedence regression and failing formatter check.

Comment thread sqlmesh/core/renderer.py Outdated
Comment thread tests/core/test_model.py Outdated
@mday-io
fix: preserve normalized table mapping override precedence
e3cf6b7 
Signed-off-by: mday-io <mdaytn@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 - Avoid full table mapping in renderer replace_table(s) calls

1 participant

@mday-io