Skip to content

feat(model-capabilities): reshape match kinds to exact+prefix (Phase 1, data only) - #5597

Draft
wpfleger96 wants to merge 2 commits into
mainfrom
duncan/model-capabilities-manifest
Draft

feat(model-capabilities): reshape match kinds to exact+prefix (Phase 1, data only)#5597
wpfleger96 wants to merge 2 commits into
mainfrom
duncan/model-capabilities-manifest

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Aug 11, 2026

Copy link
Copy Markdown
Member

Summary

Phase 1 of the ratified "manifest-sourced model capabilities — no codegen" plan: a data-only reshape of the model-capability manifest. This PR carries no interpreter code — the Rust/TS interpreters, CI wiring, and generator/artifact deletion land in Phases 2/3 as follow-up dispatches. Because there is no interpreter in-tree yet, Phase 1 verification is structural, not executable (the corpus can't run in CI until Phase 2's resolvers exist; it was validated during development against a throwaway resolver matching the ratified semantics).

Two files change, both under scripts/:

  • model-capabilities.json — the reshaped manifest (single runtime artifact)
  • normative-corpus.json — the cross-language contract corpus

What changed in the data

  • 30 exact records materialized to complete six-axis authoritative snapshots. Sparse records (24 label-only, 6 effort-override) were expanded via one final generator run out of the 3603 lift-source branch; each record carries a _provenance field naming which rule/fallback supplied its axes at materialization time. Exact records no longer inherit at runtime.
  • family_rules flattened from 6 match kinds → exact + prefix only. No match_priority, no exotic segment/token grammars. Prefix semantics are boundary-aware (a token matches at string start only when the next character is end-of-string, -, or .), longest-match wins, lexicographic rule-id tie-break. The 6 openai-gpt5-* prefix rules retain a small match_aliases list of alternate literal spellings (e.g. gpt5-pro, gpt-5-6, gpt5.6) that share the rule's payload — this stays strictly within exact+prefix semantics (an alias is just another literal prefix token), no exotic kind returns.
  • _generated_by removed and the root _comment rewritten to state the manifest is the runtime artifact + the maintenance rule (any capability change to a prefix rule or provider fallback must audit the exact snapshots whose _provenance names that source).
  • Doc keys normalized per the verified inventory — kept as explicit documented keys or intentionally dropped; none left to be discovered at parse time.
  • Corpus updated to 121 entries / 103 executable vectors (from 95/81): 81 re-pointed to the new shape, plus the gpt-5/gpt-5-pro/gpt-5-10 collision family, uncurated internal-token cases, the gptoss/gpt-neox-20b negatives, boundary vectors per distinct grammar, and two vectors guarding the highest-value pins (the class-B2 databricks_v2 multi-digit case and the sole deliberately-divergent exact record).

Curated-model immunity (the load-bearing invariant)

All 30 curated Databricks v2 records resolve at exact lookup, before any prefix matching. 30/30 resolve byte-identical (all six axes, case-insensitive path included) through the reshaped resolver, verified against the prior generated TS interpreter as oracle. Every enumerated delta below is an uncurated or adversarial input — zero curated regressions.

One curated record (databricks-gpt-5-2) carries a snapshot that deliberately diverges from what the kept gpt-5 prefix would now produce; it pins the removed dbv2-gpt-code-names-segment outcome (none..xhigh + openai-clamp-max-to-xhigh vs the prefix's minimal..high + openai-standard). The divergence is why it's pinned rather than derived, its _provenance says so, and the corpus now guards it (dbv2-gpt-5-2-divergent-pin-preserved).

Compatibility contract — match-kind flatten

Six match_kind variants collapse to two (exact + prefix). The delta is outcome-preserving for every curated model and drops only uncurated/adversarial inputs to the two provider fallbacks; each drop below names its fallback destination.

Removed match kinds → disposition (6 kinds)
Removed match_kind Prior rule(s) What it did Disposition
gpt5-base openai-gpt5-base gpt-5 prefix guarded against a bare 1–3 digit minor version (gpt-5-10, gpt-5-2) → those fell to provider fallback Kept as prefix on gpt-5; multi-digit guard dropped (class B1/B2)
gpt5-token openai-gpt5-{pro,6,5,4,1} matched the token on a left boundary with end/- right boundary → caught it embedded mid-string, treated .-suffix as non-match Kept as prefix; alias spellings retained as match_aliases; embedded-token capture dropped (C), .-boundary widened (D)
gpt-version-segment dbv2-gpt-code-names-segment DBv2: routed any gpt-segment-then-numeric endpoint to openai-responses Dropped. Curated GPT endpoints pinned exact; uncurated gpt-* DBv2 → mlflow-chat (E1/E2). Segment-anywhere isn't expressible as a prefix
segment (Claude names) dbv2-claude-code-names-segment DBv2: routed any endpoint containing a Claude code-name segment to anthropic-messages (omit-fields) Partially replaced by new dbv2-claude-prefix (token claude); bare code-name segments with no leading claude droppedmlflow-chat (F)
segment (OpenAI code names) dbv2-sol-luna-terra-segment DBv2: routed sol/luna/terra segment endpoints to openai-responses Dropped. Curated databricks-gpt-5-6-{sol,luna,terra} pinned exact; bare segments → mlflow-chat (G)
segment-prefix (unused) declared in the enum, no rule used it Removed, zero behavior impact
31-row literal spelling inventory (rule × alias/grammar → disposition)

Mechanical completion target = 31 literal spellings: 20 gpt5 + 6 claude/code-names + 2 gpt grammar tokens + 3 sol/luna/terra. gpt-followed-by-numeric is a grammar, represented here by its two anchor tokens.

# Prior rule Literal spelling Disposition
1 openai-gpt5-pro gpt-5-pro retained-prefix
2 openai-gpt5-pro gpt5-pro retained-prefix (alias)
3 openai-gpt5-6 gpt-5.6 retained-prefix
4 openai-gpt5-6 gpt5.6 retained-prefix (alias)
5 openai-gpt5-6 gpt-5-6 retained-prefix (alias)
6 openai-gpt5-6 gpt5-6 retained-prefix (alias)
7 openai-gpt5-5 gpt-5.5 retained-prefix
8 openai-gpt5-5 gpt5.5 retained-prefix (alias)
9 openai-gpt5-5 gpt-5-5 retained-prefix (alias)
10 openai-gpt5-5 gpt5-5 retained-prefix (alias)
11 openai-gpt5-4 gpt-5.4 retained-prefix
12 openai-gpt5-4 gpt5.4 retained-prefix (alias)
13 openai-gpt5-4 gpt-5-4 retained-prefix (alias)
14 openai-gpt5-4 gpt5-4 retained-prefix (alias)
15 openai-gpt5-1 gpt-5.1 retained-prefix
16 openai-gpt5-1 gpt5.1 retained-prefix (alias)
17 openai-gpt5-1 gpt-5-1 retained-prefix (alias)
18 openai-gpt5-1 gpt5-1 retained-prefix (alias)
19 openai-gpt5-base gpt-5 retained-prefix (multi-digit guard dropped → B1/B2)
20 openai-gpt5-base gpt5 retained-prefix (alias)
21 dbv2-claude-code-names-segment claude retained-prefix (new dbv2-claude-prefix)
22 dbv2-claude-code-names-segment opus dropped → mlflow-chat (F); curated pinned exact
23 dbv2-claude-code-names-segment sonnet dropped → mlflow-chat (F); curated pinned exact
24 dbv2-claude-code-names-segment haiku dropped → mlflow-chat (F); curated pinned exact
25 dbv2-claude-code-names-segment mythos dropped → mlflow-chat (F); curated pinned exact
26 dbv2-claude-code-names-segment fable dropped → mlflow-chat (F); curated pinned exact
27 dbv2-gpt-code-names-segment gpt (segment-then-numeric grammar) dropped → mlflow-chat (E1/E2); curated pinned exact
28 dbv2-gpt-code-names-segment gpt5 (dashless grammar) dropped → mlflow-chat (E2); curated pinned exact
29 dbv2-sol-luna-terra-segment sol dropped → mlflow-chat (G); curated pinned exact
30 dbv2-sol-luna-terra-segment luna dropped → mlflow-chat (G); curated pinned exact
31 dbv2-sol-luna-terra-segment terra dropped → mlflow-chat (G); curated pinned exact

Intentional deltas (8 mechanism classes, all uncurated/adversarial)

The empirical delta measurement surfaced 8 distinct mechanism classes, not the plan's estimated 3. All are documented here rather than collapsed. Class H is an intentional correctness improvement (boundary-aware prefix narrowing an over-broad startsWith), not merely a flatten side effect.

Class Trigger Prior → Reshaped Change
B1 openai gpt-5-<1–3 digit> (gpt-5-10) fallback → base GPT-5 prefix base guard lost
B2 databricks_v2 gpt-5-<1–3 digit> (databricks-gpt-5-10) segment → base GPT-5 prefix; route stays openai-responses effort/normalization narrows; route unchanged
C token embedded mid-string (gpt-4-gpt-5-pro) openai-gpt5-pro → fallback embedded-token capture lost
D .-suffixed token (gpt-5.6.x) mixed → prefix binds on . boundary boundary widened
E1 databricks_v2 gpt-<non-5> (gpt-6, gpt-4o) openai-responsesmlflow-chat segment routing lost
E2 databricks_v2 dashless gpt5-segment (databricks-gpt5-custom) openai-responsesmlflow-chat segment routing lost
F databricks_v2 bare Claude code-name segment (goose-opus-5) anthropic-messagesmlflow-chat segment routing lost (soft degradation — loses the anthropic-messages cache opt only)
G databricks_v2 bare sol/luna/terra segment openai-responsesmlflow-chat segment routing lost
H kept-prefix token followed by a digit/letter with no separator (claude-35) startsWith bound → fallback boundary-aware prefix narrows an over-broad startsWith

Only E1/E2, F, G change the databricks_v2 wire route, and every one moves to mlflow-chat (Databricks's universal OpenAI-compatible wire). The corpus pins all eight classes with prior→reshaped values, including the class-B2 databricks_v2 multi-digit vector added in this PR.

Verification (structural — no interpreter in-tree yet)

  • Both JSON files parse; _generated_by absent; no match_priority; match kinds are exact/prefix only.
  • 30 exact records, each complete six-axis + _provenance.
  • 30/30 curated records byte-identical through the reshaped resolver (oracle diff); 103/103 corpus vectors green against a throwaway resolver matching the ratified semantics (exact → boundary-aware longest-prefix incl. aliases → provider fallback, with provider canonicalization inside per Contract 5).
  • Executable verification of the committed corpus is deferred to Phase 2, when the Rust + TS interpreters exist.

Originating discussion: buzz thread 30388ec72996901d324edd690681be71f49ac73f014e40776c3b1f50087fdb0a (channel tyler-wes-will).

Duncan and others added 2 commits August 11, 2026 16:00
Flatten the family-rule matcher from six match kinds (exact, prefix,
gpt5-token, gpt5-base, segment, segment-prefix) to two (exact, prefix),
dropping the per-rule integer match_priority. Materialize all 30 curated
Databricks v2 models as complete six-axis exact records so their outcomes
are structurally immune to matcher changes (verified byte-identical, all
axes, against the prior interpreter as oracle). Add a boundary-aware
longest-prefix resolver contract: prefix tokens bind only on -, ., or
end-of-string.

The reshape preserves every curated outcome and drops only uncurated /
adversarial inputs to the provider fallbacks. scripts/model-capabilities-compat.md
enumerates all eight delta mechanism classes with grammar, prior result,
and reshaped destination; the normative corpus re-points its eight
intentional-delta vectors to the new behavior and adds collision,
uncurated-token, negative, and boundary vectors (101 vectors total).

Data-only: interpreters and consumers land in later phases.
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…pat doc

Adds the two highest-value corpus vectors the reshape review flagged as
missing: a databricks_v2 gpt-5-<multi-digit> case (class B2, base prefix
binds while the openai-responses route is preserved) and a guard for the
sole deliberately-divergent exact record (databricks-gpt-5-2). Removes the
standalone compat table so the delta contract lives in the PR description,
restoring the two-file boundary; corpus notes now point at the PR.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant