AddFiles: CommitSchemaUnion - #40104
Open
claudevdm wants to merge 1 commit into
Open
Conversation
…ble in one transaction
The commit side of the pre-pass, for an existing table (creation comes
next). commit(catalog, tableId, schemas, config, handling, committer)
takes the distinct canonical file schemas of one window, most common
first, and returns the table's schema id afterwards.
One attempt, in order:
1. Fresh load of the table; the classification below must run against
the current schema, never a cached one.
2. SchemaDelta.classify for each schema. Empty deltas are skipped;
deltas the config does not allow go to the incompatible list with
their reason; the rest are accepted.
3. Staging, most common first, on one Transaction: per accepted schema
one unionByNameWith plus an explicit makeColumnOptional for every
path the delta reported as an absent required column. Two accepted
schemas can still conflict with each other (file A says score is
long, file B says string; each is fine against the table alone).
That only surfaces while staging and poisons the transaction, so the
offender is moved to the incompatible list ("conflicts with another
file schema in the same window") and the transaction is rebuilt
without it. Most common first means the majority wins such a tie.
4. relaxNewRequiredFields: the union adds top-level columns as optional
but keeps the file's optionality for fields inside a struct it just
added, so one file's luck would impose required nested columns on
every future file. Every field new in this transaction is made
optional at every level, except pinned paths and fields under lists
and maps (which the union already handles). Found by fuzzing;
testFieldsInsideAnAddedStructAreOptional covers it.
5. Name mapping repair: if the table's schema.name-mapping.default is
absent, malformed or does not cover the staged schema, it is
regenerated (NameMappingUtils.regenerate, preserving custom names by
field id). Zero-copy files carry no field ids, so a column missing
from the mapping is unreadable in every registered file; a missing
mapping alone is reason enough to commit even when the schema did
not change.
6. Incompatible schemas are reported: under FAIL_PIPELINE an
IncompatibleSchemaException listing every one (schema, file count,
reason) is thrown before anything is committed; under
ROUTE_TO_ERRORS they are logged and the rest proceeds (their files
fail the per-file coverage check later, with the same reason).
7. Single commit through the injectable Committer, then refresh.
Nothing is committed when nothing was staged.
Retry: CommitFailedException (a concurrent writer moved the table
between load and commit) restarts the whole attempt from step 1, up to
MAX_ATTEMPTS (5); classification is redone against the new state rather
than replayed, since the concurrent change may have made a delta empty
or incompatible. A persistent failure propagates.
Worked example: table {id required long, name required string}; window
schemas A x5 {id, name optional, score long}, B x1 {id, score string}
under all options. A: relax name, add score. B: absent name
(relaxation), score conflicts with A's addition while staging, so B is
incompatible. FAIL_PIPELINE: exception, table untouched. ROUTE_TO_ERRORS:
one commit with name optional and score optional long; B's file will be
routed to errors at registration.
The tests use a local HadoopCatalog and check metadata versions to
prove "nothing committed" claims, and inject a Committer that fails
once to exercise the retry path.
Contributor
|
Assigning reviewers: R: @chamikaramj for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The commit side of the schema pre-pass for an existing table. Takes one window's distinct canonical file schemas (most common first) and evolves the table once to cover them all, adding exactly one schema version per window regardless of how many distinct schemas it held.
Flow, per attempt:
Retry: CommitFailedException restarts from the load, up to 5 attempts, thrown by the committer on a concurrent commit, and by a guard on every transaction creation (Iceberg refreshes per newTransaction(), so mid-attempt concurrent schema changes are detected as drift from the classified snapshot and retried rather than failed). Classification is redone against the new state, not replayed.
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.