WIP new migration#5399
Draft
denik wants to merge 39 commits into
Draft
Conversation
Collaborator
Integration test reportCommit: 64dfb18
24 interesting tests: 14 SKIP, 7 KNOWN, 2 FAIL, 1 flaky
Top 24 slowest tests (at least 2 minutes):
|
denik
force-pushed
the
denik/new-migration
branch
from
13572c4 to
82112d7
Compare
denik
force-pushed
the
denik/new-migration
branch
from
82112d7 to
969c201
Compare
denik
force-pushed
the
denik/new-migration
branch
from
d9d854f to
f9471b6
Compare
denik
force-pushed
the
denik/new-migration
branch
from
fbe207c to
8a1ad59
Compare
denik
force-pushed
the
denik/new-migration
branch
from
6cd9860 to
c8f326f
Compare
The command now reads from the local TF state file without invoking the Terraform binary, so there is nothing for the plan check to verify. Also simplify getCommonArgs to return only the display string (the args slice was only needed to forward to the plan subprocess). Co-authored-by: Isaac
Remove --noplancheck from expected output after dropping that flag. Co-authored-by: Isaac
… update outputs
TF state stores single-block fields (e.g. continuous, deployment) as
single-element arrays [{}], not plain objects. json.Unmarshal into the
generated Go schema structs fails with type mismatch. Switch LookupTFField
to navigate via map[string]any + custom navigateTFState that auto-unwraps
single-element lists when a string-key step follows.
Also:
- Restore --noplancheck as a no-op flag (backward compat; used by invariant
tests for job_with_depends_on config).
- Remove the plan-check lines from acceptance test output.txt files.
- Update help output to include --noplancheck.
Co-authored-by: Isaac
…lancheck The refs map is shared with sv.Refs and gets mutated (entries deleted) during reference resolution. depends_on must be computed before that loop runs. Also restore --noplancheck as a no-op flag kept for backward compatibility (used by the invariant test suite for job_with_depends_on config). Co-authored-by: Isaac
…lias Two categories of field lookup failures in LookupTFField: 1. Postgres resources (postgres_projects, postgres_branches, etc.) use a 'spec' wrapper via DABsToTerraformWrappers, but some fields like 'name' are at the TF state root, not under spec. When the spec-prefixed path fails, retry with the original unwrapped path. 2. Model permissions reference 'model_id' (the numeric model ID) which TF stores as 'registered_model_id'. Add a tfStateFieldAliases map for such state-only field name mismatches. Also update acceptance output files: - default-python: remove musterr/--noplancheck pattern; the plan check that made the first migrate call fail no longer exists, so just call migrate once. - runas: remove plan-check output lines that appeared before 'Success!'. Co-authored-by: Isaac
Co-authored-by: Isaac
Tests cover: - Basic job stored with correct ID - Resource absent from TF state is skipped - Cross-resource string ref: depends_on computed, field resolved from TF state - Cross-resource numeric ref: int value stored as number not string - Dashboard etag stored from etags map Co-authored-by: Isaac
Co-authored-by: Isaac
Both callers were reading and parsing the same .tfstate file twice —
once for resource IDs, once for full attributes. Now a single
ParseTFStateFull reads and unmarshals the file once, returning attrs,
IDs, and lineage/serial together.
Also drop the separate `etags map[string]string` parameter from
BuildStateFromTF. The dashboard etag is a regular attribute in the TF
state JSON ("etag" field), so LookupTFField finds it directly without
any special-case plumbing.
Implementation:
- terraform: expose ParseResourcesStateFromBytes so callers can pass
already-read bytes
- migrate: add ParseTFStateFull / parseTFStateAttrsFromBytes
- migrate: remove etags param; look up "etag" via LookupTFField
- tests: etag test now puts the etag in the TF attributes JSON
Co-authored-by: Isaac
Still useful as a standalone API; suppress the dead-code checker. Co-authored-by: Isaac
Co-authored-by: Isaac
Return (nil, nil, ...) when the state file doesn't exist, matching the old parseResourcesState behaviour. Empty bundles with no resources don't create a terraform.tfstate file. Co-authored-by: Isaac
Remove the musterr + --noplancheck pattern; the plan check was removed so migration succeeds on the first call. Regenerate output files. Co-authored-by: Isaac
Co-authored-by: Isaac
YAML + TF JSON as input, expected state fields as output. Co-authored-by: Isaac
Co-authored-by: Isaac
Replace the verbose LookupTFField call for etag with a direct JSON read via TFStateAttrs.ETagFor(group, name). Co-authored-by: Isaac
- Introduce rawTFState that captures lineage/serial alongside resources in one unmarshal, eliminating the separate meta struct parse. - parseTFStateAttrsFromBytes and parseTFStateAttrsFromRaw now share the same struct instead of defining an anonymous one. - Move Lineage/Serial after Attrs/IDs in TFState struct. Co-authored-by: Isaac
Co-authored-by: Isaac
Co-authored-by: Isaac
DABsPathToTerraform now correctly handles root-level TF fields for wrapped groups (e.g. postgres) via DABsToTerraformWrapperFields, so the retry-with-unwrapped-path fallback is no longer needed. Co-authored-by: Isaac
Co-authored-by: Isaac
denik
force-pushed
the
denik/new-migration
branch
from
4932be2 to
dcd1c2e
Compare
- Remove the unused `rest` variable in the alias resolution block (SkipPrefix(1) on a single-segment path always returns nil) - Remove ParseTFStateAttrs which has no callers; the deadcode:allow annotation was papering over the linter Co-authored-by: Isaac
Co-authored-by: Isaac
BuildStateFromTF silently skips resources not in TF state (like schema grants in terraform mode), so the old "state entry not found" warning no longer fires. Remove the two spurious warning lines. Co-authored-by: Isaac
The ETag field is no longer needed in the migration path — etags are read directly from TFStateAttrs.ETagFor. Replace ExportedResourcesMap with map[string]string throughout migrate, build_state, and callers. Co-authored-by: Isaac
json.Unmarshal into map[string]any decodes all JSON numbers as float64. Integers beyond 2^53 (~9e15) lose precision: 9007199254740993 becomes 9007199254740992. run_job_task.job_id is a JSON number in TF state and realistic job IDs can exceed this threshold. Fix: use json.NewDecoder.UseNumber() when parsing TF state attributes so numbers are preserved as their original decimal string. Add a unit test case with job_id = 2^53+1 that asserts the raw state JSON contains the exact value, and an invariant config (job_run_job_ref.yml.tmpl) that exercises the run_job_task.job_id cross-reference in cloud tests. Co-authored-by: Isaac
Extend job_run_job_ref so watcher_job references trigger_job's max_concurrent_runs literal of 2^53+1 (9007199254740993). This drives the migration to resolve a large integer from TF state end-to-end; with the json.Number fix the migrated state keeps the exact value, without it the value is truncated to 2^53. The value is out of range for the real backend, so the config is local-only (no_run_job_ref_on_cloud). Note: the unit test (TestBuildStateFromTF/large_integer) is the deterministic regression guard, since the "no drift" plan check cannot detect this — the plan diff also collapses ints to float64. Co-authored-by: Isaac
The INPUT_CONFIG matrix is shared by all invariant test consumers; continue_293's generated out.test.toml was missed in the prior commit. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Why
Tests