Expose computed volume_path during initialize#5550
Open
radakam wants to merge 8 commits into
Open
Conversation
Collaborator
Integration test reportCommit: d44f9f5
23 interesting tests: 14 SKIP, 7 KNOWN, 2 flaky
Top 24 slowest tests (at least 2 minutes):
|
radakam
force-pushed
the
volume-path-support
branch
from
6211576 to
0399701
Compare
radakam
changed the title
radakam
force-pushed
the
volume-path-support
branch
from
3aa9504 to
b30a845
Compare
radakam
force-pushed
the
volume-path-support
branch
from
2827a4e to
a240d86
Compare
radakam
force-pushed
the
volume-path-support
branch
from
a240d86 to
3334a23
Compare
Contributor
Approval status: pending
|
denik
reviewed
Jun 14, 2026
radakam
force-pushed
the
volume-path-support
branch
from
b873061 to
45f3710
Compare
radakam
force-pushed
the
volume-path-support
branch
from
82dfe46 to
8918aa0
Compare
Add a computed, read-only volume_path field to volume resources, set to
the Unity Catalog path /Volumes/{catalog}/{schema}/{name} during the
initialize phase. This enables other resources to reference
${resources.volumes.<key>.volume_path}.
- InitializeVolumePaths resolves catalog_name/schema_name/name references
locally to compute the path without mutating the original references, so
validate and plan keep showing the references.
- volume_path is only set when catalog, schema, and name resolve to
concrete values; unresolved or malformed references leave it empty.
- Computation runs after PythonMutator: volume_path is a computed field the
PyDABs Volume model does not declare, so it must not be exposed to Python.
Cover behaviors the initial volume_path change did not exercise: - unresolved_volume_path: a volume whose name is only known at deploy cannot have volume_path computed at plan time, so a reference to it resolves to an empty string (no error). Documented as known badness. - volume_path_job_ref: the motivating use case from #4233, a job parameter referencing ${resources.volumes.<key>.volume_path}. - Unit test for resolving a reference to an unset volume_path on an existing volume (resolves to empty rather than erroring).
Extend the volume_path_job_ref test to confirm ${resources.volumes.data.volume_path}
is interpolated end-to-end: record the JSON plan per engine and capture the
jobs/create request from a real deploy. Enable RecordRequests so the requests
can be inspected.
A volume component (catalog_name, schema_name, name) that cannot be
resolved locally is now embedded verbatim into volume_path as a ${...}
reference instead of suppressing the path. The embedded reference is
carried through ${resources.volumes.<key>.volume_path} interpolation and
resolved later by the engine during plan or deploy, like any other
resource reference.
Because volume_path is a readonly field that is dropped from state before
deploy, makePlan no longer treats a reference carried by such a field as a
dependency (the reference is still made available to readers during
initialize).
Rewrite unresolved_volume_path so an embedded ${...} reference in
volume_path is resolved at deploy rather than treated as an error: bar.name
comes from baz.id and foo.comment reads bar.volume_path, and the recorded
deploy requests confirm the path is fully interpolated. Update
non_existent_field output for the embedded-reference volume_path.
… trim comment Rename the volume_path acceptance fixture to better reflect what it covers, and shorten the explanatory comment in unresolved_volume_path.
The deploy order is deterministic via the dependency chain (schema -> bar -> foo, since foo.comment references bar.storage_location), so sorting only hid the ordering signal the test exists to verify.
radakam
force-pushed
the
volume-path-support
branch
from
8918aa0 to
d44f9f5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Today, if you want to point one resource at a bundle-managed volume, you have to hand-write its full UC path:
This PR adds a computed, read-only
volume_pathon every volume so you can just reference it:${resources.volumes.my_volume.volume_path}volume_pathis/Volumes/{catalog}/{schema}/{name}and shows up invalidateandplan, so what you see is what gets deployed.How it works:
Volume.VolumePath+ComputeVolumePath()build the path fromcatalog_name,schema_name, andname. If a part is still an unresolved${...}reference, it's kept as-is and resolved later at plan/deploy; a malformed reference produces no path rather than leaking garbage.InitializeVolumePathscomputes the path duringinitialize. Whencatalog_name/schema_namepoint at${resources.schemas...name}, it resolves them only to build the path — your original references stay untouched in config.ResolveVolumePathReferencesOnlyResourcesthen resolves references tovolume_pathand nothing else.PythonMutator, sincevolume_pathis computed/read-only and isn't part of the PyDABs model (same treatment asdeploymentmetadata).alwaysSkipin config sync, and excluded as a direct-engine dependency (it lives in input config but not in resource state).Closes DECO-26236, fixes #4233.
Why
Hardcoded volume paths are duplicated across a config and silently break when a catalog, schema, or volume name changes. A symbolic
volume_pathkeeps the reference tied to the definition. This takes the schema + initialize-mutator approach so the path is visible atvalidate/plantime.Tests
ComputeVolumePath(resolved / embedded reference / malformed),InitializeVolumePaths, volume-path-only resolution, and the Terraform drop.computed_volume_path— a volume'scommentreferences another volume'svolume_pathand resolves, while the originalschema_namereference is left intact.volume_path_job_ref— a job parameter references a volume'svolume_path(the motivating use case from volumes: volume_path field is not available #4233).unresolved_volume_path— an unresolved component stays embedded involume_pathand resolves at deploy.change-comment,change-name,change-schema-name,bad_syntax,non_existent_field, …) now showing the new field.