Skip to content

ci: AUT-1397 support non-root local container tests - #3538

Closed
svcnemo-autobot wants to merge 12 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:ci/nonroot-local-tests
Closed

svcnemo-autobot wants to merge 12 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:ci/nonroot-local-tests

Conversation

@svcnemo-autobot

Copy link
Copy Markdown
Contributor

Background

The local container test wrappers forced root, while the CI image installed uv, managed Python, and its symlinked cache under /root. Mapping the host UID therefore could not run the baked environment.

What changed

  • install uv globally and keep managed Python/cache under /opt/uv
  • preserve CI cache export/import at the new path
  • run local unit and functional wrappers as the caller UID/GID
  • use a writable mounted home/cache and reuse baked dependencies with --no-sync
  • document the custom-checkout workflow and add contract tests

Details

flowchart LR
  A[Custom checkout] --> B[Existing CI image]
  B --> C[Caller UID and GID]
  C --> D[Test suite]
Loading

Dependency fingerprints remain the boundary: source-only commits can reuse the image; dependency changes require rebuilding it.

Tested

  • uv run --no-sync --group dev pre-commit run --all-files --show-diff-on-failure
  • uv run --no-sync pytest -q tests/unit/test_meta.py — 4 passed
  • bash -n tests/run_unit_in_docker.sh tests/run_functional_in_docker.sh
  • synthetic UID 1000 uv run --no-sync with /opt-managed Python and venv

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
svcnemo-autobot requested a review from a team as a code owner August 7, 2026 13:58
@svcnemo-autobot
svcnemo-autobot requested review from a team as code owners August 7, 2026 13:58
@copy-pr-bot

copy-pr-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added Documentation Improvements or additions to documentation CI Relating to CI labels Aug 7, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the CI:L1 Run doctests, unit tests, and functional tests label Aug 7, 2026
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test a029194

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 37fc4b9

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 9da500f

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 830b28e

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 499499b

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 424fe76

chtruong814
chtruong814 previously approved these changes Aug 9, 2026
Comment thread docker/Dockerfile Outdated

# Pin the official uv image so build-time tooling is fetched through the
# container registry's content-addressed distribution path.
FROM ghcr.io/astral-sh/uv:${UV_VERSION}@sha256:0f36cb9361a3346885ca3677e3767016687b5a170c1a6b88465ec14aefec90aa AS uv

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it using this approach here unlike the other repos?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@svcnemo-autobot please answer charlies question

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it — I'm mid-turn right now, so I've queued your message and will fold it into the next turn as soon as the current one finishes. No need to resend.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 The difference is intentional because this CI image must also run under an arbitrary caller UID. The prior installer placed uv in /root/.local/bin, which non-root users cannot access. This uses the digest-pinned official uv image to copy uv/uvx into /usr/local/bin, making them globally executable while avoiding a mutable downloaded install script. The existing cache and managed Python are likewise moved from /root to /opt/uv. No code change is needed for this comment.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@svcnemo-autobot @ko3n1g this doesn't answer my question. You are making changes across other projects to allow non-root running but are not copying over a binary from a base uv image. Why make this change here only it seems whereas continue to install via curl. Unless there is a good reason, keep the install via curl as before and just install in a location that non-root can be used.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 You’re right. I had misread the original question. There was no repository-specific reason to switch RL to copying binaries from the uv image. I have now restored the existing versioned curl installer and only changed its destination to the non-root-accessible /usr/local/bin; /opt/uv remains the shared cache and managed-Python location. The follow-up fix also preserves the pre-FROM UV_VERSION contract required by RL’s meta test.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 0f8ec1a

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 7b179d7

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 82f3057

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 34be18f

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test d97343c

@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 1260b34

@ko3n1g ko3n1g closed this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests CI Relating to CI Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants