CI: automate dynamic e2e test for kapp-controller against latest & minimum supported kubernetes releases

[himsngh]

What this PR does / why we need it:

This PR introduces a fully automated, decoupled nightly pipeline that dynamically calculates the Kubernetes support window and tests the latest versions of kapp-controller against it.

This PR adds workflow files that will be used to test kapp-controller against latest k8s release and a minimum supported version release.

This will be an automation which will be triggered at 2am everyday and run kapp-controller tests on the latest kubernetes kind node image generated from the latest commit of that branch.

Nightly Build:

https://github.com/himsngh/kapp-controller/actions/runs/26330738915

E2E Tests:

https://github.com/himsngh/kapp-controller/actions/runs/26330756076
https://github.com/himsngh/kapp-controller/actions/runs/26330757422

Which issue(s) this PR fixes:

Fixes # #1787

Does this PR introduce a user-facing change?

Additional Notes for your reviewer:

Review Checklist:

• Follows the developer guidelines
• [] Relevant tests are added or updated
• [] Relevant docs in this repo added or updated
• [] Relevant carvel.dev docs added or updated in a separate PR and there's
  a link to that PR
• Code is at least as readable and maintainable as it was before this
  change

Additional documentation e.g., Proposal, usage docs, etc.:

[aroradaman]

Seeing this as matrix on running locally. Is it a range (start/end) or list of all the supported versions?

{"k8s_branch":["release-1.36","release-1.33"]}

[aroradaman]

How are we caching it? Is the CLN tag or version itself is the tag here?

[aroradaman]

Also, can we include main/master here as well?

[aroradaman]

We can reduce the frequency here, release branches are not actively developed.
Or can we magically skip if TOT of release branch hasn't changed?

[aroradaman]

Review: Nightly Build Kind Images + E2E Workflows

This PR adds two workflows to automate nightly Kind image builds from Kubernetes source and downstream E2E test dispatch. The overall approach is sound, but there are several correctness issues that will cause silent failures or hard failures on clean runners.

---

Bugs (actionable)

1. kind binary is never installed in nightly-e2e.yml

The e2e-tests job calls /usr/local/bin/kind directly in both the "Create Kind Cluster" and "Build, Load and Deploy" steps, but no step installs kind. GitHub-hosted ubuntu-latest runners do not ship kind by default. This will fail immediately on any fresh runner with No such file or directory.

nightly-e2e.yml — "Create Kind Cluster" and "Build, Load and Deploy" steps.

Fix: Add a uses: helm/kind-action@v1 or a manual install step (e.g. go install sigs.k8s.io/kind@latest) before those steps, analogous to how the build workflow checks out the kind source.

---

2. BRANCH_SHA can be empty in nightly-build-kind.yml with no guard

In the "Setup Environment Variables" step:

BRANCH_SHA=$(git ls-remote https://github.com/kubernetes/kubernetes.git refs/heads/${{ matrix.k8s_branch }} | awk '{print $1}' | cut -c1-7)
echo "BRANCH_SHA=$BRANCH_SHA" >> "$GITHUB_ENV"

If git ls-remote returns nothing (transient network error, unknown ref), BRANCH_SHA is empty and set -u does not catch it because the variable is assigned (just to an empty string). Downstream, CACHE_IMAGE becomes …:release-1.31- — a malformed tag — and the cache-check logic produces a false miss, triggering a full multi-hour rebuild unnecessarily (or worse, pushing a broken tag).

Fix: Add [[ -n "$BRANCH_SHA" ]] || { echo "::error::Failed to resolve SHA for ${{ matrix.k8s_branch }}"; exit 1; } after the assignment.

---

3. nightly-e2e.yml resolve-versions has no empty-BRANCHES guard

nightly-build-kind.yml correctly validates:

if [[ -z "$BRANCHES" ]]; then
  echo "::error::Failed to fetch Kubernetes branches."
  exit 1
fi

nightly-e2e.yml skips this guard entirely. If git ls-remote returns nothing, LATEST_BRANCH and MIN_BRANCH are empty strings, MATRIX_JSON becomes {"k8s_version": ["", ""]}, and the matrix spawns two jobs that try to pull an image tagged :-<timestamp> with a confusing registry error instead of a clear "branch resolution failed."

Fix: Add the same guard as in nightly-build-kind.yml.

---

4. success() + fail-fast: false interaction silently skips E2E triggering

In nightly-build-kind.yml:

trigger-tests:
  needs: [resolve-versions, build-and-push]
  if: success()

build-and-push uses fail-fast: false across a 2-leg matrix. If one leg fails and the other succeeds, GitHub evaluates success() as false for the downstream job — meaning trigger-tests is skipped entirely, and a successfully-built image never gets its E2E tests run.

Fix: Replace if: success() with if: ${{ !cancelled() && !failure() }} or use needs.build-and-push.result == 'success' || needs.build-and-push.result == 'failure' with a more granular check, or simply remove the if: clause and rely on matrix-level reporting.

---

5. git fetch upstream --tags without --force may check out the wrong commit

In nightly-e2e.yml "Fetch Upstream Tags":

git fetch upstream --tags
git checkout "$TARGET_REF"

If the checked-out fork already has a local tag v0.45.2 pointing to a different commit than the upstream tag, git fetch upstream --tags will not overwrite the existing local tag (that requires --force). git checkout v0.45.2 silently lands on the fork's stale tag, and E2E tests run against the wrong code with no diagnostic output.

Fix: Use git fetch upstream --tags --force to ensure upstream tags always win.

---

6. sed-based version injection is fragile

sed -i "s|function get_kappctrl_ver() {|function get_kappctrl_ver() { echo \"$VERSION\"; return; |g" hack/version-util.sh

This is a brittle string patch: if hack/version-util.sh is ever reformatted (e.g., whitespace changes around {), the pattern silently matches nothing, get_kappctrl_ver returns the wrong version, and tests pass or fail against an incorrectly-tagged image with no diagnostic. Consider exposing a VERSION env var that hack/version-util.sh already reads, or adding a dedicated --version flag to the build script.

---

Minor

• if: success() on trigger-tests is the default behavior when a job has needs: — it can be removed (see bug #4 above for the more important implication of this).
• The resolve-versions logic (parsing K8s branches from git ls-remote, building the latest/N-3 matrix) is duplicated verbatim between the two workflows. Extracting it to a reusable workflow would prevent the guard asymmetry in finding #3 from reoccurring.