Skip to content

ci: reduce the acceptance test matrix to version boundaries#294

Merged
tnj merged 2 commits into
masterfrom
ci/minimize-acceptance-test-matrix
Jun 16, 2026
Merged

ci: reduce the acceptance test matrix to version boundaries#294
tnj merged 2 commits into
masterfrom
ci/minimize-acceptance-test-matrix

Conversation

@tnj

@tnj tnj commented Jun 12, 2026

Copy link
Copy Markdown
Member

Summary

Running the acceptance tests (acceptance-test-runtime-env) against every AGP minor version (17 rows) is costly. This reduces the matrix to the 4 rows where behavior can actually change.

Rationale

The plugin only touches AGP through the stable public API (com.android.build.api.*):

  • androidComponents.onVariants(selector().all())
  • variant.name / artifacts / buildType / flavorName (ComponentIdentity)
  • artifacts.get(SingleArtifact.APK / BUNDLE), getBuiltArtifactsLoader(), BuiltArtifacts.elements, BuiltArtifact.outputFile
  • com.android.Version.ANDROID_GRADLE_PLUGIN_VERSION and AndroidProject.MODEL_LEVEL_LATEST (via reflection, wrapped in try/catch)

Extracting these classes from the gradle-api / builder-model jars of AGP 8.0.0 / 8.7.0 / 8.11.0 / 8.13.0 / 9.2.0 and comparing them with javap shows that every method and field the plugin calls is byte-identical across all versions (MODEL_LEVEL_LATEST still exists in 9.2.0). The Variant/Artifacts API has only ever changed additively, with no breaking diffs to the surface the plugin uses. In other words, the variable that actually changes across the matrix is not AGP but the paired Gradle version.

Versions kept

AGP Gradle Boundary covered
8.0.0 8.0 Minimum supported pair (oldest AGP + oldest Gradle)
8.13.0 8.13 End of the 8.x line + newest Gradle 8
9.0.0 9.1.0 Major boundary (Gradle 9 / Groovy 4 / built-in Kotlin / V1 API removal)
9.2.0 9.4.1 Latest pair, catches regressions in new releases early

Notes

  • The skipAssemble paths (apkInfoFromConventionalOutput / aabInfoFromConventionalOutput) rely on AGP's conventional output directories (outputs/apk/..., outputs/bundle/...), which are not API-guaranteed. This is the one place a future minor could silently break, and it is covered by keeping the latest row (currently 9.2.0) up to date. Please bump the latest row whenever a new AGP minor is released.
  • run_acceptance_tests.bash still listed unsupported AGP 4.2-7.x and stopped at 8.7, so it is synced to the same 4 pairs as the CI matrix.

🤖 Generated with Claude Code

The plugin only uses the stable public Variant/Artifacts API
(com.android.build.api.*). Comparing the signatures of every method the
plugin calls (onVariants/selector, Artifacts.get(SingleArtifact.APK/BUNDLE),
BuiltArtifactsLoader.load, BuiltArtifact.outputFile, ComponentIdentity, and
AndroidProject.MODEL_LEVEL_LATEST) across AGP 8.0.0 through 9.2.0 shows they
are byte-identical, so running every minor version adds no coverage.

Keep only the rows where behavior can actually change:
- 8.0.0 / Gradle 8.0: minimum supported pair
- 8.13.0 / Gradle 8.13: last 8.x line with the newest Gradle 8
- 9.0.0 / Gradle 9.1.0: major boundary (Gradle 9 / Groovy 4 / built-in Kotlin)
- 9.2.0 / Gradle 9.4.1: latest pair, catches regressions in new releases

Also sync run_acceptance_tests.bash, which still listed AGP 4.2-8.7.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings June 12, 2026 08:48

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the run_acceptance_tests.bash script by removing several older Android Gradle Plugin (AGP) and Gradle version combinations and adding newer versions (8.13.0, 9.0.0, and 9.2.0). It also adds a comment reminding developers to keep this list in sync with the GitHub Actions workflow. I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces the cost of running acceptance-test-runtime-env by shrinking the GitHub Actions matrix from “every AGP minor” to a small set of boundary AGP/Gradle pairs, and aligns the local run_acceptance_tests.bash matrix with CI.

Changes:

  • Reduce acceptance-test-runtime-env matrix to four AGP/Gradle boundary pairs and document the rationale in CI.
  • Update run_acceptance_tests.bash to use the same four pairs and add a note to keep it in sync with CI.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
run_acceptance_tests.bash Removes legacy AGP/Gradle pairs and syncs the local acceptance test loop to the reduced boundary matrix.
.github/workflows/build-and-test.yml Shrinks the runtime acceptance test matrix to boundary versions and adds explanatory comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread run_acceptance_tests.bash
Comment thread .github/workflows/build-and-test.yml Outdated
Comment thread .github/workflows/build-and-test.yml
- Define the matrix with include-only entries so each AGP/Gradle/Java
  pair lives in a single place
- Move the keep-in-sync comment above the loop in run_acceptance_tests.bash
- Reword "newest Gradle 8" to "its pinned Gradle 8.x" so the comment
  stays accurate over time

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tnj tnj requested a review from satsukies June 15, 2026 09:17

@satsukies satsukies left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tnj tnj merged commit 23fcab1 into master Jun 16, 2026
10 checks passed
@tnj tnj deleted the ci/minimize-acceptance-test-matrix branch June 16, 2026 01:45
@tnj

tnj commented Jun 16, 2026

Copy link
Copy Markdown
Member Author

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants