ci: reduce the acceptance test matrix to version boundaries#294
Conversation
The plugin only uses the stable public Variant/Artifacts API (com.android.build.api.*). Comparing the signatures of every method the plugin calls (onVariants/selector, Artifacts.get(SingleArtifact.APK/BUNDLE), BuiltArtifactsLoader.load, BuiltArtifact.outputFile, ComponentIdentity, and AndroidProject.MODEL_LEVEL_LATEST) across AGP 8.0.0 through 9.2.0 shows they are byte-identical, so running every minor version adds no coverage. Keep only the rows where behavior can actually change: - 8.0.0 / Gradle 8.0: minimum supported pair - 8.13.0 / Gradle 8.13: last 8.x line with the newest Gradle 8 - 9.0.0 / Gradle 9.1.0: major boundary (Gradle 9 / Groovy 4 / built-in Kotlin) - 9.2.0 / Gradle 9.4.1: latest pair, catches regressions in new releases Also sync run_acceptance_tests.bash, which still listed AGP 4.2-8.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request updates the run_acceptance_tests.bash script by removing several older Android Gradle Plugin (AGP) and Gradle version combinations and adding newer versions (8.13.0, 9.0.0, and 9.2.0). It also adds a comment reminding developers to keep this list in sync with the GitHub Actions workflow. I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
There was a problem hiding this comment.
Pull request overview
This PR reduces the cost of running acceptance-test-runtime-env by shrinking the GitHub Actions matrix from “every AGP minor” to a small set of boundary AGP/Gradle pairs, and aligns the local run_acceptance_tests.bash matrix with CI.
Changes:
- Reduce
acceptance-test-runtime-envmatrix to four AGP/Gradle boundary pairs and document the rationale in CI. - Update
run_acceptance_tests.bashto use the same four pairs and add a note to keep it in sync with CI.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
run_acceptance_tests.bash |
Removes legacy AGP/Gradle pairs and syncs the local acceptance test loop to the reduced boundary matrix. |
.github/workflows/build-and-test.yml |
Shrinks the runtime acceptance test matrix to boundary versions and adds explanatory comments. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Define the matrix with include-only entries so each AGP/Gradle/Java pair lives in a single place - Move the keep-in-sync comment above the loop in run_acceptance_tests.bash - Reword "newest Gradle 8" to "its pinned Gradle 8.x" so the comment stays accurate over time Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks! |
Summary
Running the acceptance tests (
acceptance-test-runtime-env) against every AGP minor version (17 rows) is costly. This reduces the matrix to the 4 rows where behavior can actually change.Rationale
The plugin only touches AGP through the stable public API (
com.android.build.api.*):androidComponents.onVariants(selector().all())variant.name/artifacts/buildType/flavorName(ComponentIdentity)artifacts.get(SingleArtifact.APK / BUNDLE),getBuiltArtifactsLoader(),BuiltArtifacts.elements,BuiltArtifact.outputFilecom.android.Version.ANDROID_GRADLE_PLUGIN_VERSIONandAndroidProject.MODEL_LEVEL_LATEST(via reflection, wrapped in try/catch)Extracting these classes from the
gradle-api/builder-modeljars of AGP 8.0.0 / 8.7.0 / 8.11.0 / 8.13.0 / 9.2.0 and comparing them with javap shows that every method and field the plugin calls is byte-identical across all versions (MODEL_LEVEL_LATESTstill exists in 9.2.0). The Variant/Artifacts API has only ever changed additively, with no breaking diffs to the surface the plugin uses. In other words, the variable that actually changes across the matrix is not AGP but the paired Gradle version.Versions kept
Notes
skipAssemblepaths (apkInfoFromConventionalOutput/aabInfoFromConventionalOutput) rely on AGP's conventional output directories (outputs/apk/...,outputs/bundle/...), which are not API-guaranteed. This is the one place a future minor could silently break, and it is covered by keeping the latest row (currently 9.2.0) up to date. Please bump the latest row whenever a new AGP minor is released.run_acceptance_tests.bashstill listed unsupported AGP 4.2-7.x and stopped at 8.7, so it is synced to the same 4 pairs as the CI matrix.🤖 Generated with Claude Code