Skip to content

ci: capture pytest exit codes and build test-status.json per label - #913

Open
CagriYonca wants to merge 1 commit into
mainfrom
currency-update
Open

CagriYonca wants to merge 1 commit into
mainfrom
currency-update

Conversation

@CagriYonca

@CagriYonca CagriYonca commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Why

When pytest failed in CI, the test job stopped right away, before coverage, pip freeze output and test metadata were saved to the workspace. Downstream jobs then had no artifacts on failed runs, so rupy-currency-reporter never got updated dependency or test-status data.

How

  • run-tests-with-coverage-report now saves the pytest exit code and lets the remaining steps run instead of stopping the job.
  • A new fail-if-tests-failed command runs last in every test job (when: always). It fails the job only after all artifacts and workspace files are saved, and it prints the pytest short summary.
  • Failed test module names are parsed from the pytest output (e.g. tests/frameworks/test_fastapi.py → fastapi) and saved per job as pytest_failed_<label>.txt.
  • capture-installed-versions now runs with when: always, and the autowrapt job also saves its pip freeze output.
  • update-currency-versions combines the failed modules from all jobs into a package-level test-status.json ({"<package>": "fail"}; packages that aren't listed passed). The file is published next to installed_*.txt.
  • final_job and update-currency-versions now use requires: <job>: terminal, so they run whether the test jobs pass or fail.

Verification

  • Checked that the exit codes, failed-module lists and pip freeze files are saved on both passing and failing runs.
  • Checked that the generated test-status.json works with rupy-currency-reporter (instana/rupy-currency-reporter#9).

@CagriYonca CagriYonca self-assigned this Oct 7, 2026
@CagriYonca CagriYonca added do NOT merge Depends on another PR or WIP ci/cd labels Oct 7, 2026
@CagriYonca
CagriYonca force-pushed the currency-update branch 7 times, most recently from 68fc532 to 0a145f1 Compare October 8, 2026 09:04
@CagriYonca CagriYonca removed the do NOT merge Depends on another PR or WIP label Oct 8, 2026
@CagriYonca
CagriYonca marked this pull request as ready for review October 8, 2026 11:27
@CagriYonca
CagriYonca requested a review from a team as a code owner October 8, 2026 11:27
@CagriYonca
CagriYonca force-pushed the currency-update branch 5 times, most recently from fbf25af to 1c31418 Compare October 12, 2026 12:03
Comment thread .circleci/config.yml
@CagriYonca
CagriYonca force-pushed the currency-update branch 2 times, most recently from 649d208 to ad98e78 Compare October 21, 2026 12:24
@CagriYonca
CagriYonca requested a review from GSVarsha October 22, 2026 08:40
Comment thread .circleci/config.yml
mkdir -p coverage_results
cp -R .coverage coverage_results/.coverage.${CIRCLE_BUILD_NUM}
cd coverage_results
exit 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please explain why is an exit 0 needed here? I can see that you're capturing the $TEST_EXIT_CODE safely.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In CircleCI, if a step exits with a non-zero code, subsequent steps like saving coverage files and pytest results are skipped.

Ending this step with exit 0 ensures that all coverage reports and artifact files (pip-freeze and test-status.json) are safely generated and saved to the workspace even when tests fail.

The job is still properly failed at the very end by the fail-if-tests-failed step using the captured $TEST_EXIT_CODE. This allows final_job (SonarQube) and update-currency-versions to run and report the failing packages to the currency reporter pipeline.

Comment thread .circleci/config.yml Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants