Skip to content

fix(core): retry cancelled asynchronous actions after resume - #1032

Open
linhongyu510 wants to merge 1 commit into
apache:mainfrom
linhongyu510:fix/cancelled-async-step-persistence
Open

linhongyu510 wants to merge 1 commit into
apache:mainfrom
linhongyu510:fix/cancelled-async-step-persistence

Conversation

@linhongyu510

Copy link
Copy Markdown

Cancelling an ordinary asynchronous action currently records its step as completed, even though its state update never ran. Rebuilding the application with resume_at_next_action=True then skips that action and advances to its successor. For example, cancelling work while it waits for a response resumes at after with the old state.

Changes

  • Preserve the original asyncio.CancelledError for _astep's post_run_step hooks before re-raising it. Persisters consequently record the step as failed and resume at the interrupted action.
  • Cover astep and arun, with both function-based single-step actions and class-based multi-step actions. The tests cancel at an event-controlled await, rebuild from persisted state, retry, and verify the final result.

How I tested this

  • On upstream 36e3ff3048c8cabd7aac6d5c373307934bf3c2df, all four new cases fail because the next action after recovery is after rather than work.
  • With the fix, pytest tests/core/test_application.py tests/core/test_persistence.py tests/core/test_parallelism.py -q: 256 passed on Python 3.12.13.
  • Black 23.11.0, isort 5.12.0, Flake8 6.1.0, ASF license-header checks, and git diff --check pass.
  • Complete pre-commit run --files burr/core/application.py tests/core/test_application.py: passed, including the frontend lint-staged and ASF header hooks.

Notes

This concerns non-streaming step persistence. The streaming cleanup change in #1027 is already in the base. The top-level execute-hook reporting in #897 does not update _astep's step hooks or persistence status.

The existing cancellation still propagates to the caller. This does not shield persistence from repeated cancellation or change the handling of cancellation during a lifecycle hook.

This change was developed with an AI assistant. The reproduction and regression checks use no model or network service.

Checklist

  • PR has an informative and human-readable title.
  • Changes are limited to a single goal.
  • Code passed the complete pre-commit check.
  • The behavior change is tested.
  • No new public functions or API are introduced.
  • No placeholder code is added.
  • The change corrects existing retry behavior without changing documented APIs.

@github-actions github-actions Bot added the area/core Application, State, Graph, Actions label Nov 6, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core Application, State, Graph, Actions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant