Skip to content

feat(flow): add paginated read-only sqlite state history - #8253

Open
shenjm (Jiaming07) wants to merge 9 commits into
crewAIInc:mainfrom
Jiaming07:feat/flow-state-history
Open

shenjm (Jiaming07) wants to merge 9 commits into
crewAIInc:mainfrom
Jiaming07:feat/flow-state-history

Conversation

@Jiaming07

@Jiaming07 shenjm (Jiaming07) commented Oct 18, 2026 •

Copy link
Copy Markdown

Closes #8138

Summary

SQLiteFlowPersistence retains every saved snapshot in flow_states, but the public API only exposed the latest one via load_state. This PR adds a bounded, paginated history listing of snapshot metadata (row id, method name, timestamp) that never loads state payloads.

  • FlowStateHistoryRecord (in base.py): frozen typed record with id, method_name, timestamp. id is the storage row id, not state.id.
  • FlowPersistence.list_state_history(): new non-abstract base method that raises NotImplementedError by default, so existing custom backends are unaffected.
  • SQLiteFlowPersistence.list_state_history(): bound parameters, ORDER BY id ASC, validated limit (1–MAX_HISTORY_LIMIT=1000) and offset (≥0), same connection handling as load_state.
  • SQLiteFlowPersistence.read_history(db_path, flow_uuid, *, limit, offset): staticmethod for inspection tools such as the CLI. It opens the database with mode=ro without calling __init__, so it never creates the file or tables and never changes the journal mode. Returns [] when the file, table, or flow does not exist; any other SQLite error propagates.
  • Exported FlowStateHistoryRecord, DEFAULT_HISTORY_LIMIT, MAX_HISTORY_LIMIT from crewai.flow.persistence.
  • Docs: new "Inspecting State History" section in docs/edge/{en,ar,ko,pt-BR}/concepts/flows.mdx.

Writes, load_state, pending feedback, and resume code paths are untouched (the only changed lines in existing methods are import statements).

Design notes

  • Reading a WAL-mode database read-only may create -wal/-shm sidecars. If they cannot be created (e.g. the directory is not writable), sqlite3.OperationalError is raised rather than falling back to an immutable read that could miss rows still in the WAL.
  • Timestamps written by older releases were naive UTC; they are normalised to aware UTC so records from one database always compare.

Tests

lib/crewai/tests/test_flow_state_history.py (27 tests, no API calls): multiple flow UUIDs, empty histories (unknown uuid / missing table / missing file), pagination, equal timestamps ordered by id, no state_json read, limit/offset validation, rows still in an un-checkpointed WAL are returned, read-only file, unwritable directory raises, naive-timestamp normalisation, base class default, and unchanged load_state/pending-feedback behaviour.

Quality checks run locally: ruff check, ruff format --check, mypy lib/crewai/src/crewai/flow/persistence/ (no errors in flow/persistence), and the flow persistence test suites.

🤖 Generated with Claude Code

Expose bounded history metadata (row id, method name, timestamp) for a
flow UUID from SQLiteFlowPersistence without loading state payloads.
Adds a module-level read-only reader for inspection tools that avoids
schema and journal initialization side effects.

Closes crewAIInc#8138

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@strix-security

strix-security Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 241 commits after the last Strix review (c4a3347). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

No security issues found.

Review summary

Reviewed all 8 changed files (4 docs, 3 source, 1 test). The PR introduces a bounded, paginated, read-only state-history listing for SQLiteFlowPersistence. The new SQL query uses bound parameters throughout, pagination inputs are strictly validated, and the read-only inspection path carefully avoids creating files/tables or silently truncating history. No injection, path-traversal, secret, cryptographic, or authorization issues were identified in the changed code.

Updated for c4a3347.


Reviewed by Strix
Re-run review · Configure security review settings

@coderabbitai

coderabbitai Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: bb6d2b23-d12e-4968-90f2-50f37f2ef330
📥 Commits

Reviewing files that changed from the base of the PR and between b86361d and 3706850.

📒 Files selected for processing: 3
  • lib/crewai/src/crewai/flow/persistence/__init__.py
  • lib/crewai/src/crewai/flow/persistence/base.py
  • lib/crewai/src/crewai/flow/persistence/sqlite.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

📝 Walkthrough

The persistence API now supports paginated flow snapshot history. SQLite provides instance-based listing and a standalone read-only reader that returns snapshot metadata. The change also adds tests and documentation in four languages.

Changes: Flow State History

Layer / File(s) Summary
History API and documentation
lib/crewai/src/crewai/flow/persistence/base.py, lib/crewai/src/crewai/flow/persistence/__init__.py, docs/edge/*/concepts/flows.mdx
Adds the immutable FlowStateHistoryRecord model, pagination limits, an optional base list_state_history method, and package exports. Documents pagination, record IDs, and read-only inspection in English, Arabic, Korean, and Brazilian Portuguese.
SQLite history queries and validation
lib/crewai/src/crewai/flow/persistence/sqlite.py, lib/crewai/tests/test_flow_state_history.py
Adds validated pagination and SQLite history readers that return metadata in ascending row-ID order. The standalone reader opens the database read-only and returns an empty list when the database file, table, or flow is missing. Tests cover pagination, timestamps, read-only access, errors, and unchanged saved state and pending feedback.

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant read_flow_state_history
  participant SQLite
  Caller->>read_flow_state_history: Request history with database path and pagination
  read_flow_state_history->>SQLite: Open database read-only and query metadata
  SQLite-->>read_flow_state_history: Return matching history records
  read_flow_state_history-->>Caller: Return records or an empty list
Loading

Suggested reviewers: joaomdmoura

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to 37068

The new read-only history reader can return an incomplete page, or fail, when it uses its fallback mode while a flow is writing to the same database. This affects only inspection output; stored flow state and normal persistence behavior are unchanged. The change is mergeable with owner awareness of this edge case.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 28.21% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 39 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: adding paginated, read-only SQLite flow state history.
Description check Passed The description links issue #8138, explains the implementation and rationale, documents verification and test results, and includes additional design context. It satisfies the required template.
Linked Issues check Passed Issue #8138 requires bounded SQLite history metadata, typed records, payload exclusion, bound SQL parameters, deterministic ID ordering, validated pagination, tests, custom-backend compatibility, unch…
Out of Scope Changes check Passed The changes stay within issue #8138. The APIs, exports, SQLite implementation, tests, and translated docs/edge/ sections directly support state-history inspection. No unrelated behavior change is es…
Full details: Linked Issues check

Explanation

Issue #8138 requires bounded SQLite history metadata, typed records, payload exclusion, bound SQL parameters, deterministic ID ordering, validated pagination, tests, custom-backend compatibility, unchanged existing paths, and a read-only inspection path. The changed code adds FlowStateHistoryRecord, a non-abstract base method, paginated metadata-only SQLite queries, and read_flow_state_history. The reported tests cover UUID isolation, empty histories, pagination, equal timestamps, read-only behavior, compatibility, and unchanged load and pending-feedback behavior. The cohort allocation request is coordination guidance, not a coding requirement.

Full details: Out of Scope Changes check

Explanation

The changes stay within issue #8138. The APIs, exports, SQLite implementation, tests, and translated docs/edge/ sections directly support state-history inspection. No unrelated behavior change is established.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Jiaming07

Copy link
Copy Markdown
Author

Lorenze Jay (@lorenzejay) Per CONTRIBUTING.md this PR should carry the llm-generated label, but I don't have permission to add it. Could you apply it? Thanks!

shenjm (Jiaming07) and others added 3 commits October 18, 2026 21:51
Do not fall back to immutable=1 when a hot rollback journal exists, since
reading the main file would surface uncommitted rows. Bound offset to the
SQLite integer range so an out-of-range value raises ValueError instead of
OverflowError.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ory reader

Note in the docs that reading a WAL-mode database may create or leave
behind -wal/-shm sidecar files, and state in the reader docstring that an
immutable read requires the database to stay unchanged for the whole read
with no protection against a concurrent writer. Also scope the
frozen-datetime monkeypatch in the equal-timestamps test with
monkeypatch.context() instead of undo().

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
shenjm (Jiaming07) and others added 2 commits October 19, 2026 15:01
Resolve conflict in flow/persistence/sqlite.py: keep both
list_state_history (this branch) and load_state_from_history (crewAIInc#8272).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 02fc530.

Approve preview

@Vidit-Ostwal

Copy link
Copy Markdown
Member

Review: simplify and keep the history API on the persistence class

Nice work on the feature. I'd like to cut it back, and put the behavior in the class so another backend can reuse it.

1. Drop the immutable=1 fallback.
It needs sidecar-file checks and string-matching on error messages. It also has a documented race with concurrent writers. It only helps for a WAL database in an unwritable directory. Letting the mode=ro error propagate is simpler and safer.

2. Move the read-only path into the class.
Today read_flow_state_history and _fetch_history_uri sit outside SQLiteFlowPersistence. If we add another backend (Postgres, etc.), the pattern doesn't carry over. Prefer:

  • FlowPersistence (base): a non-abstract list_state_history(flow_uuid, *, limit, offset) that raises NotImplementedError by default. That keeps custom implementations working, per the acceptance criteria.
  • SQLiteFlowPersistence: implement it, plus a @staticmethod/classmethod read_history(db_path, ...) that opens mode=ro without calling __init__. That gives the CLI its side-effect-free path.

The history API then lives in one place per backend, with the same signature.

3. Remove extra helpers.
_fetch_history_uri is a one-use wrapper. _HISTORY_QUERY and _fetch_history can be inlined or kept as a single private helper.

4. Scope.
load_state_from_history (#8272) isn't part of #8138's criteria. It could go in a separate PR.

Keep: _validate_page, bound params, ORDER BY id, and the metadata-only SELECT.

Address review feedback on crewAIInc#8253:
- Drop the immutable=1 fallback; mode=ro errors now propagate.
- Replace module-level read_flow_state_history with the
  SQLiteFlowPersistence.read_history staticmethod, which opens the
  database read-only without calling __init__.
- Inline _HISTORY_QUERY and remove _fetch_history_uri.
- Add a test that rows still in an un-checkpointed WAL are returned.
- Update the flow docs in en, ar, ko and pt-BR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jiaming07

Copy link
Copy Markdown
Author

Thanks for the review! Addressed all points in 02fc530:

  1. Dropped the immutable=1 fallback — mode=ro errors now propagate.
  2. Moved the read-only path to SQLiteFlowPersistence.read_history(db_path, flow_uuid, *, limit, offset) (staticmethod, no __init__, so no files/tables are created). The base list_state_history stays non-abstract and raises NotImplementedError.
  3. Removed _fetch_history_uri and _HISTORY_QUERY; _fetch_history is the single private helper.
  4. load_state_from_history (feat(flow): add exact historical state lookup by row id #8272) already landed on main separately, so it isn't part of this PR's diff.

Kept _validate_page, bound params, ORDER BY id, and the metadata-only SELECT. I also added a test that rows still in an un-checkpointed WAL are returned by read_history, since the fallback-specific tests were removed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Add paginated read-only SQLite Flow state history

2 participants