Skip to content

test: increase MV schema-notification wait budget in metadata tests - #1021

Open
mykaul wants to merge 2 commits into
scylladb:masterfrom
mykaul:fix/1020-mv-schema-notification-lag
Open

test: increase MV schema-notification wait budget in metadata tests#1021
mykaul wants to merge 2 commits into
scylladb:masterfrom
mykaul:fix/1020-mv-schema-notification-lag

Conversation

@mykaul

@mykaul mykaul commented Sep 13, 2026

Copy link
Copy Markdown

Problem

test_base_table_column_addition_mv and test_base_table_type_alter_mv (tests/integration/standard/test_metadata.py) intermittently fail in CI. Both already contain a self-documented workaround for a known issue: materialized-view schema-change notifications arrive as a separate control-connection event from the base-table schema response, so there's an inherent lag between a base-table ALTER completing and the MV's own metadata reflecting it.

The existing retry loops capped this wait at 10 x 0.2s = 2s total, which is apparently not always enough under CI load, causing the final assertion to fail intermittently. This surfaced most recently as a seemingly-unrelated CI failure on PR #786 — that PR's own change was confirmed correct; the failure was this pre-existing flake.

Fix

Replace both hand-rolled for i in range(10): ... time.sleep(.2) loops with the existing tests.util.wait_until(condition, delay, max_attempts) helper (already used elsewhere in this same file), raising the budget to 60 x 0.5s = 30s. This reuses the codebase's standard wait-for-condition primitive instead of duplicating another ad hoc loop, and gives the control connection realistic headroom to deliver the MV notification.

Also removes the now-unused import time left behind once the last time.sleep() calls in this file were removed.

Grepped the file for any other occurrence of this same fragile pattern — these were the only two.

Fixes: #1020

Testing

  • ast.parse / syntax check passes.
  • Full unit suite passes (805 passed, 90 skipped) — no regressions from the import removal.
  • These are integration tests requiring a live Cassandra/Scylla cluster, which isn't available in this sandbox; the diff was reviewed carefully instead:
    • Confirmed both view keys (alltimehigh, monthlyhigh) are already asserted present earlier in each test before the new wait_until calls, so the lambda's dict/key access can't raise before the condition is met.
    • Confirmed wait_until's contract (checks condition first, sleeps between attempts, raises on exhaustion) matches the intent, and mirrors existing usage elsewhere in the same file (line ~128).

🤖 Generated with Claude Code

Materialized-view schema-change notifications arrive as a separate
control-connection event from the base-table schema response, so there is
an inherent lag between a base-table ALTER completing and the MV's own
metadata reflecting it. The existing workaround in
test_base_table_column_addition_mv and test_base_table_type_alter_mv capped
this wait at 10 x 0.2s = 2s, which is too tight under CI load and flakes
intermittently.

Replace both hand-rolled retry loops with the existing tests.util.wait_until
helper, raising the budget to 60 x 0.5s = 30s. This reuses the codebase's
standard wait-for-condition primitive instead of duplicating another ad hoc
loop, and gives the control connection realistic headroom to deliver the MV
notification. Also drops the now-unused 'import time' left behind by
removing the last time.sleep() calls in this file.

Fixes: scylladb#1020

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 37 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: QUIET

Plan: Advanced

Run ID: 5d7fadde-25ed-4a79-9854-53971cbd71af

📥 Commits

Reviewing files that changed from the base of the PR and between 48ccc22 and 96da800.

📒 Files selected for processing (3)
  • tests/integration/standard/test_metadata.py
  • tests/unit/test_wait_until.py
  • tests/util.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@qodo-scylladb

qodo-scylladb Bot commented Sep 13, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. A final-poll notification still fails the metadata test ✓ Resolved 🐞 Bug ☼ Reliability
Description
wait_until raises whenever its attempt counter reaches max_attempts, even if the condition
becomes true on the poll immediately after the final sleep. If the materialized-view notification
arrives at that 30-second boundary, either updated metadata test reports a timeout instead of
accepting the already-refreshed schema.
Code

tests/integration/standard/test_metadata.py[R2391-2392]

+        wait_until(lambda: "fouls" in self.cluster.metadata.keyspaces[self.keyspace_name].views["alltimehigh"].columns,
+                   delay=.5, max_attempts=60)
Relevance

●●● Strong

Concrete polling boundary bug can miss success after the final sleep; aligns with accepted polling
reliability improvements.

PR-#774

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new calls rely on wait_until for the expanded polling window. The helper evaluates
condition() before the loop-counter check, increments the counter and sleeps when false, then
unconditionally raises when attempt >= max_attempts; consequently, a true condition observed after
the 60th sleep is discarded. Both changed call sites have the same control-flow behavior.

tests/util.py[23-38]
tests/integration/standard/test_metadata.py[2391-2392]
tests/integration/standard/test_metadata.py[2446-2447]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new metadata tests call `tests.util.wait_until` with `max_attempts=60`, but that helper raises whenever its counter reaches 60 even when the condition succeeds on the final poll after the last sleep. A notification arriving at the configured timeout boundary is therefore reported as a test failure.

## Fix Focus Areas
- tests/util.py[23-38]
- tests/integration/standard/test_metadata.py[2391-2392]
- tests/integration/standard/test_metadata.py[2446-2447]

## Recommended Fix
Correct `wait_until` so it returns immediately when the condition is true, including on the final poll, and raises only after a final false condition. Add or update unit coverage for success immediately after the last delay, then keep both metadata tests using the corrected helper.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Cross-repo context — repo relationships
  Explored: repo: scylladb/scylladb (sha: cf2633bf)
✅ REVIEW.md
Review mode: 🚀 Fast: This is a localized, low-risk test-only change that replaces two equivalent polling loops with an existing wait helper and removes an unused import.

Grey Divider

Tip of the day
💡 Did you know, you can start a comment with 'qodo' or '@qodo' to chat about any finding

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread tests/integration/standard/test_metadata.py
… poll

wait_until() checked condition() first, then looked at the *attempt* counter
to decide whether to raise, instead of the condition's own last result. When
the condition finally became true exactly on the poll after the last sleep
(attempt == max_attempts), the loop correctly exited, but the post-loop
'if attempt >= max_attempts: raise' fired anyway, reporting a false timeout.

This was latent before but surfaced by PR scylladb#1021 raising the metadata tests'
max_attempts from 10 to 60: a wider window makes hitting that exact boundary
far more likely under real CI load.

Cache each condition() call's own result and check that instead of the
attempt counter.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_base_table_column_addition_mv flakes: MV schema-notification lag exceeds retry budget

1 participant