Trigger master upgrade from the election vote-tally, not bdb_open_int - #6093
Conversation
cb4f499 to
054d999
Compare
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Success ✓.
Regression testing: Success ✓.
The first 10 failing tests are:
bulkimport_simpleauth [setup failed]
simpleauth_password [setup failed]
blockaccess [setup failed]
simpleauth [setup failed]
sc_timepart_logicalsc_generated [db unavailable at finish]
sc_resume_logicalsc_generated **quarantined**
timepart_auth
sqllogfill_reset_gen
consumer_non_atomic_default_consumer_generated **quarantined**
auth_cleanup
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Success ✓.
Regression testing: Success ✓.
The first 10 failing tests are:
disttxn [setup failed]
bulkimport_simpleauth [setup failed]
simpleauth_password [setup failed]
blockaccess [setup failed]
simpleauth [setup failed]
timepart_auth
sqllogfill_reset_gen
sp_snapshot_generated
consumer_non_atomic_default_consumer_generated **quarantined**
auth_cleanup
054d999 to
f77ae71
Compare
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Success ✓.
Regression testing: Success ✓.
The first 10 failing tests are:
sc_redo [failed with core dumped]
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
f77ae71 to
8b9a80f
Compare
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: Success ✓.
The first 10 failing tests are:
sqllogfill_reset_gen
ssl_san
consumer_non_atomic_default_consumer_generated **quarantined**
ssl_set_cmd
ssl_prefer
ssl_dbname
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**
8b9a80f to
e30f2d5
Compare
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: Success ✓.
The first 10 failing tests are:
sc_resume_logicalsc_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Success ✓.
Regression testing: Success ✓.
The first 10 failing tests are:
sc_resume_logicalsc_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
ab1f278 to
d212e4e
Compare
emelialei88
changed the title
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: Success ✓.
The first 10 failing tests are:
ssl_san
consumer_non_atomic_default_consumer_generated **quarantined**
builtin_ruleset
ssl_set_cmd
ssl_prefer
ssl_dbname
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: Success ✓.
The first 10 failing tests are:
consumer_non_atomic_default_consumer_generated **quarantined**
builtin_ruleset
sc_downgrade [timeout] **quarantined**
On a contested cold-restart election the election could complete without triggering bdb_upgrade, so bdb_open_int's fallback rep_start ran with the stale recovered gen and the generation regressed (noresetgen: prev == new). Emit DB_REP_NEWMASTER on the first majority-crossing vote2 of the term, discriminated by REP_F_MASTERELECT rather than the vote count, and remove the bdb_open_int master-start rep_start fallback so the election path is the single upgrade source. Validated on a 3-node cluster: noresetgen gen strictly increasing across 10 restarts, sqllogfill_reset_gen still resets, no abort/DUPMASTER. Signed-off-by: Emelia Lei <wlei29@bloomberg.net>
d212e4e to
e307300
Compare
roborivers
left a comment
roborivers
left a comment
There was a problem hiding this comment.
Cbuild submission: Error ⚠.
Regression testing: Success ✓.
The first 10 failing tests are:
sc_resume_logicalsc_generated **quarantined**
cdb2jdbc
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
phys_rep_tiered [timeout]
phys_rep_tiered_nosource_generated [timeout]
truncatesc_offline_generated [timeout] **quarantined**
phys_rep_tiered_firstfile_generated [timeout]
reco-ddlk-sql [timeout] **quarantined**
3 participants
Why
On a contested cold-restart election the election could complete (a node wins) without ever triggering
bdb_upgrade.bdb_open_intthen force-upgraded the winner via its fallbackrep_start, using the stale recovered generation instead of the won one, so the committed generation regressed (noresetgen:prev == new).What
The root cause is in the berkdb vote tally. The winner's self-vote is tallied into
rep->votesfrom the(GEN_)VOTE1path but never emitsDB_REP_NEWMASTER. The(GEN_)VOTE2path decided whether to emitNEWMASTERfrom the vote count (votes > nsites/2 + 1), so once the self-vote had pushed the count past the bare-majority slot, the peer vote2 that completed the majority returnedDB_HAS_MAJORITYinstead andbdb_upgradenever fired.DB_REP_NEWMASTERon the first majority-crossing vote2 of the term, discriminated byREP_F_MASTERELECT(set at elect, cleared at upgrade) instead of the fragile count.NEWMASTERthere wouldabort()in the dispatch.bdb_open_intmaster-startrep_startfallback entirely; the election path is now the single upgrade source.Risk
berkdb election/upgrade path. Validated on a 3-node cluster:
noresetgengen strictly increasing across 10 restarts (7→33, master rotating across all nodes);sqllogfill_reset_genstill resets; server logs show oneNEWMASTERper election, noabort()/DUPMASTER/ gen regression.