fix(replay): don't execute the replay target height as a checkpoint round

[mraszyk]

deliver_batches() derived requires_full_state_hash partly from its max_batch_height_to_deliver argument, so the last batch of a bounded delivery was always flagged as requiring a full state hash:

let persist_batch = Some(height) == max_batch_height_to_deliver;
let requires_full_state_hash = block.payload.is_summary() || persist_batch;

That flag does not only decide whether a checkpoint is written: it also selects ExecutionRoundType::CheckpointRound, which changes execution. A checkpoint round charges every canister for resource allocation and usage, bypassing the CHARGE_INTERVAL_ROUNDS gate, and aborts all paused executions instead of only those above a limit.

Only ic-replay passes Some(..) here, and when replaying a consensus pool it always does -- even without --replay-until-height, it passes Some(finalized_height). So the last replayed height was executed differently from the way the subnet executed that very same height, and the resulting state differed in the canisters' cycle balances and consumed cycles.

That difference used to be invisible to the certified state. If the current certification version was bumped to V29, /subnet/<subnet_id>/metrics includes CanisterStates::total_consumed_cycles(), so it changes the certification hash, and ic-replay reports

Hash mismatch! State divergence detected for outstanding shares!

against the subnet's certification shares at that height, refusing to proceed without manual inspection. Subnet recoveries replay to the highest certification share height, which is essentially never a summary height, so every recovery is affected.

Derive requires_full_state_hash from the block alone, and have ic-replay create the checkpoint it needs by delivering an extra batch at the end, one height above the last replayed block. The replayed heights are then executed exactly as the subnet executed them, and the checkpoint round happens at a height no node ever certified. A unit test in batch_delivery.rs pins this down: the batch at the delivery bound must not be a checkpoint round.

A new BatchContent::Checkpointing batch carries that extra round. It executes nothing: it only aborts all paused executions and resets heap_delta_estimate and expected_compiled_wasms in SystemMetadata, analogously to a round after subnet splitting. Like the splitting round, it does still advance batch_time (to one nanosecond past the last replayed block) and refresh network_topology / own_subnet_info, so the checkpointed state is not byte-identical to the state at the last replayed height -- but no canister is executed and no message is inducted, timed out or routed.

The extra batch is delivered whenever a consensus pool was replayed, with two exceptions:

• No consensus pool: no batches were replayed, so the on-disk checkpoint is untouched and there is nothing to persist. Delivering a batch anyway would mutate the state based on the wall clock time, producing a non-deterministic state hash.
• The latest state height is already above the replay target height: such a state can only come from the extra batch of a previous invocation over the same data directory, i.e. the checkpoint to persist already exists. Delivering another extra batch would move the checkpoint (and thereby the state hash) one height further on every re-run, making the hash depend on how many times the replay was run rather than only on the replayed data.

blockmaker_metrics becomes Option<BlockmakerMetrics>, None for the extra batches, which do not correspond to any block. This removes a second source of divergence: the extra batch used to observe BlockmakerMetrics::new_for_test() -- crediting node 0 -- into SystemMetadata::blockmaker_metrics_time_series, and to bump blocks_proposed_total.

restore-from-backup delivers no extra batches. With --replay-until-height at a height that is not a CUP height, it therefore no longer creates a checkpoint at that height and makes no persistent progress beyond the latest CUP at or below it. The --replay-until-height prompt is restricted to this subcommand and reworded accordingly (the consensus pool replay needs no warning any more), and the restore now prints which checkpoint the reported state corresponds to.

The replayed height is one above the subnet's; account for it in ValidateReplayStep, which already models this via extra_batches: 1 for the app subnet and NNS failover recoveries, and 1 + upgrade for the NNS same-nodes recovery, which delivers a further extra batch to update the registry local store.

ReplayStep now fails loudly when the work directory contains no consensus pool. Without one, ic-replay silently replays no blocks and creates no checkpoint at all, so a missing pool means the state was downloaded incorrectly.

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to a6c4a15.

Security Overview

• 🔎 Scanned files: 15 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/f5e9d2bb-29a4-4b64-902a-391d65fd8b2d?scanId=04a14943-d719-41fe-ab54-f0cbe9ed2e78&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type

Relevant files

Enhancement

► rs/consensus/src/consensus/batch_delivery.rs     Derive requires_full_state_hash from block payload only; adjust logic and related tests ► rs/consensus/src/consensus/batch_delivery.rs     Use Some(blockmaker_metrics) in blockmaker metrics setup ► rs/messaging/src/message_routing.rs     Handle optional blockmaker_metrics and record metrics only when present ► rs/messaging/src/state_machine.rs     Add handling for BatchContent::Checkpointing in state machine ► rs/messaging/src/state_machine/tests.rs     Update tests to account for Checkpointing and optional metrics ► rs/recovery/src/app_subnet_recovery.rs     Adjust replay validation step height to account for extra batch ► rs/recovery/src/nns_recovery_failover_nodes.rs     Adjust replay validation step height to account for extra batch ► rs/recovery/src/nns_recovery_same_nodes.rs     Update replay validation step to include extra batch behavior ► rs/recovery/src/steps.rs     Fail when no consensus pool exists; ensure proper checkpoint handling ► rs/replay/src/lib.rs     Commentary updates about extra batch and CUP-related checkpoint creation ► rs/replay/src/player.rs     Introduce target_height handling; ensure extra batch yields checkpointing batch when appropriate ► rs/types/types/src/batch.rs     Batch supports Optional blockmaker_metrics and new Checkpointing content; update related logic ► rs/test_utilities/types/src/batch/batch_builder.rs     Default builder now uses Some(BlockmakerMetrics) for batch creation

[pierugo-dfinity]

In ic-replay, instead of predicting the operator's thought process and behave as what we think makes more sense today (i.e. execute an extra checkpointing batch, but actually not if there's no consensus pool, and actually not if we replayed beforehand), what about leaving the decision of adding an extra checkpointing batch as a CLI argument?
By default, we wouldn't include an extra batch and thus not create a checkpoint. The operator could safely replay as many times as they want without "committing" to/persisting anything. When they want to commit, they would pass the flag.
I think it would simplify the ic-replay implementation

During recoveries, I guess we can always enable that flag, even when no consensus pool was downloaded (maybe we weren't able to SSH in? See next comment), still displaying a confirmation prompt maybe.

[pierugo-dfinity]

Suggested change

	let node_ids = [node_test_id(0)];
	let record = SubnetRecordBuilder::from(&node_ids)
	.with_dkg_interval_length(dkg_interval_length)
	.build();
	let subnet_id = subnet_test_id(0);
	let Dependencies {
	registry, mut pool, ..
	} = DependenciesBuilder::single_subnet(pool_config, subnet_id, vec![(1, record)])
	.build();
	let Dependencies {
	registry, mut pool, replica_config, ..
	} = DependenciesBuilder::new(pool_config, 1)
	.with_dkg_interval_length(dkg_interval_length)
	.build();

and use replica_config.subnet_id below

[pierugo-dfinity]

I'd also fire unimplemented! as we probably should not expect this variant here for the moment

[pierugo-dfinity]

I'd probably leave those arguments unchanged (to not make these files bear with the implementation detail of ic-replay) and instead add 1 to the argument in get_validate_replay_step

[pierugo-dfinity]

Suggested change

	"No consensus pool found at {}",
	"No consensus pool found at {}. You should execute the `DownloadConsenusPool` step prior to `ICReplay`.",

[pierugo-dfinity]

I guess I'm still a bit reluctant to have to download a consensus pool before replaying. In the case of NNS Recovery where we'd like to execute an extra batch, maybe we wouldn't have downloaded a consensus pool (because the orchestrators are broken and we cannot SSH in) and we'd want to execute the extra batch on top of whatever latest state we have locally.

[pierugo-dfinity]

style nit: This isn't obvious that target_height being a Some is equivalent to having a consensus pool without reading what's above (which could change and thus become inconsistent)

[pierugo-dfinity]

Suggested change

	// batch anyway would mutate the state based on the wall clock time from
	// batch anyway would mutate the state based on the latest registry version and wall clock time from

[pierugo-dfinity]

I think this made sense before because when delivering a "proper" extra batch, it should indeed increase its batch time. But now, when delivering the a Checkpointing batch, this time is artificially increased, which mutates the system metadata, even though the subnet never did so. I think we could keep it the same as the previous batch and we would ignore it in the DSM implementation. Not sure if having a non-increasing batch time could have other undesirable consequences though.

[pierugo-dfinity]

When do we expect this condition to be true?