Convert the wire format to pure CBOR - #38
Draft
plaidfinch wants to merge 43 commits into
Draft
Conversation
…sign doc The design doc now records today's rulings: the signal keeps its redundant stream component; listings are spelled as maps; the greeting map carries a protocol-magic entry; the observation hook is three-level (peer, session, directed stream) with no library-imposed cross-stream ordering; both consumers (the CBOR reflection snapshot extractor and the tracing adapter) are in scope; and the provisional atom-tag block is based at 0xD255 in the IANA first-come-first-served range, pinned in the new public tags module pending registration. Foundation commit for the CBOR-legible wire lane (rumors#35).
The owner-ruled, pre-release wire format change of rumors#35: every
directed stream of a V2 session now parses as an RFC 8742 CBOR sequence
under standard tag unwrapping, per design/cbor-legible-wire.md.
The spellings: frames are arrays carrying the dense signal code as a
uint item; query listings are {radix: hash} maps, whose deterministic
key order and the wire's canonical child order are now one rule; supply
runs and their records are embedded-sequence byte strings (tag 63) with
the version atom carrying its provisional tag; the greeting is one
embedded item (tag 24) wrapping a text-keyed map with a "protocol":
"rumors" magic entry; the preamble is a 30-byte self-described item,
55799(["rumors", version, network, intent]); the party hand-off is a
tagged byte string; stream labels are two uint items; the epilogue is
the text item ".". Decoders reject non-shortest-form and indefinite
heads, so the byte-pinning snapshot discipline stays meaningful. The
V1 wire, codec, and snapshots are untouched (the preamble and party
codecs branch on the selected dialect).
Re-derived pins, old -> new, each read from its instrument: the
calibrated dispute intercept 35 -> 43 B (the calibration cells; +3.9%
per disputed message at the design record); the default crossover
60 -> 52 B and the u64 BDP-scale window 65,404 -> 91,941 scopes (the
exact solve); DEFAULT_TARGET_MESSAGE_SIZE 1,638,912 -> 1,830,400 B
(const-evaluated from the new listing arithmetic, with a new
construct-and-encode adequacy pin); the disputed-reply transient
ceiling 3,380,000 -> 3,570,000 B (measured 3,468,800); the trade-off
table regenerated by its own renderer. The full wire snapshot corpus
re-accepts as this one deliberate format change; the renderer keeps
its hexdump vocabulary, decoding the new wire through the codec's own
functions. The stale "~4.6x" note in the window solve's assert message
(pre-existing drift: a figure that never matched the quoted prose) is
corrected by the re-pinned message it lives in.
The committed form of the legibility promise: randomized peer contents drive real gossip, bootstrap, and retire sessions through the recording link, and every directed stream must parse as a CBOR sequence under a generic ciborium walk — standard tag unwrapping only, zero bytes outside CBOR items, recursively through every embedded byte string. No rumors codec type appears in the walker, so a wire change that smuggles a non-CBOR byte onto any V2 stream fails here whatever the snapshots say.
A deliberate, owner-ruled pre-release format change (rumors#35): the
on-disk bookmark frame becomes one self-described CBOR item,
55799([ format_version: uint, integrity: bstr .size 32,
payload: 24(bstr .cbor map) ])
with each stored clock spelled as a clock-tagged byte string wrapping
its canonical encoding, so a generic CBOR tool unfolds the whole file
with no rumors knowledge. Both format pins move under the format-change
re-accept class; BOOKMARK_FORMAT_VERSION bumps to 4 (now a u64).
The integrity hash covers the encoded format-version item and the
encoded payload item — every frame-array item except the integrity item
itself — matching the prior format's effective protection (corruption
of the fixed opening fails shape validation before the hash is
consulted). The single-byte-corruption sweep stays total, truncation
rejection is strengthened to every strict prefix, and a rumors-blind
parse test pins the generic-parseability promise.
Error surface is typed throughout: BadMagic carries a FrameDefect
naming the failed shape position, Decode carries a RecordDefect
(replacing stringified io::Errors), and both enums are public.
BOOKMARK_MAGIC is removed: a self-described CBOR file has no raw magic
analogue, and the integration-test helper now walks the frame
generically instead of stripping a transcribed header.
Pre-existing drift, split from the wire change's attribution: the ignore-gated probe transcribed a 28-byte dispute intercept (the shipped constant it validates against was 35 before the wire change), priced u64 records at 8 encoded bytes (a random u64's CBOR encoding is 9), and minted its "design record" as a 170-byte CBOR array rather than the byte-string record the design cell is denominated in. The probe now reads the intercept through the testing accessor and mints the same Bytes records the calibration cells use. Re-run readings, quoted at Peer::sync_memory_budget: at 8-26 MB budgets on the minimal and design corpora, observed sessions ran 1.35-1.96x the closed form's figure, and the form's slowdown reads ~2-3x low at ~10 MB budgets and ~1.4x low at ~26 MB. The previous 1.3-1.65x band was measured through the drifted denominators on the old wire, so the band's movement is jointly attributable to the denominator fixes and the wire change and is deliberately not decomposed further.
…contract, and the measured session-bytes check The bookmark section states the v4 file as built (integrity region, typed defect carriers, no raw magic); the committed contract names capture validity as a permanent external/internal two-property pair; the cost summary carries the one-off whole-session measurement at the metadata-dominated worst corner; the decision record gains the capture contract and sanity-check rulings.
The public `observe` module is the crate's capture surface: a peer-attached handler (`Peer::observe`, `Bootstrap::observe`) yields a per-session handler for every session the peer enters — gossip, bootstrap, and retire — which yields a per-directed-stream handler as each stream opens, which receives that stream's protocol messages as raw wire bytes, exactly one CBOR item per invocation. The hook is rumors-blind (no protocol type in its signature), imposes no cross-stream ordering (consumers reconstruct interleaving from their own session-scoped ordinal), never re-encodes (received items are captured off the transport), costs one branch per frame when unattached, and observes only the CBOR dialect (V1's frozen wire cannot honor the one-item contract). The role election is delivered by a dedicated notification when decided, since it does not exist at session start. Threading follows the stats recorder's shape: a cheap-clone session handle reaches the preamble, greeting, party hand-off, and epilogue at their call sites, and every data stream mints its own observer at open or claim. Attaching an observer to the Bootstrap builder retires its `Copy` (an `Arc` field cannot copy); the retry affordance survives as `Clone`. tests/observe.rs is the internal-capture half of the two-property capture contract: a proptest differential holding the hook's view byte-for-byte against the recording-link transport capture on arbitrary gossip sessions (bootstrap and retire covered by fixed pairings), the one-item property on every invocation, complementary role elections consistent with every stream's speaker, and a byte-identity property proving observation never changes the wire. No snapshot moves.
The observation-hook commit landed with a red gate, two documentation legs deep. First, five doc comments (the session ordinal's field doc and four test invariants in the internal-capture suite) ran their whole statement into the summary paragraph, past doclint's 220-rendered-character bound; each now leads with a one-sentence summary and carries the rest below the fold, unchanged in content. Second, the private-items rustdoc build rejects redundant explicit link targets, and `Peer::observe`'s prose spelled two links through `crate::` paths that already resolve bare in `peer.rs` scope; they are bare now.
The wire snapshot corpus now pins each captured session as fully unfolded CBOR value trees — extended-diagnostic-style notation with a rumors naming layer (signal semantics, listing children and digests, tagged-atom meanings, supply runs and records by name) — in place of annotated hexdumps. The re-accept of every V2 snapshot in this commit is the deliberate, owner-ruled renderer supplanting Finch commissioned for rumors#35 (decision record in design/cbor-legible-wire.md: the snapshot extractor consumer, 2026-08-19); it does not qualify under, and does not use, the hex-line-preservation class. The byte-pinning discipline survives on two legs, both stated in the renderer's module doc: the rendering is injective on wire bytes (the walk enforces the wire's canonical head grammar and shows every item's complete content, falling back to explicit exact hex wherever it cannot vouch for inversion — non-canonical heads, invalid UTF-8, embedded content that does not fill its byte string, nesting past the depth bound), and the harness holds the rendered items to the transport capture as a totality oracle: each stream's open label plus its concatenated observed items must reproduce the wire bytes exactly. Capture enters through the public observation hook — the instrument enters through the public door: the snapshot harness attaches a recording observer to each peer (bootstrap builders included) and renders what the hook delivers; the instrumented link stays solely as the oracle and as the V1 timelines' capture, which are untouched, as are the V1 wire snapshots and the transport-captured wire-legibility property. tools/digestshare speaks the new vocabulary (byte-count headers and annotated digest lines instead of hexdump arithmetic) and now fails loudly if the corpus stops yielding wire or digest bytes, instead of reporting a silent zero. Re-measured over the re-accepted corpus: 8,441 wire B, 1,656 digest B, 69 digests at 24 B (19.6%), recorded in the design doc. Pre-existing defect fixed in passing, attributed here: the corpus carried an orphaned snapshot (gossip_snapshot__both_redact_same_key) that no test regenerates — it still held pre-conversion wire bytes and silently survived the format change's re-accept, polluting the corpus denominator. Deleted; the corrected corpus figure above supersedes the one recorded at the format change.
The bookmark's FormatError variants say what the bytes are, not what stage tripped: BadMagic (a misnomer against a magic-less file) becomes NotABookmark, and Decode becomes Record, each keeping its typed defect carrier. The handshake's failure cases are enumerated typed and public: a structurally-wrong preamble field surfaces as PreambleMalformed carrying a PreambleDefect naming the field, and a peer hanging up mid-handshake surfaces as PreambleTruncated carrying the exact byte counts — both formerly collapsed into io::Error(InvalidData) and io::Error(UnexpectedEof). The disruption harness's honesty classifier admits the typed truncation as an injected-cut outcome, and the new seed it shook out is committed. Owner-ruled (rumors#35 follow-up rulings, 2026-08-19).
…C is protocol-v1 vocabulary Session numbering is the observer's own concern, counted inside its handler with its own synchronization, exactly like message interleaving: SessionInfo carries kind and protocol only, and the peer keeps no session counter. PROTOCOL_MAGIC is the legacy V1 preamble's marker and is exposed only under the protocol-v1 feature; the handshake's cross-dialect detection reads the same bytes through a crate-internal constant, since a V2 endpoint recognizes a legacy peer without ever speaking its dialect. Owner-ruled (rumors#35 follow-up rulings, 2026-08-19).
The capture renderer bounded one parse_node tree by MAX_DEPTH but re-parsed each embedded byte string (tag 24/63) at depth zero, so a chain of nested embedded byte strings recursed one Rust stack frame chain per level with no budget surviving the byte-string boundary: a deeply nested application payload aborted the harness with a stack overflow. Thread one depth budget through the whole render walk: render_node, render_tag, render_listing, render_embedded, and render_embedded_as carry the depth already consumed, structural descent increments it, and an embedded byte string's content re-parses at that depth instead of a fresh zero. Every render_* call site passes a depth no greater than the depth its node was parsed at (the invariant stated on render_node), so a node in hand always fits the remaining budget and only parse_node checks the bound; nesting past MAX_DEPTH lands on the existing explicit hex fallback, keeping the rendering injective. Two committed tests pin the property at ten times the bound, on both paths: a nested embedded chain rendered directly (render_item) and the same chain arriving as a supply record's payload in an encoded frame (render_frame). Both abort with SIGABRT before this change. No snapshot moves: the depth budget only decides where the fallback fires, and no committed corpus nests near the bound.
…nimal cell's residual The three calibration cells shared a +/-2 byte tolerance band whose rationale claimed it only absorbed integer-division adjacency; in fact it also absorbed the minimal cell's systematic 2-byte offset below the intercept, and with it any small-record per-message regression of up to 4 bytes (the minimal cell sat at the band's edge, so +1..+4 B moves all stayed green while the payload-dominated design cell stayed exact). Split the band into what each part actually is: - MINIMAL_CELL_RESIDUAL = 2 names the systematic offset: small records batch more densely, so their share of per-frame framing is smaller (mechanism stated at DISPUTE_OVERHEAD_BYTES). The minimal cell now expects intercept + payload - residual (50 B) exactly. - The tolerance dissolves entirely: the counters are deterministic (seeded corpus, in-memory link, pure byte tallies), all three cells sit exactly on their pins (50/107/215), and any slack would absorb real per-message movement, not measurement noise. All three cells assert exact equality, so a mean per-message cost change of one byte in either direction fails its cell. - The negative control now bounds the converged session's fixed overhead below one byte per message at calibration scale (measured 14560 B against the 2 * DIVERGENT = 16384 B bound), which is the premise that keeps the per-message division's truncation from ever shifting a cell's quotient; it also prints the measured figure. - window.rs's intercept doc points at the residual constant instead of a tolerance band. Mutation check performed by hand: perturbing the minimal cell's measured value by +1 fails the cell (51 vs pinned 50); reverted. The adversarial review's resolution for this finding specified keeping a +/-1 tolerance; that band contradicts its own acceptance criterion (a +1 B perturbation must fail, but sits inside +/-1) and the goal (a +1 B small-record regression must not pass), so the band resolves to exact pins per goal-over-mechanism.
decode_v2's intent-width arm is unreachable by width arithmetic (the fixed 30-byte preamble with validated version and network heads leaves exactly one byte for the intent item, whose one-byte head's value is at most 23): make it an expect instead of fabricating an IntentInvalid diagnostic carrying a byte the peer never sent. The two remaining defensive arms (PreambleDefect::NetworkTruncated, TrailingBytes) stay, each now stating its defensively-reachable-only status in the variant rustdoc and carrying a one-line width derivation at its construction site. Add the missing reachable-defect constructions as unit tests (Version by wrong major and by widened spelling, Network by wrong declared length), with an exemption note naming the two defensive variants an auditor will find unconstructed. Re-export HeadError through the codec module's export cluster and the crate's error module, so LeafRunError::Head's source type is publicly nameable and its rustdoc link renders live; the enum stays exhaustive (RFC 8949's head grammar is closed). A nameability pin in tests/decode_alloc.rs imports rumors::error::HeadError and matches NotShortest from outside the crate.
Determinism is the encoder's promise: shortest-form heads, definite lengths, one spelling per value, which is what keeps the byte-pinning snapshot discipline meaningful. Ingress validates structure and definite lengths everywhere, and every head the codec hand-parses additionally rejects non-shortest spellings — but a record's version atom and application payload, and the bookmark's embedded payload, are decoded by a general CBOR reader that does not re-judge spelling (the version atom's content canonicality is enforced by its own strict decoder). The module docs at the three claim sites now state that boundary instead of claiming spelling rejection the code does not perform. Two acceptance tests pin the boundary so it cannot drift silently in either direction: a widened version-bstr-head record decodes Ok through the run's record iterator, and a bookmark payload spelled as an indefinite-length map decodes Ok to the empty record. Flipping either to rejection is a deliberate contract change, not drift. Zero snapshot movement; no behavior change.
…uidance The stream-open label is two leading unsigned-int items, not a fixed two bytes: an epoch of 24 or more takes a wider head. The observe module doc now describes the label by its structure, and tests/observe.rs parses captured labels through the stream_label helper instead of hardcoding the two-byte shape, so the suite stays correct at any epoch. Observer::session's "called once per session" now points at the module docs' Protocol::V1 exclusion, matching the dialect contract already stated there. SessionKind is #[non_exhaustive]: it enumerates lifecycle operations the crate could plausibly grow, and the crate marks open diagnostic taxonomies non-exhaustive. No in-tree consumer matches on it exhaustively. The Link transport contract now carries the one operational sentence readers of the framing internals already had: reads are exact and item-granular, so wrap an unbuffered transport's read half in tokio::io::BufReader — caller-owned buffering outlives a session and is safe across session boundaries.
… no ordinal vestige
OverbatchedRun's declared field is the frame's charged wire size — its
run body plus the SUPPLY_FRAME_OVERHEAD envelope at its widest — which
may exceed the actual frame by the envelope's head slack (at most
4 bytes). The field doc now says so, and the error text reads
"charges {declared} wire bytes" instead of presenting the charge as an
occupancy fact. The error-atlas snapshot re-accept is exactly that
deliberate diagnostic-vocabulary rewording: the four display/source
lines carrying the phrase, nothing else — the atlas pins error
rendering, not wire bytes, and no wire byte moves.
The greeting's uint reader now returns a per-key diagnostic ("set_len
is not an unsigned int" and siblings) through its previously unused
parameter, replacing the shared "greeting size entry" string.
Bootstrap::observe's doc no longer speaks of session ordinals the hook
does not carry: an observer that numbers sessions counts the join as
the first session it sees.
The epilogue test doc claims the marker's bytes, not a byte count that
rotted once already.
Extend the typed-error taxonomy one layer below the preamble, so an application can programmatically distinguish a dead transport from a counterparty that sent malformed bytes at the identity hand-off and the greeting, without inspecting io::ErrorKind. - New public HandOffDefect enum (mirroring PreambleDefect's style), carried by the new Error::HandOffMalformed; a clean close before or inside the promised hand-off is the new Error::HandOffTruncated. Wired through the V2 receive path (receive_v2, read_head, decode_party); the frozen V1 dialect keeps its io::Error vocabulary. - ReadGreetingError::Decode now carries the typed GreetingError (public, with ListingIssue), surfaced through the mirror's HandshakeDecode variant as a typed source; GreetingError::Version carries the before decode error directly. - Error doc table gains the two rows; the disruption honesty classifier admits HandOffTruncated as an honest cut while malformed stays dishonest; the party and greeting suites re-pin every failure to its typed variant, with new pins for the non-byte-string, malformed-head, truncated-listing, and version-defect arms. No wire change: zero bytes move, all snapshots unchanged.
…5799 spellings to it TAG_SELF_DESCRIBED is consumed only by the test-gated capture renderer, so the constant now carries the same cfg gate; a bare default-feature 'cargo check -p rumors' is warning-free again. The two production writers that pin the tag's rendered bytes inside literals are each held to the constant by a committed pin test: prefix_matches_the_writers now spells the tag through the constant instead of a third literal, and a new bookmark pin test asserts the frame's opening bytes equal the head writer's rendering of it. The gate gains a bare-lib clippy leg (cargo clippy -p rumors --lib): an invocation that also builds test targets compiles the lib with cfg(test)-gated modules alive, so an item dead only in the shipped default-feature lib never surfaced there. Demonstrated live: with the constant left ungated, the --lib --tests leg passes while the bare-lib leg fails on the dead constant.
… and the three bookmark frame defects The greeting codec's shortest-form rejection and the bookmark frame's Integrity, PayloadTag, and PayloadByteString defects were covered only positionally (the corruption sweep rejects the bytes) but never asserted as their typed variants, so a misclassification could pass unseen. Four targeted constructions now name each variant in a matches!: - a greeting whose set_len value head is re-spelled in the widened 0x18 form parses to GreetingError::Head(HeadError::NotShortest); - a flipped integrity-item header byte is FrameDefect::Integrity, the offset computed from the frame's fixed shape rather than hardcoded; - a flipped payload tag byte is FrameDefect::PayloadTag, likewise computed; - a widened payload byte-string head, with the hash recomputed over the re-spelled covered region so only the spelling check can reject, is FrameDefect::PayloadByteString.
…resolve to a live generating test Modeled on the seed-liveness sweep: the corpus is the wire-format pin, and an orphaned .snap — its test renamed, moved, or retired — is a dead instrument that still looks like a byte-for-byte guarantee. The sweep reverses insta's path resolution for both layouts (tests/snapshots suite__name stems; src-side snapshots directories keyed by module path), requires the resolved source to contain the generating function or the explicit snapshot name, and convicts naming the orphan and the file searched. Nothing under a snapshots directory is skipped: a committed .snap.new fails loudly as unaccepted insta output, and so does any other stray file. Fixture tests commit the conviction paths (missing generator, deleted suite, pending output) alongside live duals, so the sweep itself cannot rot green.
…nd gate-wire the digestshare vocabulary check
The session corpus pinned only QueryEmpty on the wire (nonempty
listings rode only inside greetings), so the one frame form that lists
a disputed node's children had no session-level byte pin. The new
fixture shared_subtree_dispute_pins_a_nonempty_query stages two peers
holding the same two-child branch under one root radix, then adds a
third leaf on one side: the subtree hashes differ while neither side
is absent and no listing is empty, so the dispute answer lists
children — a Query frame carrying a nonempty {radix => digest} listing
(three children in the accepted snapshot). An in-test liveness floor
asserts the nonempty listing before the snapshot comparison, so the
fixture cannot silently degrade back to QueryEmpty under a future
corpus change.
This change adds the one new snapshot
(gossip_snapshot__shared_subtree_dispute_pins_a_nonempty_query.snap)
and re-accepts nothing: every existing snapshot is byte-identical.
The gate's lint tier gains a digestshare leg: the tool's exit code
already carries a liveness verdict (V2 captures with zero countable
wire bytes or zero digests mean the renderer vocabulary moved out from
under its patterns), so wiring it in makes that check fire on every
commit instead of only on interactive runs. The leg checks the
renderer-vocabulary contract, never a threshold on the measured
ratio.
The module doc claimed ingress validates definite lengths everywhere; the two positions a general CBOR reader decodes — a record's version atom (its byte-string head) and the application payload — accept indefinite spellings, refuted by construction. Rescope the sentence: structure is validated everywhere, the hand-parsed heads reject indefinite lengths and non-shortest spellings, and the general-reader positions judge neither spelling rule. Drop the parenthetical roster of hand-parsed positions rather than completing it (it omitted the preamble and greeting heads and would rot as positions move). Commit both indefinite-length acceptances as boundary pins beside the existing widened-head pin: an indefinite-length version-atom byte string and an indefinite-length payload map each decode Ok, so flipping either to rejection is a deliberate contract change, not drift.
The atlas's module doc claimed witnesses for every codec error, but the codec module also publishes the handshake layer's GreetingError and ListingIssue, neither inventoried here. Scope the claim to the frame-stream taxonomies the describe_* matches inventory, and hold the two handshake-layer types as reasoned EXEMPT_MARKERS entries naming greeting/tests.rs as their witness home, so a witness landing here is flagged for promotion. Add the one reachable collapse the atlas lacked: a wire query listing whose key is a well-formed head of the wrong kind reaches the listing gate and surfaces as Malformed(part=QueryChildren). The atlas snapshot gains exactly the two new listing-key witness blocks (one per speaker); every existing line is byte-identical. Three prose corrections in the same file: the FramePart marker comment no longer claims every part rides the encode Write witnesses (Signal renders only from decode-side witnesses); the snapshot test's doc no longer claims every witness pins an origin (the record-level witnesses carry none); and the coverage failure message now names both required edits for exempting a variant (the marker must also leave WITNESS_MARKERS).
Six testdoc corrections toward what the bodies actually pin: - The run-budget proptest's rejection is decided from at most the first record's heads, not before any body byte is read (the budget check reads those heads out of the body); its doc and the early-rejection comment now say so, and the unexercised flush-rule complement claim is gone (that shared boundary lives in the budget module). - The structural-decode proptest pins byte identity of the carried run; deferral of record decoding is pinned by the zero-length-record test, and the doc now points there instead of claiming laziness this body cannot distinguish. - The async-EOF truncation cases now decode through both decoders via decode_both, making the doc's sync-parity claim exercised rather than asserted; the clean close stays async-only and the doc says why. - The chunk-boundary truncation doc now names every seeded cut, including the zero-byte, one-byte, and one-short-of-total cuts. - The empty-listing doc no longer implies this test exercises the 256 ceiling; the new oversized_query_listing_is_rejected pins the ceiling in both decoders, decided on the map head alone. - The Stream constructor pin moves out of invalid_signals_are_rejected into its own documented test. Also: the untagged-version arm now asserts its deterministic InvalidData kind; the LeafRun records doc enumerates every content failure class (tag defects and general-reader version-item failures included); the observe module doc reflows an over-wide line; and the greeting shape diagnostic drops its hand-maintained entry count.
…uals The digestshare liveness check rode gate-lints but not the ci recipe, so GitHub CI (which runs just ci) never executed it and a PR skipping the local gate merged unchecked; add it to ci's dependency list. Two defensive-variant exemption notes, per the handshake suite's house pattern: party/tests.rs documents that HandOffDefect::UnaddressableLength has no construction (the u64-to-usize length conversion cannot fail on a 64-bit host; only a 32-bit target such as wasm32 reaches the arm, and the suite has no 32-bit test host), and greeting/tests.rs documents the same for the greeting's two unaddressable-length shape diagnostics. The snapshot-liveness module doc now states the containment rule's limits as the pairing design's accepted residual: substring matching keeps a renamed test's orphan green when a same-named function, string, or comment survives, and an assertion-less survivor function also passes.
…et path The zero- and one-byte cuts exercise the earliest truncation seams, but this test's declared run sits inside the default budget, so the over-budget head read never runs here and the filler body carries no record heads; the doc no longer claims either.
…de-side limit Payload nesting depth becomes a configurable setup value. A new public `PayloadDepthLimit` newtype (default `DEFAULT_PAYLOAD_DEPTH_LIMIT`, 256 scopes: exactly the bound ciborium's `from_reader` applied implicitly, so a fleet at the default sees no acceptance change on existing content) is selected by builder-style knobs on `Peer`, `Bootstrap`, and the `BookmarkedBootstrap` passthrough, and follows the peer through `into_rumors`, cloning, reunion, bookmarking, and retirement like every other setup value. The bare minted `PayloadDeserializer` a peer carried becomes a minted `PayloadCodec`: a small `Copy` struct pairing the minted fn pointer with the configured limit, threaded everywhere the deserializer traveled (peer config, both dialects' sessions, `Message::from_wire`). A plain fn pointer cannot capture a runtime value, so the limit rides as data beside it, and sessions stay non-generic. Decode-side enforcement: the minted deserializer's inner parse is now `ciborium::de::from_reader_with_recursion_limit` at the codec's limit. Parses that stay at the library default are each structurally flat: V1's outer byte-string unwrap, the version atoms, and the bookmark payload walk. The V1 wire freeze is byte-level and this change moves no byte on either dialect: only the local acceptance bound becomes configurable, symmetrically with V2. `Message::from_slice`/`from_bytes` take the limit as an explicit parameter: their bytes are the caller's trust domain, and an application on a raised fleet limit must be able to rehydrate its own stored deep messages, which an implicit default would refuse. The depth failure surfaces through their existing io::Result as invalid data. Tests: rehydration honors the explicit limit in both directions, and a hand-crafted over-deep supplied record dies typed at wire ingress while the same shape at exactly the limit decodes clean.
… scope The send side of the symmetric bound: `Message::try_new` serializes, runs an O(n) iterative depth scan over the produced bytes (the wire's head grammar with an explicit stack of remaining-child counts, so input-controlled depth never recurses), and returns a typed `PayloadDepthError` past the limit. A `Serialize`-impl failure keeps `Message`'s documented panic contract (programmer error); a depth violation is the typed error (the value's shape can carry end-user data). The scan's accounting is not transcribed into prose or constants: a committed differential proptest generates values nested to depths around the limit (arrays, maps, and tags mixed, the decoder's big-integer form included) and holds the scanner and `from_reader_with_recursion_limit::<ciborium::Value>` to the same accept/reject verdict at the limit and on either side of it. The minted codec gains its serializing half: a `PayloadSerializer` fn pointer minted beside the deserializer, so all of `T`'s serde bounds concentrate at `Peer` construction (`Serialize` joins `DeserializeOwned` there) and drop from `Rumors::send`/`Batch::send`. Every message creation in the peer's orbit now flows through the one codec value carrying the configured limit. `Rumors::batch` becomes a closure scope: `fn batch<R, E>(&self, f) -> Result<R, E>` runs `f` with an exclusive `&mut Batch` handle and commits everything queued, via `send_if_modified`, iff `f` returns `Ok`. The scope type's `Drop` impl is deleted (with the `thread::panicking()` guard); the lifecycle collapses to commit-on-Ok, all-or-nothing: a depth error propagated out cancels everything, a user `Err` cancels deliberately, a panic unwinds past the commit call, and the async-cancellation prefix-commit hazard is unrepresentable — a cancellation lands between polls and the synchronous closure runs inside one poll. `a_cancelled_batch_commits_its_prefix` is deleted with the hazard it pinned. Leak-proofing is static: the higher-ranked `for<'s>` bound with `R`/`E` quantified outside, no `Clone`/`Default`/ public constructor on the scope type, private fields, and two `compile_fail` doctests pinning the stash and return escape routes. Batches nest (building holds no lock; inner commits land before the outer batch), documented and tested. `Batch::send(&mut self, T) -> Result<(), PayloadDepthError>` mints through the codec; `Batch::redact(&mut self, &Version)` loses its chaining return. Single-action sugar stays on `Rumors`: fallible `send`, infallible `redact`, each committing immediately. The tutorial, doctests, and every in-tree caller migrate to the closure form or the sugar (tests share a `batch_send` helper); the READMEs re-derive. Tests: the batch lifecycle pins above; the boundary suite (tests/payload_depth.rs) round-trips a payload at exactly the default depth peer-to-peer, rejects one scope past it at send with the typed error, and gossips deep content clean across a fleet raised in concert; try_new's exact boundary and the codec's carried-limit serialization are pinned unit-wise.
…ality
The deliberate, owner-ruled pre-release wire format change: the V2
greeting map gains a `"payload_depth_limit"` entry (an unsigned int, the
sender's configured limit in scopes), recomputing the deterministic key
order — at 19 characters it ties `"target_message_size"` on length and
sorts before it bytewise — and widening `parse_greeting`'s exact roster
to the seven-entry map (the diagnostic stays count-free). The wire value
is authored by the proxy from the session's payload codec at send, so
the one configuration every parse already runs under is also the one
the wire declares.
A session proceeds only if the two exchanged limits are exactly equal.
The check runs in `complete_connect` and `accept` symmetrically, after
both greetings are in hand and before the equal-versions resolution, so
mixed configurations surface even on converged, no-op sessions; the
failure is the new typed `Error::PayloadDepthMismatch { local, remote }`
on both sides (documented in the error table: the limit is a fleet-wide
parameter; align it and reconnect). Equality rather than negotiation is
the ruling of record: negotiating down is unsound — a peer whose
session bound dropped below its own configured limit may already hold
messages deeper than the negotiated bound, content it would then not be
allowed to gossip — so negotiation merely relocates the failure to
mid-session, conditional on which leaves differ. The frozen V1 greeting
cannot carry the parameter, so V1 sessions keep decode-side-only
enforcement (the knob's rustdoc states the carve-out).
With this, the knob's achieved invariant holds and is documented: no V2
session between conforming peers can fail on payload depth at all —
over-deep values are rejected at their author at the moment of choice,
and mismatched fleets are rejected at hello.
Snapshot corpus re-accepted for exactly this change: every moved line
across the 21 re-accepted pins is the greeting item's byte count, its
embedded-item length, its map entry count, or the new
`"payload_depth_limit"` entry (verified mechanically against the parent
render). The dispute-wire calibration cells are unmoved — measured
215/107/50 B per message against pins 215/107/50: the 23-byte greeting
growth amortizes below the cells' integer quotients at 8,192 divergent
messages per side. `tools/digestshare` re-measured: 9,853 wire B,
1,776 digest B, 74 digests (18.0%); the design doc's corpus paragraph
is re-derived (its previous figures had drifted from the corpus before
this change — the parent commit measures 8,887 wire B — and are
re-measured with the attribution split stated in place). The wire
design doc gains the greeting-table row and the decision-record entry.
Tests: the handshake-equality pair — mismatched limits abort both sides
with the typed error, no role elected, no data stream opened, on a
converged pair (proving the check precedes the equal-versions
short-circuit) — beside the existing equal-raised-limits control that
gossips deep content clean.
`design/payload-depth-limit.md` records the package's design and its
decision record: the depth limit as a property of the shared set (hence
pairwise equality, never negotiation, with the negotiate-down
unsoundness argument); eager, fallible send with the panic/typed-error
split; a failed batch commits nothing; the closure scope as the
no-await mechanism, with the three rejected fiats (`!Send` misstates
the type, `#[must_not_suspend]` is unstable on the pinned toolchain,
clippy's `await-holding-invalid-types` binds only in-repo runs) and
the commit-on-`Ok` inversion; the minted codec with the
Serialize-at-construction cost; the 256 default's rationale; and the
V1 carve-out. Per the house rules the doc cites code and code cites
nothing back; the knob's rustdoc carries every invariant inline. The
wire design doc's decision entry gains the cross-pointer.
The crate-level "Message payloads and compatibility" section states the
depth bound (a payload admitted anywhere is transferable everywhere)
and points at the setting; the READMEs re-derive. One typo in the same
paragraph ("compabitility") is corrected in passing.
Two smaller adjustments ride along: a prose simplification pass over
this feature's new rustdoc and design prose,
and `Batch::commit` drops its empty-action short-circuit — `Tree::act`
documents an empty batch as a complete no-op and its false changed flag
suppresses the wakeup, so the branch guarded nothing.
Send-side admission judged a payload's depth with a byte-level structural scan, differentially tested against decoding the bytes as ciborium::Value. But the decode engine's recursion accounting is type-dependent — decoding a serde enum prices its variant scope, so a unit variant costs a step no scan of the bytes finds — and wire ingress decodes the user's T, not Value. An enum payload at exactly the limit (the crate docs' own recommended versioning shape) was therefore admitted at send and failed at every receiver's ingress: a deterministic gossip wedge between conforming peers at equal limits. Admission is now the ingress computation itself: Message::try_new runs the peer-minted deserializer — the same fn every receiver's wire ingress runs, same type, same limit, same engine — over the just-serialized bytes and discards the decoded value. The structural scan and its Value-oracle differential are deleted: with one computation there is no second accounting to hold in agreement. The send-side error becomes EncodeError (replacing PayloadDepthError everywhere, public re-export included), typed in two cases: Depth for the recursion bound, and Roundtrip for a payload type whose Deserialize rejects its own Serialize output — a violation that now surfaces at the author instead of at every receiver. Crate-internally the minted deserializer returns message::DecodeError, preserving the engine's RecursionLimitExceeded as a typed variant; wire ingress still folds to io::Error (DecodeLeafError::Message's surface unchanged), with the depth case's invalid-data message now naming the exceeded limit. Docs are restated to the new mechanism: the knob's no-failure invariant holds by construction within a decode-engine version; the wire-ingress bullet is scoped to content rather than byte shapes the engine consumes without recursing; the limit's unit is the engine's recursion steps; the rehydration constructors are documented as crate-internal (the message module is private, so no application could ever name them); crate docs and the derived README follow. Regression pins: the recursive-enum boundary round-trips peer-to-peer at the deepest admissible depth and rejects one step past it typed, at the author, in both the unit suite and the peer-to-peer suite; a round-trip-violating payload type fails admission typed; and a sender whose counterparty aborts mid-session on a decode failure and discards its link exits typed (Error::Epilogue, its local commit intact), never hanging.
…ulings; state the empty-batch contract in Tree::act's rustdoc ciborium is pinned to =0.2.2 in both manifests that name it. Depth admission and wire ingress run the same compiled deserializer, so the decode-recursion accounting is symmetric within a binary by construction; across a mixed-version fleet the no-failure invariant holds only if every build shares one accounting, and ciborium documents no accounting contract — the exact pin is what makes 'same engine' a property of the build. The pin's manifest comment carries the condensed rationale and the bump procedure (a bump is a deliberate, fleet-coordinated event: two-build verdict differential over a seeded corpus; verdict-changing bumps ship with a greeting accounting stamp; strictening bumps additionally re-validate stored payloads before gossiping). The lockfile already held 0.2.2, so resolution is a no-op. The payload-depth design document's mechanism section now describes admission as the receiver's exact decode, and its decision record gains four rulings: the Value-oracle differential refuted by construction (the engine's recursion accounting is type-dependent, so no byte-level oracle can equal the T-decode), with the knob's number engine-defined and the cross-version residual stated; the exact pin's rationale in full; the engine bump playbook (the negotiate-down unsoundness argument applied across time instead of across peers); and the vendor evaluation of record (cbor2 evaluated and rejected on supply-chain provenance, with successor criteria stated). Tree::act's rustdoc now carries the empty-batch contract (a complete no-op: nothing ticks, the flag is false) that batch.rs's commit path cites; it lived only in a body comment before.
Payload types now mandate Eq — table stakes for a wire message type —
and send-side admission, after running the receiver's decode over the
just-serialized bytes, downcasts the decoded value and requires it to
equal the value sent. Inequality is the new typed
EncodeError::Unfaithful ('the payload value's encoding decodes to a
different value'). The check is send-side only: ingress holds no
original to compare against.
Why: rumors exists to synchronize causal messages so a fleet can
replicate a causally-convergent state machine from the stream. A
message that decodes to anything other than what was meant violates
that premise — any state machine driven by consuming the stream could
diverge arbitrarily — so admission now guarantees at runtime that no
implementation can pollute the set with a value replicas would read
differently. The canonical lossy shape, a nested Option holding
Some(None) (serializes to CBOR null, decodes as None), is invisible to
any byte-level check: re-serializing the decoded None is
byte-identical, so the value space is finer than the byte space and
value equality is the only total instrument; the design doc's decision
record carries that argument and the rejected byte-fixpoint
alternative.
The T: Eq bound joins Serialize + DeserializeOwned at the codec mint
and sweeps through every construction-side signature (Peer, Bootstrap,
gossip's join mint); sessions stay non-generic. The Eq mandate (rather
than PartialEq) excludes NaN-capable float fields by design: equality
must be an equivalence relation for the check to be total and never
spurious.
The payload-type contract is now documented publicly, consolidated in a
crate-root 'Choosing a payload type' section: the bounds and what each
obligates — serializability with the PathBuf-class warning (previously
stated only on the crate-private Message type), faithfulness with the
Some(None) example, the float exclusion, and the depth-limit pointer.
Rumors::send/Batch::send panic sections and the tutorial point at that
contract instead of restating it; READMEs re-derived.
Test fixtures migrate off ciborium::Value as a payload type (Value has
no Eq: its Float variant): array spines become a nested newtype struct
whose decode prices identically (ciborium's newtype and transparent
paths do not recurse), the codec unit test's map spine becomes the
recursive enum, the ingress pin's hand-crafted bytes keep their depth
with an empty-array leaf, and the handshake/sender-exit pins use u64.
Value remains in decode-only rehydration tests, which carry no Eq
bound. The canonical faithfulness regression sends Some(None)
peer-level: rejected typed, nothing stored, faithful values of the
same type admitted.
The error module's public flattening carried the streaming adapter's
EncodeError/DecodeError under their bare names while the crate root's
EncodeError was the send-side admission error: two public EncodeErrors
whose meaning depended on the path. The unqualified name now goes to
the type applications touch, at both paths — the send-side EncodeError
is re-exported from rumors::error too, so the error module is the
complete one-stop taxonomy and agrees with the crate root — and the
adapter pair carries its scope in the public flattening, per the
module's own precedent for the codec pair (CodecEncodeError,
CodecDecodeError): ReplyEncodeError and ReplyDecodeError, named for
what they diagnose — the adapter's own docs state each type's domain
as one scoped protocol reply crossing the wire-frame boundary ('a
protocol reply could not be rendered faithfully as wire frames', 'wire
frames could not be reconstructed into one scoped protocol reply').
Module-internal declared names are unchanged, so no Debug rendering
moves and no snapshot is touched.
Crate-internally, message::DecodeError becomes PayloadDecodeError for
greppability; no public effect.
Cross-references follow: the OversizedVersion link in streaming's
message docs, the proxy declaration tests' public-path imports, and
the frame-fuzz design doc's taxonomy tour now name the flattened
public spellings.
Both documents shipped: the CBOR-legible wire and the payload-depth package are implemented on this branch, their invariants stated inline at the code they govern, so the documents move to the notes directory as provenance, byte-identical, with orientation READMEs resolving their mutual citation. The notes directory's LLM ground rules move from README.md into new AGENTS.md/CLAUDE.md files so agents working under the directory read the right instructions; README.md keeps the human-facing caveat.
plaidfinch
commented
Aug 20, 2026
The packet was the working spec for the fix-and-feature campaign: findings with executable resolutions, seed dispositions, owner rulings, and the payload-depth-limit implementation spec. The body is the document's final revision, byte-identical; the README orients a reader on its historical anchors (pre-resign SHAs, pre-rename vocabulary).
… calls PR #38 review directive: six sites flagged as way too verbose. Every stated invariant survives in fewer words; doc examples propagate errors with ? (examples that ?-propagate both EncodeError and Error return Box<dyn std::error::Error>, since Error has no From<EncodeError>). README re-derived from the crate docs via just readme.
…e; PR #38 doc fixes Review rulings from PR #38, the mechanical batch: - The word "mint" no longer appears in the rumors crate's code, prose, comments, or test names: PayloadCodec::mint is now PayloadCodec::new (the idiomatic constructor name), and every prose use is reworded to plain construction/creation vocabulary in context. One occurrence remains in tests/session_stats.proptest-regressions, a comment line in a committed seed artifact that stays byte-stable. - Every dependency resolves through [workspace.dependencies]: member manifests carry only workspace = true plus their own feature additions and optional flags. The exact ciborium pin and its rationale live once, at the root table. cargo tree --workspace --all-features -e all is byte-identical to the parent commit and Cargo.lock is untouched. - Bootstrap's builder-mirror fields are uniformly documented: protocol, window, and run_budget now carry the same selected-by comments as payload_depth_limit and observe. - The gossip Reconciliation codec doc states the codec's role in both wire directions, not ingress alone: ingress decodes through it, egress replays bytes it previously admitted, and its limit rides the greeting. - tree::mirror::cbor's module doc states why hand-written head primitives exist instead of a general CBOR reader: head-level incremental parsing against the transport, O(1) skip by declared length, and canonical- spelling enforcement at the hand-parsed positions. - The tutorial's examples propagate errors with ? (main returns Box<dyn Error>, since rumors::Error has no From<EncodeError>), and the send-never-errs digression is cut.
…the magic A deliberate pre-release wire format change, owner-ruled on PR #38: the V2 greeting map drops its "protocol" => "rumors" entry. The session's protocol magic is the preamble's opening ["rumors", 2, ...] array, so the greeting entry restated what every session had already checked; a reader holding nothing but the bytes still finds the magic in the preamble, one control item earlier. - The encoder, the key roster, and the exact-roster parse drop the entry; the roster diagnostic stays count-free. - The greeting-level wrong-magic test case dissolves: its subject is diagnosed at the preamble as Error::MagicMismatch, which the handshake suite pins with its own wrong-magic case. - The wire-capture snapshot corpus is re-accepted for this change. Audit: across all 21 changed snapshots, every diff line is the removed "protocol" entry, a control-item byte-count header, or an embedded-item length annotation — verified mechanically; nothing else moved. - tools/digestshare recomputes clean over the re-accepted corpus. - The dispute-wire cost cells are exact pins and pass unchanged: the greeting shrink is per-session, amortized below every per-message quotient.
…rk pins Two annotation-class snapshot re-accepts from PR #38 review; the wire and on-disk bytes are untouched in both. - The capture renderer's version-atom annotation now reads "causal version, event tree: <tree>": the rendered quantity is the version's whole ITC event tree in paper notation, never a scalar — a flat tree renders as its single uniform height (the corpus's "version 3" was that case), a forked one as the nested (n, e1, e2) form. The old word invited reading the flat case as a tally or a protocol version. Renderer-vocabulary re-accept witness: across all 21 changed capture snapshots, the h'..' hex token sequence is byte-identical to the parent commit and every diff line is the version-annotation rewording — verified mechanically. The capture unit tests assert the new vocabulary. - The bookmark format pins are now human-readable: each snapshot's first line is the frame's exact hex (the pin), followed by a decoded, annotated reading in the wire captures' idiom — format version, integrity digest sliced from the frame's own bytes (never recomputed), and the embedded record's networks and clocks. Witness: both pins' hex lines are byte-identical to the parent commit — frame_empty attests the on-disk format unchanged, and the round-trip/corruption suite passes unchanged. tools/digestshare recomputes clean over the corpus: its patterns bind the byte-count headers and digest annotations, which are untouched.
The renderer's annotation wording is the wire snapshots' to pin: a reviewer judges wording at re-accept, and a unit test duplicating the phrase forces a lockstep edit on every rewording while adding no second check. The test now asserts the load-bearing structure alone: exactly one rendered line moves, and it carries the version's exact rendering. Containment is made collision-proof by a punctuated (non-flat) fixture whose two renderings are length-preserving on the wire.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This also folds in some API changes: notably,
Batchis now proof against async cancellation, because you can't hold one across an await-point, and we now mandateEqforTand check atsend-time that it round-trips, preventing you from accidentally sending a message which would cause divergence between your own understanding of its value and someone else's (as well as guaranteeing that everyone can deserialize whatever you serialize). This guard costs some additional compute, but I deemed it worthwhile to convert "permanent, persistent split-brain" and "poison value which can be gossiped to anyone and kill them" into local-only failures.Closes #35.