MultiSet.consolidate merges keyed records whose keys or values differ only by number/string type

[KyleAMathews]

• I've validated the bug against the latest version of DB packages (origin/main at b5d92ceb)

Describe the bug

MultiSet.consolidate() merges keyed records whose keys or primitive values differ only in type. A number and a string with the same text, such as 1 and "1", become one identity. Multiplicities for two different records are then summed together. A retraction of one record can erase an insert of a different record.

The keyed path builds a string identity. getStringId in packages/db-ivm/src/utils.ts encodes every primitive as `str_${String(value)}`, and consolidate joins the key the same way:

getStringId(value: any): string {
  if (value === null) return `null`
  if (value === undefined) return `undefined`
  if (typeof value !== `object`) return `str_${String(value)}`
  return `obj_${this.getId(value)}`
}
// ...
const compositeKey = key + `|` + valueId

So 1 and "1" both produce str_1, and the keys 1 and "1" both produce 1|.

To Reproduce

import { MultiSet } from '@tanstack/db-ivm'

new MultiSet<any>([[[1, 'v'], 1], [['1', 'v'], 1]]).consolidate().getInner()
// [[[1, 'v'], 2]]   expected two records, each with multiplicity 1

new MultiSet<any>([[['k', 1], 1], [['k', '1'], -1]]).consolidate().getInner()
// []   expected [[['k', 1], 1], [['k', '1'], -1]]

new MultiSet<any>([[['k', [1, null]], 1], [['k', ['1', null]], -1]]).consolidate().getInner()
// []   expected both records (join tuples have the same collision)

The unkeyed path is correct:

new MultiSet<any>([[1, 1], ['1', -1]]).consolidate().getInner()
// [[1, 1], ['1', -1]]

Expected behavior

Keyed consolidation should keep the identity rule it documents. Keys compare by value, and values compare by reference. A number and a string are different values, so records that differ only in that type must stay separate.

Scope

• Object values are compared by reference and are not affected. The collision needs a primitive key or a primitive value, or a join tuple that holds a primitive.

• In @tanstack/db, the collision needs one of these:

  • one collection with numeric and string keys that print the same, such as 1 and "1";
  • a keyed stream that carries primitive values.

  I have not traced which query shapes produce such a stream.

• A code-weight change to MultiSet.consolidate is in progress. It keeps this behavior unchanged on purpose. Its consolidation oracle excludes these cross-type values from its grammar and links to this issue. A fix should remove that exclusion.

Suggested fix and test

• Include the type in the primitive identity, for example `${typeof value}:${String(value)}`. Also encode the key with its type before the | join.
• Remove the exclusion from the consolidation oracle, and add these cases as witnesses. Each case should fail before the fix and pass after it.
• Add a changeset for @tanstack/db-ivm.

[KyleAMathews]

The same keyed identity string has more collisions than the number/string case above. Each pair below consolidates to [] on origin/main (b5d92ceb). Both records should stay.

const out = (xs: any) => new MultiSet<any>(xs).consolidate().getInner()

// Keys of different types (listed above, repeated for completeness)
out([[[1, 'v'], 1], [['1', 'v'], -1]])                          // []

// The `|` delimiter is not escaped
out([[['a|str_x', 'y'], 1], [['a', 'x|str_y'], -1]])            // []
// Both encode as "a|str_x|str_y"

// Symbols, functions, and bigints are encoded by their text
const f1 = () => 1, f2 = () => 1
out([[['k', f1], 1], [['k', f2], -1]])                          // []
out([[['k', Symbol('s')], 1], [['k', Symbol('s')], -1]])        // []
out([[['k', 1n], 1], [['k', 1], -1]])                           // []

The cause is the same: getStringId gives `str_${String(value)}` to every non-object value, and consolidate joins the key and value ids with |.

A fix should close all of these together:

• Give non-object values an identity that includes their type, and give symbols and functions a reference identity (as objects have).
• Make the composite id injective. For example, prefix the key with its type and use a structured key or a length-prefixed encoding, not a bare | join.
• Add each pair above as a witness in the consolidation oracle, and remove the grammar exclusions that link to this issue.