Skip to content

Fix DeepDiff on dict subclasses with non-standard iteration - #674

Open
tony1233200 wants to merge 1 commit into
qlustered:devfrom
tony1233200:fix/569-dict-subclass-iter-keys
Open

tony1233200 wants to merge 1 commit into
qlustered:devfrom
tony1233200:fix/569-dict-subclass-iter-keys

Conversation

@tony1233200

Copy link
Copy Markdown

Fix DeepDiff on dict subclasses with non-standard iteration

Fixes #569.

Problem

DeepDiff did not detect differences inside google.cloud.spanner_v1.data_types.JsonObject:

from google.cloud.spanner_v1.data_types import JsonObject
from deepdiff import DeepDiff

DeepDiff(JsonObject([]), JsonObject([{"size": "L", "color": "red"}]))
# on 9.1.0: {}
# on current dev: TypeError: unhashable type: 'dict'

Root cause

JsonObject is a dict subclass, but for the JSON-array case it stores the elements in a separate _array_value list and overrides __iter__ to yield the array values instead of keys:

def __iter__(self):
    if self._is_array:
        return iter(self._array_value)  # values, not keys

_diff_dict() collected keys with for key in t1 / for key in t2, so it tried to build a key set out of the (unhashable) array elements and crashed. For the scalar case (JsonObject(5)) __iter__ raises TypeError outright, and for a scalar the object exposes no keys at all, so even a non-crashing path silently returned {}.

Change

deepdiff/diff.py, _diff_dict() only:

  • Key collection now iterates t1.keys() / t2.keys() instead of the dicts directly. For well-behaved dicts iter(d) and iter(d.keys()) are identical, so this is a no-op for normal input and only changes behaviour for subclasses with a non-standard __iter__.
  • When neither side exposes any raw key yet the objects are not equal (a dict subclass holding hidden state, e.g. a scalar JsonObject), the level is reported as values_changed instead of silently returning {}. Raw keys (before private-key filtering) are used for the check, so a dict differing only in ignored private keys is still correctly reported as {}.

Tests

  • tests/test_diff_text.py: two new tests using local stub dict subclasses that mimic JsonObject (no new test dependency on google-cloud-spanner):
    • test_dict_subclass_whose_iter_yields_values — array case: added/removed/changed/nested/equal.
    • test_dict_subclass_with_no_keys_but_hidden_state — scalar case: changed vs equal, plus empty-dict sanity.
  • Verified against the real JsonObject (installed google-cloud-spanner in a scratch venv): before the fix the issue repro raised TypeError: unhashable type: 'dict'; after the fix it returns {'dictionary_item_added': ['root[0]']}. Scalar case before: TypeError: 'int' object is not iterable; after: {'values_changed': {'root': {'new_value': 6, 'old_value': 5}}}. Plain-object JsonObject, null, nested and ignore_private_variables cases all behave correctly, with no false positives on empty/equal dicts.
  • Full suite: pytest tests/ — 1346 passed, 45 skipped, 0 failed.
  • Lint: flake8 on the touched files reports the same 288 pre-existing findings as the clean tree — no new violations.

Notes

  • The upstream JsonObject.__iter__ arguably violates the Mapping protocol, but DeepDiff should not crash on a dict subclass; this fix keeps the workaround on our side and is behaviour-preserving for normal dicts.
  • AI assistance: this change was prepared with AI assistance (Claude).

Dict subclasses may override __iter__ to yield values instead of keys
(google.cloud.spanner JsonObject holding a JSON array does this), which
made _diff_dict crash building its key sets on unhashable values. Collect
keys via .keys() instead, which is identical for well-behaved dicts.
Dict subclasses exposing no keys at all but unequal content (JsonObject
holding a scalar) are now reported as changed instead of silently
returning {}.

Fixes qlustered#569
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant