Skip to content

live Python tiers fail instead of skipping when the attached graph is below the analysis level they need #424

Description

@rahlk

Describe the bug

Several live Python modules fail rather than skip when the attached graph was emitted below the analysis level they need. The suite already knows how to do this correctly in places, so the inconsistency is the bug.

Measured on a graph of odoo-slim-19 emitted by codeanalyzer-python 1.5.3 at analysis_level=1 (1,626 modules, 15,549 callables) — a graph that answers every level-1 query correctly:

  • test_e2e_neo4j_live.py::test_get_config_keys_and_config_uses_line_up asserts get_config_uses() is non-empty whenever get_config_keys() is. The two are not tied: config keys come from artifact extraction at level 1, while config uses are resolved at analysis_level >= 2 (codeanalyzer-python's core.py, the literal tier). The graph has 93 :ConfigKey nodes and zero PY_USES_CONFIG edges. Both numbers are correct for a level-1 emit.
  • Six failures across test_resolve.py raise SelectorNotInGraph ('invoice_id', 'vals', 'AccessError'). resolve_value addresses formal_in ports, which are a level-4 overlay.

The assertion message compounds it: "came back empty on a graph with PY_USES_CONFIG edges" states as fact something the test never checked, and it is false on this graph. I believed that message over the graph and reported the wrong finding once already.

The correct shape is already in the same file. test_get_config_readers_finds_the_callable_that_reads_the_key queries for PY_USES_CONFIG edges first and skips cleanly when there are none.

To Reproduce

CLDK_TEST_NEO4J_URI=bolt://localhost:7698 CLDK_TEST_NEO4J_USER=neo4j \
CLDK_TEST_NEO4J_PASSWORD=... CLDK_TEST_NEO4J_APP=odoo-slim-19 \
pytest tests/analysis/python/test_e2e_neo4j_live.py tests/analysis/python/test_resolve.py

against a level-1 graph: 8 failed, 100 passed, 17 skipped. The same run on release/2.0 before #412 gives the identical failure set, so this predates that change.

Expected behavior

A live module states the analysis level it needs and skips with that reason when the attached graph is below it, the way the config-readers test already does. A test that needs an overlay checks the overlay is present, not a neighbouring one. The skip reason names the level, so a green run with skips is readable rather than reassuring.

Additional context

Scope boundary

  • Test-suite only. No facade change, no query change, no new accessor.
  • Not about whether a level-1 graph should answer these queries. It should not, and it does not; this is about how the suite reports that.
  • The eighth failure in the same run is not in scope: :PyModule now carries a source property, so the module_source_unavailable divergence the live twin pins is genuinely stale. That is the read-path uptake tracked in Incorporate codeanalyzer-python v1.5.2 #396.

Caveats and known risks

  • A guard that is too broad silently stops testing something. python-sdk#362 is the precedent: a retired env-var spelling made a run end green having verified nothing. The skip reason must name the level and the overlay it probed, so a reader can tell "not applicable here" from "quietly stopped checking".
  • The probe must read the data (are there PY_USES_CONFIG edges? are there formal_in ports?), never an analyzer version string. A graph's level is not recorded on it, and :PyApplication carries no analysis_level property — verified on this graph.

Definition of done

Against a level-1 graph the four modules report skips with reasons naming the missing overlay and zero failures; against a level-4 graph they run and assert as they do today; the misleading assertion message is gone.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions