feat: Generalize metrics and add nullability delta metric by MoritzPotthoffQC · Pull Request #43 · Quantco/diffly

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-10T13:39:32Z

Motivation

I like the concept of metrics to give the user as much information about their data at first glance. I think it would be nice to extend it even further by generalizing metrics to non-numeric columns and provide metrics that describe the two datasets, not only the change. The specific use-case I have for this is to explain how the share of null values in a column changed.

Changes

Introduce metrics subpackage
Move existing metrics to change submodule
Introduce data submodule for the new metric type (not only numeric)
Keep existing behavior for passing callables (numeric only), but additionally support metrics with custom column selector
Add a nullability change metric with format <old> -> <new> (<delta>).
Update docs and CLI accordingly

codecov · 2026-07-10T13:40:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (ea4ee01) to head (901590c).

Additional details and impacted files

@@            Coverage Diff            @@
##              main       #43   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           11        14    +3     
  Lines         1018      1052   +34     
=========================================
+ Hits          1018      1052   +34

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR extends Diffly’s per-column summary metrics to support non-numeric columns via an explicit Metric(fn, selector=...) wrapper, and introduces a built-in metric for nullability changes so users can quickly see how null fractions evolve between left/right frames.

Changes:

Allow summary(metrics=...) / testing helpers to accept either a bare metric callable (numeric-only) or a Metric with a Polars selector (any column types).
Add the null_fraction_change preset and expose it as ΔNull% in DEFAULT_METRICS and the CLI.
Update summary formatting + docs, and add tests + generated summary fixtures for the new metric behavior.

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
diffly/metrics.py	Adds `Metric` wrapper type, null-fraction change preset, and updates `DEFAULT_METRICS` to include selector-based metrics.
diffly/comparison.py	Accepts `Metric` objects in `summary(metrics=...)` and resolves bare callables as numeric-only.
diffly/summary.py	Allows metrics to render string results in column summaries.
diffly/testing.py	Broadens `metrics` typing/docs to allow `Metric` values in assertion helpers.
diffly/cli.py	Updates CLI help text and passes selector-based preset metrics through to `summary()`.
docs/api/metrics.rst	Documents selector-based metrics and adds `Metric` + `null_fraction_change` to the API docs.
tests/test_metrics.py	Adds unit tests for `null_fraction_change` output (including non-numeric columns).
tests/cli/test_cli.py	Adds CLI coverage for selecting the `ΔNull%` metric preset.
tests/summary/fixtures/metrics_null_fraction/test_metrics_null_fraction.py	New fixture generator test for summaries including `ΔNull%` and a selector-based custom metric.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

EgeKaraismailogluQC

Thank you Moritz Potthoff (@MoritzPotthoffQC), this was pleasant to review 😄 I only have some cosmetic suggestions, but also one question regarding the definition of the Null% metric:

The current definition compares the mean Null%s to each other. An alternative definition would be to check this on a per-row basis, i.e. to check how often a row that was Null is now not Null, and vice versa. When implementing the metrics feature, I rather had this second variant in mind since it makes use of the matching between each pair of rows, but the variant you implemented is of course also legitimate. My question: is this alternative definition potentially also interesting for your use case, or are you only interested in the variant which you implemented in this PR?

Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

…_fraction.py Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

Co-authored-by: quant-ranger[bot] <132915763+quant-ranger[bot]@users.noreply.github.com> Co-authored-by: Marius Merkle <marius.merkle@quantco.com> Co-authored-by: Marius Merkle <122545105+MariusMerkleQC@users.noreply.github.com>

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-12T20:52:01Z

+def null_fraction_change(left: pl.Expr, right: pl.Expr) -> pl.Expr:
+    """Change in the fraction of null entries, rendered as ``<old> -> <new> (<delta>)``.
+
+    ``old`` and ``new`` are the null percentages of the left and right side, and
+    ``delta`` is their signed difference (``+`` when the right side has proportionally
+    more nulls, ``-`` when it has fewer). This metric function can be applied to columns
+    of any type.
+    """


(moving this here to get a thread)

An alternative definition would be to check this on a per-row basis, i.e. to check how often a row that was Null is now not Null, and vice versa. When implementing the metrics feature, I rather had this second variant in mind since it makes use of the matching between each pair of rows

I see, good point!

My question: is this alternative definition potentially also interesting for your use case, or are you only interested in the variant which you implemented in this PR?

Right now, I am really only interested in the variant implemented here right now.

As I see it, we currently follow two objectives:

The existing metrics are metrics about the change between numeric columns. I.e., we try to describe the change itself.

My new metric describes the old and new datasets individually, so that you can understand how a change affects the data.

I think (1) makes sense for numeric columns and in the context of diffly. (2) is also nice since it allows us to understand how the overall data changed (and it generalizes to non-numeric columns).

I could imagine that it would even be nice to expand on (2) in the future (e.g., by providing more metrics in this way). If we do that, we should probably make the differentiation obvious (e.g., through different modules). Wdyt, EgeKaraismailogluQC ?

If we do that, we should probably make the differentiation obvious (e.g., through different modules)

My impression is that we should probably even do this now already

Yes, I tend to agree after some more thought. How should we name these two kinds of metrics? I thought of "change metrics" for (1) and "data metrics" for (2). Do you have any other suggestions? Would it make sense to expose two name spaces, i.e. metrics.change and metrics.dataset, or do you have another idea?

I'd go for metrics.data, but sounds good otherwise. I will extend this PR accordingly

Implemented this. PTAL again

Makes sense! Pinging Marius Merkle (@MariusMerkleQC) and Oliver Borchert (@borchero) here since we already discussed these two types of metrics before. The implementation looks sound to me, and the docs are clear. Thanks!

I think the distinction between these two types of metrics makes a lot of sense. However, I currently find the summary report a little confusing. In my mind, the column-level section only describes the change and it feels a little strange to have dataset-level information in there 🤔

I think the different layout (old -> new (delta) for the data metrics vs scalar for the change metrics) helps, but I agree that the difference is subtle and it is easy to miss what we compute here (especially with the current metric names). I think to address this, we could keep all the change metrics where they currently are (along with the match rate) and add a new section that contains the new data metrics for each column.
For naming, I propose Data Metrics to make clear that this section does not directly describe the change anymore (as above), but actually covers the data.
On an implementation level, I think this would require significantly more changes than we currently have (especially to have a clean interface to specify both of these types).

Do you see any other alternatives?

I think the "match rate" (which does not include the ->) is the obvious example where this argument breaks 😅

That being said, I would be fine to keep it like it is as a temporary "workaround". After all, it only affects display, not the package's API.

Going forward, we could (1) either have a new section (as you described) or (2) somehow partition columns to not repeat columns. I think (1) makes a lot more sense. Personally, I would probably go with a naming along the lines of "inspection" but don't have a strong preference for anything.

I think the "match rate" (which does not include the ->) is the obvious example where this argument breaks 😅

I guess one could see the match rate as a metric of the change ((left == right).mean()) 😂

That being said, I would be fine to keep it like it is as a temporary "workaround". After all, it only affects display, not the package's API.

Okay 👍 Then I will remove the new data metric from the default metrics for now so that we don't display them side-by-side by default for now.

I think (1) makes a lot more sense. Personally, I would probably go with a naming along the lines of "inspection" but don't have a strong preference for anything.

It has the additional advantage that we have more horizontal space to work with 😅

…d-nullability

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

diffly/metrics/change.py:73

Renaming DEFAULT_METRICS to DEFAULT_CHANGE_METRICS in diffly.metrics.change is a backwards-incompatible API change for consumers importing the old name. Consider keeping DEFAULT_METRICS as an alias to avoid breaking downstream code.

EgeKaraismailogluQC

Thanks Moritz Potthoff (@MoritzPotthoffQC)!

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-14T14:28:18Z

Waiting for this discussion to be settled before merging

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

init

e97b755

Moritz Potthoff (MoritzPotthoffQC) self-assigned this Jul 10, 2026

github-actions Bot added the enhancement New feature or request label Jul 10, 2026

Moritz Potthoff (MoritzPotthoffQC) added 3 commits July 10, 2026 15:46

better output

1e72823

cleanup

1d7bdeb

refactor

a74429f

This was referenced Jul 10, 2026

feat: Improve existing numeric metrics for new output format #44

Closed

ci: Fix docs build #45

Merged

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 14:28

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 14:28 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/metrics.py Outdated

Comment thread docs/api/metrics.rst Outdated

Comment thread diffly/comparison.py

Comment thread tests/test_metrics.py

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 10, 2026 16:36

Potential fix for pull request finding

71338c9

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix

cd33f6a

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 14:40

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 14:40 View session

fix

f66bfa0

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/comparison.py Outdated

Comment thread diffly/metrics.py Outdated

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 10, 2026 17:15

Potential fix for pull request finding

db3b168

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Merge branch 'main' into generalize-metrics-add-nullability

eb1c427

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 15:34

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 15:35 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/summary.py

Comment thread diffly/metrics/data.py

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as ready for review July 10, 2026 15:42

Moritz Potthoff (MoritzPotthoffQC) requested review from EgeKaraismailogluQC, Marius Merkle (MariusMerkleQC) and Oliver Borchert (borchero) as code owners July 10, 2026 15:42

EgeKaraismailogluQC reviewed Jul 12, 2026

View reviewed changes

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 12, 2026 22:39

Update docs/api/metrics.rst

2923f2e

Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

Update tests/summary/fixtures/metrics_null_fraction/test_metrics_null…

e1ea6bd

…_fraction.py Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

quant-ranger Bot and others added 3 commits July 12, 2026 22:40

chore: Update pixi lockfile (#42)

fc0426a

Co-authored-by: quant-ranger[bot] <132915763+quant-ranger[bot]@users.noreply.github.com> Co-authored-by: Marius Merkle <marius.merkle@quantco.com> Co-authored-by: Marius Merkle <122545105+MariusMerkleQC@users.noreply.github.com>

review

f5b9117

fix

e374f0d

Moritz Potthoff (MoritzPotthoffQC) commented Jul 12, 2026

View reviewed changes

fix

627721e

Moritz Potthoff (MoritzPotthoffQC) requested a review from EgeKaraismailogluQC July 12, 2026 20:57

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as draft July 13, 2026 15:23

Moritz Potthoff (MoritzPotthoffQC) added 3 commits July 13, 2026 18:01

Merge remote-tracking branch 'origin/main' into generalize-metrics-ad…

cf686b4

…d-nullability

fix

529b917

fix

6f5671d

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 13, 2026 17:09

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 13, 2026 17:10 View session

fix docs

bfc8d8c

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread diffly/metrics/__init__.py

Comment thread diffly/metrics/__init__.py

Comment thread diffly/testing.py

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as ready for review July 13, 2026 17:17

Moritz Potthoff (MoritzPotthoffQC) commented Jul 13, 2026

View reviewed changes

Comment thread diffly/metrics/__init__.py

fix docs

bfc03ad

EgeKaraismailogluQC approved these changes Jul 14, 2026

View reviewed changes

Moritz Potthoff (MoritzPotthoffQC) enabled auto-merge (squash) July 14, 2026 14:27

Moritz Potthoff (MoritzPotthoffQC) disabled auto-merge July 14, 2026 14:27

review

b309e7a

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 15, 2026 13:33

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 15, 2026 13:34 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread diffly/cli.py

fix CLI

901590c

Uh oh!

Conversation

Moritz Potthoff (MoritzPotthoffQC) commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

codecov Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

EgeKaraismailogluQC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Moritz Potthoff (MoritzPotthoffQC) Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Oliver Borchert (borchero) Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EgeKaraismailogluQC left a comment

Choose a reason for hiding this comment

Uh oh!

Moritz Potthoff (MoritzPotthoffQC) commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Moritz Potthoff (MoritzPotthoffQC) commented Jul 10, 2026 •

edited

Loading

codecov Bot commented Jul 10, 2026 •

edited

Loading

Moritz Potthoff (MoritzPotthoffQC) Jul 15, 2026 •

edited

Loading

Oliver Borchert (borchero) Jul 15, 2026 •

edited

Loading