FFL-2783: add tests for observeFullEvaluationData PII protection (gated as missing_feature)

[vjfridge]

Motivation

FFL-2783 — part of FFL-2780 (Protecting PII in flagevaluations track)

See informal RFC for more context

Summary

• Adds three test classes to existing tests/ffe/test_flag_eval_evp.py covering the new per-env UFC observeFullEvaluationData field
• Extends make_ufc_fixture with an observe_full_evaluation_data: bool | None = None kwarg. None omits the field entirely, so existing tests are unaffected
• All three classes are deactivated as missing_feature (FFL-2783) on Go — the only SDK where the file-level gate would otherwise activate them. Other SDK manifests already deactivate the whole file via missing_feature (FFL-2446) or irrelevant.

Truth-table coverage

UFC observeFullEvaluationData	Expected targeting_key	Expected context.evaluation
absent	sha256_-prefixed (71 chars, lowercase hex suffix)	omitted
false	sha256_-prefixed	omitted
true	raw customer value, no prefix	full object with all attributes

Canonical PII vector (shared across SDKs)

Every SDK's unit tests should assert against the same input/output. Input: "jane.doe@datadoghq.com" → "sha256_b4698f9b6d186781fa8dc59e533578fa2d8379a46b1cf6db85cda6aa9c99e51b". Canonical attributes cover string, integer, dotted-key, and email PII shapes.

Forward-compat for existing SDKs

Introducing a new field inside the UFC environment object could theoretically break SDKs if their UFC parser rejects unknown fields. That risk is already covered by tests/ffe/test_dynamic_evaluation.py::Test_FFE_Unknown_Fields_Tolerance, which injects random unknown fields at every UFC level (including inside environment) and asserts each SDK continues to evaluate flags correctly. That test runs on every SDK where test_dynamic_evaluation.py is active, so no per-fixture retrofit is needed here.

Out of scope

• Existing kill switch (DD_FLAGGING_EVALUATION_COUNTS_ENABLED) — per-SDK unit test territory, no system-tests scenario variant needed.

Test plan

• ./format.sh clean (mypy, ruff, node linters)
• Once Go SDK ships FFL-2790, flip Go's three missing_feature (FFL-2783) rows to v2.X.Y-dev and confirm all three cells pass

🤖 Generated with Claude Code

[github-actions]

CODEOWNERS have been resolved as:

manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
tests/ffe/test_flag_eval_evp.py                                         @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/utils/fixtures.py                                             @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

[datadog-datadog-us1-prod]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 08669d5 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d1ad980b2f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Move fallible setup work into the test

These new setup_* methods perform remote-config application and the weblog request before the test body; if either path raises or times out in CI, pytest reports a setup error and the scenario can stop instead of recording a normal test failure. .cursor/rules/pr-review.mdc says setup methods must only trigger observability and failures/assertions belong in test_*, so keep setup minimal and move the fallible validation path into the matching test method.

Useful? React with 👍 / 👎.

[vjfridge]

Not addressing this one. The other eight test classes in this file (Test_FFE_EVP_Flagevaluation_Basic, _Count, _Context_Bounds, _Runtime_Default, _Load_Aggregation, _Burst_Aggregation, _High_Cardinality_Aggregation, _Degradation) all place rc...apply() and evaluate_flag(...) in setup_* and only run assert self.r.status_code == 200 in test_*. Diverging in only the three new classes would create an inconsistency reviewers would flag; if the setup-vs-test split needs enforcement here, it belongs in a dedicated refactor across the whole file.

[cbeauchesne]

From framework usage, just a small request change.

Cold you also get a review from someone familiar with the feature for the test logic ?

[cbeauchesne]

There are semantic decorators in utils that allow to achieve the same, and also declare why we want to skip them : @slow and @scenario_crash, could you use them ?

[vjfridge]

Thanks for raising this! You're right that using @pytest.mark.skip_if_xfail directly is too raw. We looked at @slow and @scenario_crash but neither quite fits — these tests aren't slow, and they don't crash the scenario. They're gated on SDK implementation status.

So we added a new alias @not_yet_implemented in utils/_decorators.py (same underlying skip_if_xfail mark, just with a clearer name) and used it consistently across all 11 classes in the file.

[nccatoni]

Why is it not enough to mark the tests as xfails in that case?

[vjfridge]

See #7316 (comment)!

[leoromanovsky]

logic is good, before merging please:

• add a env var=false case
• adjust labels on existing test cases

[leoromanovsky]

hehe @cbeauchesne skipped these because they are not set up correctly; I needed to break system test convention to get them to work with PHP, which has its own sidecar that needs an additional await to count received EVP events in the backend. It's on my backlog to address, but a more correct skip decorator would be flaky or bug

[vjfridge]

Thanks for raising Leo! I looked into this more carefully. The existing @flaky/@bug decorators are all-or-nothing across SDKs. Per-SDK skipping lives in the manifests, so I think it would be best to wait until PHP gets enabled and then add class-level manifest entries in manifests/php.yml.

For now I've added a docstring note to Burst, HighCardinality, and Degradation pointing to FFL-2529 and the manifest pattern, so whenever we drop the file-level missing_feature for PHP we will know to add those entries at the same time.

I left @not_yet_implemented on all three for now since that's still the accurate for PHP and most SDKs. Let me know if this works for you or if you were looking for something different!

[nccatoni]

The flaky and bug decorators accept arguments that let you filter based on several context elements, such as the library, library version, or weblog. See the (documentation) for more details.

However, you should prefer using manifests wherever possible. Also, these decorators do not behave the same way as skip_if_xfail. The skip_if_xfail decorator skips disabled tests and should only be used when running the test would have a significant downside, for example, if it is very slow or could crash the scenario.

The terminology can be a little confusing, so you may also want to review our glossary for more details.

I may be missing some context, but from what I can see, you do not need decorators here and should be able to put everything in the manifests.

Please do not hesitate to reach out if you have any questions.

[vjfridge]

Ahhh thanks for the additional context and docs @nccatoni! This is my first PR in system-tests so those links are great context.

I tried removing the @pytest.mark.skip_if_xfail / @not_yet_implemented decorators entirely to let the manifest missing_feature (FFL-2446) gate do its job (and preserve xpass detection for the easy-win activation script). But CI showed running these tests against the PHP weblog crashes the /ffe endpoint mid-run (it starts returning HTTP 500 with empty body), which then breaks sibling tests in test_flag_eval_metrics.py that share the scenario.

• See the failure at job 90074003763 (test_flag_eval_metrics starts passing and then fails partway through once the weblog has been destabilized). Root cause is a PHP sidecar timing issue tracked in FFL-2676.

That's exactly the case @scenario_crash is documented for in docs/edit/skip-tests.md: "a failing test that brings down the whole scenario prevents other tests from running." So I've pushed a change (221e146) that replaces @not_yet_implemented with @scenario_crash on all EVP test classes (same force-skip behavior Leo originally added, but with the semantically-correct decorator name from utils).

The tradeoff (as @cbeauchesne noted): once the feature is implemented across SDKs, @scenario_crash continues to force-skip xpasses so we won't get automatic easy-win activation on this file. That's a conscious cost given how badly the PHP weblog crashes when we don't force-skip. Once FFL-2676 is fixed we can drop @scenario_crash entirely.

Let me know if I'm missing something! I think this is what Leo intended with his review but I misunderstood what @scenario_crash meant so thanks for helping clarify!

[leoromanovsky]

Please add a test for DD_FLAGGING_EVALUATION_COUNTS_ENABLED=false + observeFullEvaluationData=true - sorry I missed adding coverage for the false case.

[vjfridge]

Happy to add coverage for DD_FLAGGING_EVALUATION_COUNTS_ENABLED =false + observeFullEvaluationData=true, but I think it lives better at L1 unit tests/L2 SDK integration layers.

The assertion is "SDK-level killswitch overrides UFC opt-in" which is purely internal to the SDK and doesn't exercise anything system-level. Adding an L3 scenario here means a new weblog boot per SDK (~minutes of CI × N languages) to verify the boolean killswitch.

I propose we cover this in each SDK's unit or integration suite instead. Let me know if you have strong feelings otherwise!

[vjfridge]

Local system-tests validation — Java SDK ✅

Ran the three new observeFullEvaluationData PII-protection tests locally against dd-trace-java#12042 (java@1.65.0-SNAPSHOT+81af153ca7).

All three pass:

Test	Result
Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed	✅ PASS
Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed	✅ PASS
Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed	✅ PASS

36 passed, 8 skipped, 2704 deselected, 1 xfailed, 3 xpassed in 301.86s

Run command:

./run.sh +l java FEATURE_FLAGGING_AND_EXPERIMENTATION \
  -F tests/ffe/test_flag_eval_evp.py::Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed \
  -F tests/ffe/test_flag_eval_evp.py::Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed \
  -F tests/ffe/test_flag_eval_evp.py::Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed

Note: an earlier build (d65c79f266) failed True_Unhashed — the SDK was always hashing regardless of the flag. Root cause was FlagEvaluationWriterImpl.buildEventList() reading observeFullEvaluationData from CURRENT_CONFIG at flush time, which could be overwritten by a subsequent RC update before the 10s flush fired. Fixed in the dd-trace-java PR by capturing the value per-bucket at enqueue time with a privacy-preserving fold (any false in a bucket's lifetime sinks it to hashed).

Addressed the feedback and @cbeauchesne is OOO for several weeks, asking for another codeowner re-review in #apm-shared-testing