Parse observeFullEvaluationData and hash targeting_key in flagevaluations events

[vjfridge]

Background

Feature-flag evaluations are reported to Datadog so users can see how their flags behaved in production. Today, every evaluation carries the raw targeting key (the identifier of the subject being evaluated; often a user email or user ID) and the full evaluation context. Those fields can contain personally identifiable information (PII).

Motivation & High level changes

This PR lets the server decide, per environment, whether that raw data may be observed via a new boolean on the UFC. It is the Java piece of the cross-SDK "Protecting PII in flagevaluations" initiative. dd-trace-java is the pilot SDK; the same pattern will be copied to the other SDKs afterward.

This directly resolves the privacy concern raised on the parent PR #11639, where @AlexeyKuznetsov-DD flagged that flag-evaluation reporting was default-on and shipped the raw targeting_key (user IDs / emails) plus the full evaluation context in clear text, and asked for hashing / opt-in / an explicit privacy sign-off. The no-PII path is now the default.

Server says observeFullEvaluationData	targeting key	evaluation context
true	sent as-is (raw)	included (raw)
false or absent (privacy-preserving default)	SHA-256 hashed, sha256_<hex>	omitted entirely

"Absent" is treated exactly like false, so an older/cached config can never accidentally leak raw values.

How it works

• New field observeFullEvaluationData on the downloaded flag-configuration (UFC) model (top-level boolean, defaults to false when missing).
• Consent is captured at evaluation time, on the evaluation thread, and carried on the event. DDEvaluator snapshots the active UFC's observeFullEvaluationData once per evaluation and stamps it onto the ProviderEvaluation metadata; the OpenFeature hook (FlagEvalLoggingHook) reads that metadata value and stores it as FlagEvalEvent.observeFullEvaluationData. Aggregation and serialization then read that per-event snapshot and never consult the live gateway — so the hashed-vs-raw decision is pinned to the configuration active when the evaluation happened.
  • Why not read it at drain/flush? CURRENT_CONFIG can be overwritten by a later Remote Config update between when an evaluation happened and when the batch is drained and flushed. Reading the gateway at flush would retroactively apply a different environment's consent to already-collected evaluations. (This was a real bug caught by system-tests: a targeting key came back hashed even though the active UFC said true.) Snapshotting on the evaluation thread closes that window in both directions.
  • Bucket separation. observeFullEvaluationData is part of the aggregation bucket key (FullKey / DegradedKey), so events with different consent values never merge into the same bucket. Consent-off and consent-on evaluations for the same subject land in distinct buckets and are emitted with their respective policies. An AND-fold on the bucket's consent field is kept as defense-in-depth in case the key ever drifts from the field.
• When hashing, the targeting key is run through the shared ULeb128Encoder.hashTargetingKey helper and prefixed with sha256_; the context field is dropped so it's absent from the JSON (not null, not {}).
• Evaluation logic is unchanged.

SHA-256 hash

The hash is unsalted SHA-256 over the raw UTF-8 bytes (no trimming/case/Unicode normalization), emitted as lowercase hex. Unsalted is a deliberate cross-SDK contract requirement: every SDK must produce the identical digest for a given targeting key so hashed values line up across languages and against the backend. It matches the canonical test vector shared across all SDKs:

"jane.doe@datadoghq.com" -> sha256_b4698f9b6d186781fa8dc59e533578fa2d8379a46b1cf6db85cda6aa9c99e51b

Performance

Consent is read once per evaluation on the hook path — a single AtomicReference.get snapshotted onto the event (no allocation, no per-span hot path). Hashing runs at flush cadence, bounded by the number of distinct evaluation buckets, and uses a ThreadLocal<MessageDigest> (no per-call MessageDigest.getInstance).

Testing

• Hash helper matches the canonical PII vector; preserves whitespace/case; is lowercase 64-char hex.
• Config parsing covers true, false (as a @ValueSource matrix), absent (defaults to false), and explicit null (rejected / fail-closed).
• Serialization covers both raw and hashed paths, asserting on the raw wire bytes that the hashed value is present, the raw PII value never appears, and no per-event context is emitted.
• Consent-capture at evaluation time: hook tests assert the event's consent value is read from the evaluation metadata stamped by DDEvaluator at evaluation time (true, false, absent → false), and that the hook ignores the live gateway even when it disagrees with the metadata (ignoresGatewayConsentEvenWhenItDisagreesWithMetadata). FlagEvalEvent tests cover the new field and the fail-closed default on the convenience constructors.
• Consent-timing regression guards (both directions): a bucket whose event was snapshotted with consent off stays hashed even if a later RC update turns the gateway on before drain/flush; and symmetrically, an event snapshotted with consent on stays raw even if the gateway is later turned off — proving neither aggregation nor flush consults the gateway.
• Bucket separation: mixed-consent evaluations for the same subject land in distinct buckets (mixedConsentEvaluationsForSameSubjectLandInDistinctBuckets), and same-consent evaluations merge as before (sameConsentEvaluationsForSameSubjectMergeIntoOneBucket).

🤖 Generated with Claude Code

[datadog-prod-us1-5]

🎯 Code Coverage (details)
• Patch Coverage: 94.12%
• Overall Coverage: 58.03% (+0.01%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1b034cc | Docs | Datadog PR Page | Give us feedback!}

[vjfridge]

system-tests validation (L3) — Java SDK ✅

Ran the three new observeFullEvaluationData PII-protection tests from system-tests#7316 locally against this branch (java@1.65.0-SNAPSHOT+81af153ca7).

All three pass:

Test	Result
Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed	✅ PASS
Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed	✅ PASS
Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed	✅ PASS

36 passed, 8 skipped, 2704 deselected, 1 xfailed, 3 xpassed in 301.86s

Weblog: spring-boot | Scenario: FEATURE_FLAGGING_AND_EXPERIMENTATION

Note: an earlier build (d65c79f266) of this PR failed True_Unhashed — the SDK was always hashing regardless of the flag value. Root cause: FlagEvaluationWriterImpl.buildEventList() read observeFullEvaluationData from CURRENT_CONFIG at flush time, which could be overwritten by a subsequent RC update before the 10s flush fired. The fix in this PR (capturing the value per-bucket at enqueue time with a privacy-preserving fold) resolves it.

[vjfridge]

ffe-dogfooding validation (L2) - Java SDK ✅

I ran this branch's dd-java-agent + dd-openfeature build through our OpenFeature dogfooding harness against a staging UFC, exercising all three behaviors of the PII-protection contract:

Protected — observeFullEvaluationData = false (default):

• targeting_key = sha256_ + 64-char lowercase-hex SHA-256 digest (71 chars total)
• context.evaluation omitted
• consistent across every emitted flagevaluation event

Full — observeFullEvaluationData = true (opt-in):

• targeting_key raw (verbatim subject id)
• context.evaluation present with the full evaluation context

Kill switch — DD_FLAGGING_EVALUATION_COUNTS_ENABLED = false:

• zero flagevaluation events emitted regardless of observeFullEvaluationData (SDK logs Flag evaluation EVP writer disabled)

Notes:

• Pointing the app at environments with observeFullEvaluationData false vs true produced exactly the expected payload shapes.
• Hashing was applied even to FLAG_NOT_FOUND (runtime-default) evaluations — the privacy posture is enforced independent of evaluation outcome. 👍
• The SDK posts to the singular /api/v2/flagevaluation EVP route, as expected for the flagevaluation track.

Validation harness changes: ffe-dogfooding#100 — adds the per-SDK PII posture dashboard badge and fixes the singular /api/v2/flagevaluation intake route.

Verdict: the Java SDK is performing as expected for the PII-protection contract — default hashing, opt-in full data, and kill switch all behave correctly. Cross-SDK hash equality is covered in the companion system-tests work.

[datadog-prod-us1-5]

More details

The event-time snapshot, conservative bucket fold, and serialization gates preserve the no-PII default while retaining raw data only for explicitly consented evaluations. No additional tests recommended: production telemetry is unavailable and the PR already covers the relevant consent transitions and wire shape.

Was this helpful? React 👍 or 👎

_{📊 Validated against 3 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit e0e3632 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0e3632b8c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bind consent to the evaluated configuration

When Remote Config changes from observeFullEvaluationData=false to true while an evaluation is in progress, DDEvaluator.evaluate can finish using the old configuration while this hook reads the new global value. FeatureFlaggingGateway.dispatch updates CURRENT_CONFIG before notifying the evaluator, and the evaluator retains its configuration in a local variable, so this race can mark an evaluation performed without consent as consented and emit its raw targeting key and context. Carry the consent from the exact ServerConfiguration used by the evaluator, for example through evaluation metadata, instead of rereading the gateway here.

Useful? React with 👍 / 👎.

[leoromanovsky]

You had a good goal to seal the full evaluation metatadata to the configuration at the time of evaluation. The configuration you are accessing is mutable by other threads. Read it earlier than the hook, in the evaluation.

[dd-oleksii]

+1, returning the flag via evaluation metadata seems like a good way to make sure the flag is consistent with evaluation

[vjfridge]

Thanks @leoromanovsky and @dd-oleksii for the +1s here! Storing the observeFullEvaluationData value with the evaluation metadata is much cleaner.

Addressed in 933cf3c ("Bind observeFullEvaluationData consent to the evaluator's configuration"), with follow-ups in 98df3ba and b6d9d60.

DDEvaluator.evaluate() now stamps observe_full_evaluation_data onto every returned ProviderEvaluation, sourced from the UFC snapshotted at the top of evaluate(). The hook reads consent from that metadata and no longer queries the gateway. FeatureFlaggingGateway.isObserveFullEvaluationDataEnabled() was deleted so the race can't be reintroduced.

Semantics:

• Every DD-produced evaluation carries the key, including PROVIDER_NOT_READY (stamped false).
• Missing key = non-DD provider → hook falls back to false (fail-closed).

[chatgpt-codex-connector]

Stop keying redacted buckets by hidden context

perf: When observeFullEvaluationData is false—the new default—a changing context field such as a request ID still contributes to ctxKey and creates a distinct full-tier bucket, even though serialization omits that context. At sufficient traffic this emits many wire-identical rows, exhausts the 10,000-bucket per-flag cap within a flush interval, and forces subsequent evaluations into the degraded tier without targeting information. No-consent evaluations should aggregate without the context dimension that will be discarded.

AGENTS.md reference: AGENTS.md:L77-L77

Useful? React with 👍 / 👎.

[leoromanovsky]

The full aggregation key includes the evaluation context. This behavior is correct when observeFullEvaluationData=true.

For example, these evaluations have different contexts:

  targeting key: alice@example.com
  context 1: {plan: pro, request_id: req-1}
  context 2: {plan: pro, request_id: req-2}
  context 3: {plan: pro, request_id: req-3}

When full evaluation data is enabled, the SDK emits each context. Separate aggregation buckets are necessary.

When observeFullEvaluationData=false, the SDK omits the evaluation context. All three evaluations then have the same visible identity:

  {
    "flag": {"key": "checkout"},
    "variant": {"key": "enabled"},
    "targeting_key": "sha256_<alice-hash>"
  }

The current implementation still uses request_id in the aggregation key. It creates three buckets, although the emitted context is identical.

The protected path also performs unnecessary operations. It:

• copies the evaluation context
• flattens the evaluation context
• sorts and prunes its fields
• creates a canonical context key
• stores the context until the next flush
• removes the context during serialization.

High-cardinality fields can create many unnecessary buckets. Examples include request IDs, timestamps, and correlation IDs.

The per-flag limit is 10,000 full buckets. After this limit, the SDK moves later evaluations to degraded aggregation. Degraded events omit both the context and targeting key.

When observeFullEvaluationData=false, the SDK must:

• keep the targeting key in the aggregation key
• omit the evaluation context from the aggregation key
• avoid evaluation-context processing
• avoid storing the evaluation context.

Different subjects must remain in separate buckets:

  alice@example.com -> sha256_<alice>
  bob@example.com   -> sha256_<bob>

Different contexts for the same subject must use one protected bucket.

👉 Overall this feels like a "P1" to me; see if it is straight-forward to resolve.

[vjfridge]

Thanks @leoromanovsky! This is a great catch I did not think about and will definitely save us form degrading evaluations unnecessarily.

What are we doing now?

Per your example of

targeting key: alice@example.com
  context 1: {plan: pro, request_id: req-1}
  context 2: {plan: pro, request_id: req-2}
  context 3: {plan: pro, request_id: req-3}

the new behavior is:

Case A: observeFullEvaluationData = true for all three

The three ctxKeys are distinct because request_id differs:

• Event 1 → ctxKey₁ = canonical serialization of {plan:pro, request_id:req-1}
• Event 2 → ctxKey₂ = canonical serialization of {plan:pro, request_id:req-2}
• Event 3 → ctxKey₃ = canonical serialization of {plan:pro, request_id:req-3}

At flush, includeRawContext = isFullTier && observeFullEvaluationData is true, so each bucket serializes the full pruned context:

row 1: {targeting_key: "alice@example.com", context: {plan:pro, request_id:req-1}}
row 2: {targeting_key: "alice@example.com", context: {plan:pro, request_id:req-2}}
row 3: {targeting_key: "alice@example.com", context: {plan:pro, request_id:req-3}}

Results in three genuinely different raw evaluations, three rows on the wire 👍

Case B: observeFullEvaluationData = false for all three

In the hook (before enqueue):

if (observeFullEvaluationData) { ... } else {
  // no snapshotAttrs, no supplier — event carries Collections.emptyMap()
}

Each event enqueues with attrs = {} and the three FullKeys are:

• Event 1: (checkout, on, alloc1, false, null, alice@example.com, "", false)
• Event 2: (checkout, on, alloc1, false, null, alice@example.com, "", false)
• Event 3: (checkout, on, alloc1, false, null, alice@example.com, "", false)

These three are byte-for-byte identical, so they all hash to the same bucket.

Final state: one full-tier bucket, count = 3.

At flush, includeRawContext = isFullTier && observeFullEvaluationData is false, and resolveTargetingKey hashes:

row 1: {targeting_key: "sha256_b4698f9b...9e51b", count: 3, first_ms: 1000, last_ms: 3000}

Results in one row on the wire. All three raw contexts (which contained the same PII-adjacent request_id values you'd want protected) are gone. They were never allocated, never sorted, never stored.

Implementation in three commits

• 9e5253b — aggregator skips context work on the protected path.
• 79c8009 — hook skips context capture on the hot path when consent is off
• c4b9a82 — observeFullEvaluationData is now part of FullKey / DegradedKey

[leoromanovsky]

Two conditions to resolve please.

[leoromanovsky]

Good thinking

[leoromanovsky]

The full aggregation key includes the evaluation context. This behavior is correct when observeFullEvaluationData=true.

For example, these evaluations have different contexts:

  targeting key: alice@example.com
  context 1: {plan: pro, request_id: req-1}
  context 2: {plan: pro, request_id: req-2}
  context 3: {plan: pro, request_id: req-3}

When full evaluation data is enabled, the SDK emits each context. Separate aggregation buckets are necessary.

When observeFullEvaluationData=false, the SDK omits the evaluation context. All three evaluations then have the same visible identity:

  {
    "flag": {"key": "checkout"},
    "variant": {"key": "enabled"},
    "targeting_key": "sha256_<alice-hash>"
  }

The current implementation still uses request_id in the aggregation key. It creates three buckets, although the emitted context is identical.

The protected path also performs unnecessary operations. It:

• copies the evaluation context
• flattens the evaluation context
• sorts and prunes its fields
• creates a canonical context key
• stores the context until the next flush
• removes the context during serialization.

High-cardinality fields can create many unnecessary buckets. Examples include request IDs, timestamps, and correlation IDs.

The per-flag limit is 10,000 full buckets. After this limit, the SDK moves later evaluations to degraded aggregation. Degraded events omit both the context and targeting key.

When observeFullEvaluationData=false, the SDK must:

• keep the targeting key in the aggregation key
• omit the evaluation context from the aggregation key
• avoid evaluation-context processing
• avoid storing the evaluation context.

Different subjects must remain in separate buckets:

  alice@example.com -> sha256_<alice>
  bob@example.com   -> sha256_<bob>

Different contexts for the same subject must use one protected bucket.

👉 Overall this feels like a "P1" to me; see if it is straight-forward to resolve.

[leoromanovsky]

You had a good goal to seal the full evaluation metatadata to the configuration at the time of evaluation. The configuration you are accessing is mutable by other threads. Read it earlier than the hook, in the evaluation.

[dd-oleksii]

minor: given that the CURRENT_CONFIG is atomic, the result of this function is outdated as soon as it returns. Snapshotting "at evaluation time" is not enough to ensure they are in-sync. Ideally, the caller gets the flag from the configuration they are going to use for evaluation

[vjfridge]

Thanks for raising @dd-oleksii! I reworked to follow y'alls advice and isObserveFullEvaluationDataEnabled() was removed in 933cf3c. The hook now reads consent from evaluation metadata collected by DDEvaluator.evaluate() against the exact UFC it used. No caller reads the gateway for consent anymore.