test(debugger): add snapshot-correlation gate and Go weblog endpoints by grantseltzer · Pull Request #7425 · DataDog/system-tests

grantseltzer · 2026-07-30T18:30:01Z

Motivation

Changes

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-07-30T18:30:37Z

CODEOWNERS have been resolved as:

tests/debugger/test_debugger_snapshot_correlation.py                    @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/probes/probe_snapshot_log_correlation.json         @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils/probes/probe_snapshot_log_correlation_loop.json    @DataDog/debugger @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/debugger/test_debugger_probe_budgets.py                           @DataDog/debugger @DataDog/system-tests-core
tests/debugger/utils.py                                                 @DataDog/debugger @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/build/docker/golang/app/chi/debugger.go                           @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/chi/main.go                               @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/echo/debugger.go                          @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/echo/main.go                              @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/gin/debugger.go                           @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/gin/main.go                               @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http-orchestrion/debugger.go          @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http-orchestrion/main.go              @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http/debugger.go                      @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http/main.go                          @DataDog/dd-trace-go-guild @DataDog/system-tests-core

datadog-datadog-prod-us1

More details

The new correlation chain returns the expected value, and the loop handler safely returns Loop 0 for both zero and negative counts. The shared line-probe mapping targets the newly added loop body and sibling lines consistently across Go weblog variants.

Was this helpful? React 👍 or 👎

_{📊 Validated against 3 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 8431afa · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

nccatoni

LGTM (from @DataDog/system-tests-core) but you should get a review from someone familiar with the feature

tylfin · 2026-08-05T16:43:41Z

+@scenarios.debugger_probes_snapshot
+@slow
+class Test_Debugger_Coordinated_Sampling(debugger.BaseDebuggerTest):
+    """Tier 1: the sampling decision is made once per trace and applies to every probe that


I don't think this docstring is providing a lot of value on the test suite

tylfin · 2026-08-05T16:44:07Z

+        for probe in probes:
+            probe["id"] = debugger.generate_probe_id("log")
+            # Use the tracer's existing per-probe sampling config to sample below 100%.
+            probe["sampling"] = {"snapshotsPerSecond": _SNAPSHOTS_PER_SECOND}


Why do we need to configure this?

If you meant to point at dynamic_sampling_enabled=True that shouldn't have been added and I removed that change.

For this one, the test asserts at least one trace was fully dropped.

tylfin · 2026-08-05T16:44:50Z

+
+from utils import scenarios, features, slow
+
+# Snapshot rate for the correlation probes, set through the tracer's existing per-probe sampling


Do these constants need comments explaining them? I think the reader can infer what _SNAPSHOTS_PER_SECOND is...

tylfin · 2026-08-05T16:45:08Z

+
+
+def _trace_id(snapshot: dict) -> str | None:
+    return snapshot.get("dd", {}).get("trace_id")


Why does this need a helper?

It does not, removed

tylfin · 2026-08-05T16:46:42Z

+@features.debugger_probe_budgets
+@scenarios.debugger_probes_snapshot
+@slow
+class Test_Debugger_Per_Span_Budget(debugger.BaseDebuggerTest):


Does it really make sense to break these into separate classes?

Nope. Fixed.

tylfin · 2026-08-05T16:48:40Z

@@ -0,0 +1,267 @@
+# Unless explicitly stated otherwise all files in this repository are licensed under the the Apache License Version 2.0.
+# This product includes software developed at Datadog (https://www.datadoghq.com/).
+# Copyright 2021 Datadog, Inc.


Should be 2026- is this needed?

It's 2021 everywhere in this repo, are you sure it should be 26?

datadog-datadog-prod-us1 · 2026-08-07T22:23:49Z

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 6 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #2 / laravel11x 2
🧪 1 Test failed
tests.stats.test_stats.Test_Peer_Tags.test_peer_tags[laravel11x] from system_tests_suite
AssertionError: Client spans should have peer tags, found: []
assert 0 &gt; 0
 &#43;  where 0 = len([])

self = &lt;tests.stats.test_stats.Test_Peer_Tags object at 0x7faa45fc2000&gt;

    def test_peer_tags(self):
        &#34;&#34;&#34;Test that client spans include peer tags while server spans don&#39;t&#34;&#34;&#34;
        client_stats_found = False
        server_stats_found = False
...
DataDog/system-tests | K8S_LIB_INJECTION_PROFILING_OVERRIDE: [dd-lib-java-init-test-app, ${PRIVATE_DOCKER_REGISTRY}/system-tests/dd-lib-java-init-test-app, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/cluster-agent:7.81.0, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/apm-inject:latest, 3.231.1, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/dd-lib-java-init:latest_snapshot]

DataDog/system-tests | K8S_LIB_INJECTION_PROFILING_OVERRIDE: [dd-lib-java-init-test-app, ${PRIVATE_DOCKER_REGISTRY}/system-tests/dd-lib-java-init-test-app, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/cluster-agent:7.81.0, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/apm-inject:latest_snapshot, 3.231.1, 235494822917.dkr.ecr.us-east-1.amazonaws.com/ssi/dd-lib-java-init:latest_snapshot]
View all 6 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 86f3832 | Docs | Datadog PR Page | Give us feedback!}

datadog-datadog-prod-us1 · 2026-08-07T22:24:08Z

Bits found no code fix to apply

🟢 Investigated · ⚪ No code fix needed

K8S_LIB_INJECTION_PROFILING_OVERRIDE fails in the unchanged Java/Kubernetes profiling path: the profiler cannot start on the Alpine JVM because libgcc_s.so.1 is missing, and companion jobs lose access to the Kubernetes API server. No code change was made because the debugger tests and Go weblog endpoint additions do not execute or configure that path.

View in Datadog | Reviewed commit 86f3832 · Any feedback? Reach out in #deveng-pr-agent

grantseltzer mentioned this pull request Jul 30, 2026

debugger: snapshot correlation tests #7330

Closed

5 tasks

grantseltzer marked this pull request as ready for review August 3, 2026 13:58

grantseltzer requested review from a team as code owners August 3, 2026 13:58

grantseltzer requested review from dromanol, florentinl, manuel-alvarez-alvarez and ncybul and removed request for a team August 3, 2026 13:58

datadog-datadog-prod-us1 Bot reviewed Aug 3, 2026

View reviewed changes

nccatoni reviewed Aug 5, 2026

View reviewed changes

tylfin reviewed Aug 5, 2026

View reviewed changes

test(debugger): add snapshot-correlation gate and Go weblog endpoints

86f3832

grantseltzer force-pushed the grantseltzer/correlation-tests-go branch from 8431afa to 86f3832 Compare August 7, 2026 22:06


		from utils import scenarios, features, slow

		# Snapshot rate for the correlation probes, set through the tracer's existing per-probe sampling



		def _trace_id(snapshot: dict) -> str \| None:
		return snapshot.get("dd", {}).get("trace_id")

Uh oh!

Conversation

grantseltzer commented Jul 30, 2026

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

nccatoni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grantseltzer Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1 Bot commented Aug 7, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

datadog-datadog-prod-us1 Bot commented Aug 7, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bits found no code fix to apply

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jul 30, 2026 •

edited

Loading

grantseltzer Aug 7, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Aug 7, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

datadog-datadog-prod-us1 Bot commented Aug 7, 2026 •

edited by datadog-prod-us1-3 Bot

Loading