Skip to content

feat(data-pipeline): set structured span values - #2300

Draft
lloeki wants to merge 1 commit into
lloeki/native-meta-struct-m2from
lloeki/native-meta-struct-m3
Draft

feat(data-pipeline): set structured span values#2300
lloeki wants to merge 1 commit into
lloeki/native-meta-struct-m2from
lloeki/native-meta-struct-m3

Conversation

@lloeki

@lloeki lloeki commented Jul 30, 2026 •
edited by atlassian Bot
Loading

Copy link
Copy Markdown
Member

AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

What does this PR do?

Adds direct structured-value insertion into TracerSpan.meta_struct, reusing the validator and encoder from DataDog/libdatadog PR 2304 while moving the encoded allocation directly into the span. The opaque blob and standalone encoder APIs remain available.

Motivation

The standalone encoder still requires consumers to manage an intermediate encoded blob and an additional value copy. The direct setter provides the final efficient API for structured payload parity tracked by APMSP-3219.

Additional Notes

Stack position: L5, current PR: DataDog/libdatadog PR 2300, stacked on L4 DataDog/libdatadog PR 2304.

Complete expected review and merge order for one libdatadog release:

Level libdatadog PR Matching dd-trace-rb consumer
L0 DataDog/libdatadog PR 2302 DataDog/dd-trace-rb PR 6131
L1 DataDog/libdatadog PR 2303 DataDog/dd-trace-rb PR 6130
L2 DataDog/libdatadog PR 2305 DataDog/dd-trace-rb PR 6129
L3 DataDog/libdatadog PR 2301 DataDog/dd-trace-rb PR 6134
L4 DataDog/libdatadog PR 2304 DataDog/dd-trace-rb PR 6132
L5 (current) DataDog/libdatadog PR 2300 DataDog/dd-trace-rb PR 6133

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

This PR remains a draft because its review and final rebase depend on L4 and, transitively, L1. Its matching consumer DataDog/dd-trace-rb PR 6133 is also a draft pending this API and the single libdatadog/libdatadog-rb release. L4 remains draft until L1 is ready.

L0 tracks APMSP-3830; L1-L5 implement the payload work tracked by APMSP-3219.

How to test the change?

Prior local validation completed successfully:

  • nix develop --command cargo test -p libdd-data-pipeline-ffi --locked passed 68 tests, including direct-setter/standalone-encoder equivalence, atomic failure, invalid handles, overwrite behaviour, and ownership.
  • nix develop --command cargo clippy -p libdd-data-pipeline-ffi --all-targets --locked -- -D warnings passed.
  • nix develop .#nightly --command cargo fmt --all -- --check passed.
  • The release artifact built successfully with nix develop -c cargo run --bin release -- --out /tmp/opencode/libdatadog-m3/x86_64-linux.
  • The matching consumer's native suite passed 122 examples, and AppSec passed 807 examples against the local artifact.
  • nix develop .#ruby40 --command nix shell nixpkgs#valgrind --command bundle exec ruby_memcheck vendor/bundle/ruby/4.0.0/gems/rspec-core-3.13.6/exe/rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/send_traces_spec.rb spec/datadog/tracing/transport/native/trace_exporter_spec.rb passed with no Valgrind memory errors.
  • Full Standard and Steep checks passed on the matching consumer branch.

@lloeki
feat(data-pipeline): set structured span values
bad622f 
Add a span setter that accepts structured-value tokens and encodes them
directly into `meta_struct`. Native tracers can populate structured
metadata without managing an intermediate encoded blob or paying an
additional value copy.

Validate the key and complete token stream before replacing an existing
entry. Encoding failures and invalid handles therefore leave the span
unchanged.
@lloeki lloeki added the AI Generated PR largely written by AI tools label Jul 30, 2026
@github-actions

github-actions Bot commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

📚 Documentation Check Results

⚠️ 1336 documentation warning(s) found

📦 libdd-data-pipeline-ffi - 1336 warning(s)

Updated: 2026-07-30 12:06:04 UTC | Commit: f166fff | missing-docs job results

@github-actions

github-actions Bot commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔒 Cargo Deny Results

⚠️ 3 issue(s) found, showing only errors (advisories, bans, sources)

📦 libdd-data-pipeline-ffi - 3 error(s)

Show output 
error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:78:1
   │
78 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── libdd-common v5.1.0
                     │   ├── libdd-capabilities-impl v3.0.0
                     │   │   ├── libdd-data-pipeline v7.0.0
                     │   │   │   └── libdd-data-pipeline-ffi v38.0.0
                     │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │   ├── libdd-shared-runtime v2.0.0
                     │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │   │   ├── libdd-dogstatsd-client v4.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0
                     │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-telemetry v6.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
                     │   │   ├── libdd-trace-stats v6.0.0 (*)
                     │   │   └── libdd-trace-utils v9.0.0
                     │   │       ├── libdd-data-pipeline v7.0.0 (*)
                     │   │       ├── (dev) libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │       ├── libdd-trace-obfuscation v5.0.0
                     │   │       │   └── libdd-trace-stats v6.0.0 (*)
                     │   │       ├── libdd-trace-stats v6.0.0 (*)
                     │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
                     │   ├── libdd-common-ffi v38.0.0
                     │   │   └── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-dogstatsd-client v4.0.0 (*)
                     │   ├── libdd-shared-runtime v2.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   ├── libdd-trace-obfuscation v5.0.0 (*)
                     │   ├── libdd-trace-stats v6.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-data-pipeline v7.0.0 (*)
                     ├── (dev) libdd-ddsketch v1.1.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   └── libdd-trace-stats v6.0.0 (*)
                     ├── (dev) libdd-trace-normalization v3.0.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-trace-obfuscation v5.0.0 (*)
                     ├── (dev) libdd-trace-stats v6.0.0 (*)
                     └── (dev) libdd-trace-utils v9.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:226:1
    │
226 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── libdd-common v5.1.0
      │   ├── libdd-capabilities-impl v3.0.0
      │   │   ├── libdd-data-pipeline v7.0.0
      │   │   │   └── libdd-data-pipeline-ffi v38.0.0
      │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
      │   │   ├── libdd-shared-runtime v2.0.0
      │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
      │   │   │   ├── libdd-dogstatsd-client v4.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0
      │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-telemetry v6.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
      │   │   ├── libdd-trace-stats v6.0.0 (*)
      │   │   └── libdd-trace-utils v9.0.0
      │   │       ├── libdd-data-pipeline v7.0.0 (*)
      │   │       ├── (dev) libdd-data-pipeline-ffi v38.0.0 (*)
      │   │       ├── libdd-trace-obfuscation v5.0.0
      │   │       │   └── libdd-trace-stats v6.0.0 (*)
      │   │       ├── libdd-trace-stats v6.0.0 (*)
      │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
      │   ├── libdd-common-ffi v38.0.0
      │   │   └── libdd-data-pipeline-ffi v38.0.0 (*)
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-dogstatsd-client v4.0.0 (*)
      │   ├── libdd-shared-runtime v2.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   ├── libdd-trace-obfuscation v5.0.0 (*)
      │   ├── libdd-trace-stats v6.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-data-pipeline v7.0.0 (*)
      ├── (dev) libdd-ddsketch v1.1.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   └── libdd-trace-stats v6.0.0 (*)
      ├── (dev) libdd-trace-normalization v3.0.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-trace-stats v6.0.0 (*)
      ├── libdd-trace-utils v9.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-tinybytes v1.1.1
              ├── libdd-data-pipeline v7.0.0 (*)
              ├── libdd-data-pipeline-ffi v38.0.0 (*)
              ├── (dev) libdd-tinybytes v1.1.1 (*)
              └── libdd-trace-utils v9.0.0 (*)

error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:296:1
    │
296 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── tracing-appender v0.2.3
          └── libdd-log v1.0.0
              └── (dev) libdd-data-pipeline v7.0.0
                  └── libdd-data-pipeline-ffi v38.0.0

advisories FAILED, bans ok, sources ok

Updated: 2026-07-30 12:07:29 UTC | Commit: f166fff | dependency-check job results

@pr-commenter

pr-commenter Bot commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Comparison

Candidate

Candidate benchmark details

Baseline

Baseline benchmark details

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jul 30, 2026 •
edited by datadog-official Bot
Loading

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 92.16%
• Overall Coverage: 74.84% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bad622f | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 87.20 MB 87.26 MB +.07% (+63.38 KB) 🔍
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.01 MB 8.01 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.78 MB 10.78 MB +.02% (+2.39 KB) 🔍
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 98.45 MB 98.52 MB +.06% (+64.70 KB) 🔍
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 26.06 MB 26.07 MB +.05% (+14.00 KB) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 89.18 KB 90.63 KB +1.62% (+1.44 KB) ⚠️
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 188.08 MB 188.19 MB +.05% (+112.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 979.11 MB 979.60 MB +.05% (+506.49 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.47 MB 8.47 MB +.07% (+6.50 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 89.18 KB 90.63 KB +1.62% (+1.44 KB) ⚠️
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 25.05 MB 25.06 MB +.06% (+16.00 KB) 🔍
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 49.83 MB 49.87 MB +.08% (+40.99 KB) 🔍
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 22.62 MB 22.64 MB +.06% (+15.00 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 90.58 KB 92.05 KB +1.62% (+1.47 KB) ⚠️
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 192.55 MB 192.71 MB +.08% (+160.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 970.38 MB 970.68 MB +.03% (+305.41 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.54 MB 6.54 MB +.08% (+5.50 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 90.58 KB 92.05 KB +1.62% (+1.47 KB) ⚠️
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 26.91 MB 26.93 MB +.08% (+24.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 47.43 MB 47.47 MB +.08% (+40.26 KB) 🔍
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 77.70 MB 77.76 MB +.07% (+56.02 KB) 🔍
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.92 MB 8.92 MB +.04% (+4.00 KB) 🔍
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 93.26 MB 93.31 MB +.05% (+56.71 KB) 🔍
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.85 MB 10.86 MB +.04% (+5.21 KB) 🔍

@lloeki lloeki mentioned this pull request Jul 30, 2026
Draft
@lloeki

lloeki commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

Draft dd-trace-rb example consumer: DataDog/dd-trace-rb#6133.

@lloeki
lloeki force-pushed the lloeki/native-meta-struct-m2 branch from c402ab6 to 1c4db0d Compare July 30, 2026 12:03
@lloeki
lloeki force-pushed the lloeki/native-meta-struct-m3 branch from d8dad0b to bad622f Compare July 30, 2026 12:03
This was referenced Jul 30, 2026
Open
Open
Open
Draft
Open
Draft
Open
Draft
Open
Draft
Draft
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

AI Generated PR largely written by AI tools data-pipeline

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lloeki