feat(data-pipeline): add meta struct blob setter by lloeki · Pull Request #2303 · DataDog/libdatadog

lloeki · 2026-07-30T11:02:37Z

AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

What does this PR do?

Adds an opaque per-key meta_struct blob setter to TracerSpan. Keys are validated as UTF-8, while values are copied as uninterpreted binary data. This transitional API lets consumers pass their existing per-key MessagePack blobs without changing the Rust trace model.

Motivation

Native exporter consumers cannot currently populate meta_struct, which blocks payload parity tracked by APMSP-3219. The opaque setter provides the safe first stage before the language-neutral structured-value APIs.

Additional Notes

Stack position: L1, current PR: DataDog/libdatadog PR 2303.

Complete expected review and merge order for one libdatadog release:

Level	libdatadog PR	Matching dd-trace-rb consumer
L0	DataDog/libdatadog PR 2302	DataDog/dd-trace-rb PR 6131
L1 (current)	DataDog/libdatadog PR 2303	DataDog/dd-trace-rb PR 6130
L2	DataDog/libdatadog PR 2305	DataDog/dd-trace-rb PR 6129
L3	DataDog/libdatadog PR 2301	DataDog/dd-trace-rb PR 6134
L4	DataDog/libdatadog PR 2304	DataDog/dd-trace-rb PR 6132
L5	DataDog/libdatadog PR 2300	DataDog/dd-trace-rb PR 6133

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4. L4 and L5 remain drafts while their prerequisite APIs are reviewed and rebased; their matching consumer PRs are also drafts pending the single libdatadog/libdatadog-rb release.

L0 tracks APMSP-3830; L1-L5 implement the payload work tracked by APMSP-3219.

How to test the change?

Prior local validation completed successfully:

nix develop -c cargo test -p libdd-data-pipeline-ffi passed, including arbitrary binary values, overwrite semantics, null handles, and invalid UTF-8 keys.
nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings passed.
nix develop .#nightly -c cargo fmt --all -- --check passed.
The generated C header was rebuilt through nix develop -c cargo run --bin release -- --out /tmp/opencode/native-meta-struct-libdatadog/x86_64-linux and consumed successfully by the matching dd-trace-rb branch.
nix develop .#ruby34 --command bundle exec rspec spec/datadog/tracing/transport/native/tracer_span_spec.rb spec/datadog/tracing/transport/native/conformance_spec.rb spec/datadog/tracing/transport/native/transport_spec.rb spec/datadog/appsec/actions_handler/serializable_backtrace_spec.rb passed 116 examples, including AppSec and AI Guard-shaped wire conformance.
Targeted Standard and Steep checks for the native transport passed.

Expose a C setter that copies an opaque binary value into a span's `meta_struct` map. Native tracers can now supply pre-encoded MessagePack without transferring ownership or requiring Rust to interpret it. Reject null span handles and non-UTF-8 keys. Reusing a key replaces its value, and the copied bytes remain owned by the span after the call.

github-actions · 2026-07-30T11:04:48Z

📚 Documentation Check Results

⚠️ 1346 documentation warning(s) found

📦 `libdd-data-pipeline-ffi` - 1346 warning(s)

Updated: 2026-07-30 12:05:02 UTC | Commit: 534bf0b | missing-docs job results

github-actions · 2026-07-30T11:06:13Z

🔒 Cargo Deny Results

⚠️ 3 issue(s) found, showing only errors (advisories, bans, sources)

📦 `libdd-data-pipeline-ffi` - 3 error(s)

Show output

error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:78:1
   │
78 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── libdd-common v5.1.0
                     │   ├── libdd-capabilities-impl v3.0.0
                     │   │   ├── libdd-data-pipeline v7.0.0
                     │   │   │   └── libdd-data-pipeline-ffi v38.0.0
                     │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │   ├── libdd-shared-runtime v2.0.0
                     │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │   │   ├── libdd-dogstatsd-client v4.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0
                     │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-telemetry v6.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
                     │   │   ├── libdd-trace-stats v6.0.0 (*)
                     │   │   └── libdd-trace-utils v9.0.0
                     │   │       ├── libdd-data-pipeline v7.0.0 (*)
                     │   │       ├── (dev) libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │       ├── libdd-trace-obfuscation v5.0.0
                     │   │       │   └── libdd-trace-stats v6.0.0 (*)
                     │   │       ├── libdd-trace-stats v6.0.0 (*)
                     │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
                     │   ├── libdd-common-ffi v38.0.0
                     │   │   └── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-dogstatsd-client v4.0.0 (*)
                     │   ├── libdd-shared-runtime v2.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   ├── libdd-trace-obfuscation v5.0.0 (*)
                     │   ├── libdd-trace-stats v6.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-data-pipeline v7.0.0 (*)
                     ├── (dev) libdd-ddsketch v1.1.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   └── libdd-trace-stats v6.0.0 (*)
                     ├── (dev) libdd-trace-normalization v3.0.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-trace-obfuscation v5.0.0 (*)
                     ├── (dev) libdd-trace-stats v6.0.0 (*)
                     └── (dev) libdd-trace-utils v9.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:228:1
    │
228 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── libdd-common v5.1.0
      │   ├── libdd-capabilities-impl v3.0.0
      │   │   ├── libdd-data-pipeline v7.0.0
      │   │   │   └── libdd-data-pipeline-ffi v38.0.0
      │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
      │   │   ├── libdd-shared-runtime v2.0.0
      │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
      │   │   │   ├── libdd-dogstatsd-client v4.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0
      │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-telemetry v6.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
      │   │   ├── libdd-trace-stats v6.0.0 (*)
      │   │   └── libdd-trace-utils v9.0.0
      │   │       ├── libdd-data-pipeline v7.0.0 (*)
      │   │       ├── (dev) libdd-data-pipeline-ffi v38.0.0 (*)
      │   │       ├── libdd-trace-obfuscation v5.0.0
      │   │       │   └── libdd-trace-stats v6.0.0 (*)
      │   │       ├── libdd-trace-stats v6.0.0 (*)
      │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
      │   ├── libdd-common-ffi v38.0.0
      │   │   └── libdd-data-pipeline-ffi v38.0.0 (*)
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-dogstatsd-client v4.0.0 (*)
      │   ├── libdd-shared-runtime v2.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   ├── libdd-trace-obfuscation v5.0.0 (*)
      │   ├── libdd-trace-stats v6.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-data-pipeline v7.0.0 (*)
      ├── (dev) libdd-ddsketch v1.1.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   └── libdd-trace-stats v6.0.0 (*)
      ├── (dev) libdd-trace-normalization v3.0.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-trace-stats v6.0.0 (*)
      ├── libdd-trace-utils v9.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-tinybytes v1.1.1
              ├── libdd-data-pipeline v7.0.0 (*)
              ├── libdd-data-pipeline-ffi v38.0.0 (*)
              ├── (dev) libdd-tinybytes v1.1.1 (*)
              └── libdd-trace-utils v9.0.0 (*)

error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:298:1
    │
298 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── tracing-appender v0.2.3
          └── libdd-log v1.0.0
              └── (dev) libdd-data-pipeline v7.0.0
                  └── libdd-data-pipeline-ffi v38.0.0

advisories FAILED, bans ok, sources ok

Updated: 2026-07-30 12:06:50 UTC | Commit: 534bf0b | dependency-check job results

pr-commenter · 2026-07-30T11:08:02Z

Benchmarks

Comparison

Candidate

Candidate benchmark details

Baseline

Baseline benchmark details

datadog-prod-us1-6 · 2026-07-30T11:16:22Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 84.31%
• Overall Coverage: 74.85% (+0.03%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 27ad1af | Docs | Datadog PR Page | Give us feedback!}

dd-octo-sts · 2026-07-30T11:36:50Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	8.01 MB	8.01 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	87.20 MB	87.20 MB	+0% (+7.24 KB) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.78 MB	10.78 MB	+0% (+336 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	98.45 MB	98.46 MB	+0% (+7.56 KB) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	26.06 MB	26.06 MB	+0% (+1.00 KB) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.50 KB	+.35% (+324 B) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	188.08 MB	188.08 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	979.11 MB	979.11 MB	+0% (+4.85 KB) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.47 MB	8.47 MB	+.02% (+2.00 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.50 KB	+.35% (+324 B) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	25.05 MB	25.05 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	49.83 MB	49.84 MB	+.01% (+6.02 KB) 🔍

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	22.62 MB	22.62 MB	+0% (+1.50 KB) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.90 KB	+.35% (+330 B) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	192.55 MB	192.56 MB	+0% (+8.00 KB) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	970.38 MB	970.38 MB	+0% (+4.51 KB) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.54 MB	6.54 MB	+.01% (+1.00 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.90 KB	+.35% (+330 B) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	26.91 MB	26.91 MB	+.02% (+8.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	47.43 MB	47.43 MB	+.01% (+5.71 KB) 🔍

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	77.70 MB	77.71 MB	+0% (+5.83 KB) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	8.92 MB	8.92 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	93.26 MB	93.27 MB	+0% (+5.98 KB) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.85 MB	10.85 MB	+0% (+240 B) 👌

lloeki · 2026-07-30T11:49:01Z

Draft dd-trace-rb example consumer: DataDog/dd-trace-rb#6130.

lloeki requested a review from a team as a code owner July 30, 2026 11:02

lloeki added the AI Generated PR largely written by AI tools label Jul 30, 2026

github-actions Bot added the data-pipeline label Jul 30, 2026

lloeki mentioned this pull request Jul 30, 2026

Export meta struct through native transport DataDog/dd-trace-rb#6130

Draft

lloeki force-pushed the lloeki/native-meta-struct-m1 branch from 69926bd to 27ad1af Compare July 30, 2026 12:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(data-pipeline): add meta struct blob setter - #2303

feat(data-pipeline): add meta struct blob setter#2303
lloeki wants to merge 1 commit into
mainfrom
lloeki/native-meta-struct-m1

lloeki commented Jul 30, 2026 •

edited by atlassian Bot

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-6 Bot commented Jul 30, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

Uh oh!

dd-octo-sts Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

lloeki commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

lloeki commented Jul 30, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Documentation Check Results

📦 libdd-data-pipeline-ffi - 1346 warning(s)

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔒 Cargo Deny Results

📦 libdd-data-pipeline-ffi - 3 error(s)

Uh oh!

pr-commenter Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Candidate

Baseline

Uh oh!

datadog-prod-us1-6 Bot commented Jul 30, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Artifact Size Benchmark Report

Uh oh!

lloeki commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lloeki commented Jul 30, 2026 •

edited by atlassian Bot

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading

📦 `libdd-data-pipeline-ffi` - 1346 warning(s)

github-actions Bot commented Jul 30, 2026 •

edited

Loading

📦 `libdd-data-pipeline-ffi` - 3 error(s)

pr-commenter Bot commented Jul 30, 2026 •

edited

Loading

datadog-prod-us1-6 Bot commented Jul 30, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

dd-octo-sts Bot commented Jul 30, 2026 •

edited

Loading