feat(data-pipeline): encode structured values by lloeki · Pull Request #2304 · DataDog/libdatadog

lloeki · 2026-07-30T11:02:37Z

AI-generated code disclosure: this PR was implemented with substantial AI assistance and manually reviewed and validated.

What does this PR do?

Introduces a validated, language-neutral structured-value token ABI and standalone MessagePack encoder. It supports nil, booleans, signed and unsigned integers, floats, UTF-8 strings, binary strings, arrays, and maps, with depth and shape validation.

Motivation

The opaque blob API in DataDog/libdatadog PR 2303 restores transitional parity but still requires every consumer to encode MessagePack. This stage moves validated structured-value encoding into libdatadog as part of APMSP-3219.

Additional Notes

Stack position: L4, current PR: DataDog/libdatadog PR 2304, stacked on L1 DataDog/libdatadog PR 2303.

Complete expected review and merge order for one libdatadog release:

Level	libdatadog PR	Matching dd-trace-rb consumer
L0	DataDog/libdatadog PR 2302	DataDog/dd-trace-rb PR 6131
L1	DataDog/libdatadog PR 2303	DataDog/dd-trace-rb PR 6130
L2	DataDog/libdatadog PR 2305	DataDog/dd-trace-rb PR 6129
L3	DataDog/libdatadog PR 2301	DataDog/dd-trace-rb PR 6134
L4 (current)	DataDog/libdatadog PR 2304	DataDog/dd-trace-rb PR 6132
L5	DataDog/libdatadog PR 2300	DataDog/dd-trace-rb PR 6133

L0-L3 are independent and can be reviewed in parallel. Merging them in the listed order is still recommended because it reduces integration and release-branch conflicts, and lands panic containment before the new payload surfaces. L4 is stacked on L1, and L5 is stacked on L4.

This PR remains a draft because its review and final rebase depend on L1. Its matching consumer DataDog/dd-trace-rb PR 6132 is also a draft pending this API and the single libdatadog/libdatadog-rb release. L5 remains draft until this PR is ready.

L0 tracks APMSP-3830; L1-L5 implement the payload work tracked by APMSP-3219.

How to test the change?

Prior local validation completed successfully:

nix develop -c cargo test -p libdd-data-pipeline-ffi structured_value passed structured-value encoding, ownership, malformed-stream, shape, and depth-limit tests.
nix develop -c cargo test -p libdd-data-pipeline-ffi passed the complete data-pipeline FFI suite.
nix develop -c cargo clippy -p libdd-data-pipeline-ffi --all-targets -- -D warnings passed.
nix develop .#nightly -c cargo fmt --all -- --check passed.
The release artifact built successfully with nix develop -c cargo run -p builder --bin release -- --out /tmp/opencode/libdatadog-m2/x86_64-linux.
The matching consumer's native suite passed 122 examples, AppSec passed 807 examples, and AI Guard passed 125 examples against the local artifact.
Full Standard and Steep checks passed on the matching consumer branch.

Expose a flat preorder token ABI that encodes nil, scalar, string, binary, array, and map values into an owned MessagePack blob. This lets C callers build structured metadata without implementing MessagePack or sharing the lifetime of their input buffers. Reject malformed token streams, invalid kinds, booleans, UTF-8, counts, and nesting deeper than 64 levels. Successful output contains exactly one value and remains valid until explicitly freed.

Exercise the generated C ABI from the packaged trace exporter example. Mutate the caller's source buffers after encoding and compare the blob with fixed MessagePack bytes to prove that output ownership is independent of the input lifetime. Mirror the ownership check in Rust and document that token byte slices are borrowed only for the synchronous encoding call.

github-actions · 2026-07-30T11:04:52Z

📚 Documentation Check Results

⚠️ 1336 documentation warning(s) found

📦 `libdd-data-pipeline-ffi` - 1336 warning(s)

Updated: 2026-07-30 12:05:55 UTC | Commit: ec4eee6 | missing-docs job results

github-actions · 2026-07-30T11:06:38Z

🔒 Cargo Deny Results

⚠️ 3 issue(s) found, showing only errors (advisories, bans, sources)

📦 `libdd-data-pipeline-ffi` - 3 error(s)

Show output

error[vulnerability]: Invalid pointer dereference in `fmt::Pointer` impl for `Atomic` and `Shared` when the underlying pointer is invalid
   ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:78:1
   │
78 │ crossbeam-epoch 0.9.18 registry+https://github.com/rust-lang/crates.io-index
   │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
   │
   ├ ID: RUSTSEC-2026-0204
   ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0204
   ├ Affected versions of `fmt::Display` dereference the underlying pointer. This causes a invalid pointer dereference e.g., when a pointer created with `Atomic::null` or `Shared::null`. `fmt::Debug` impls and pre-0.9 `fmt::Display` impls, which do not dereference pointers, are not affected by this issue.
   ├ Announcement: https://github.com/crossbeam-rs/crossbeam/pull/1276
   ├ Solution: Upgrade to >=0.9.20 (try `cargo update -p crossbeam-epoch`)
   ├ crossbeam-epoch v0.9.18
     └── crossbeam-deque v0.8.5
         └── rayon-core v1.12.1
             └── rayon v1.10.0
                 └── criterion v0.5.1
                     ├── libdd-common v5.1.0
                     │   ├── libdd-capabilities-impl v3.0.0
                     │   │   ├── libdd-data-pipeline v7.0.0
                     │   │   │   └── libdd-data-pipeline-ffi v38.0.0
                     │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │   ├── libdd-shared-runtime v2.0.0
                     │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │   │   ├── libdd-dogstatsd-client v4.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0
                     │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   ├── libdd-telemetry v6.0.0
                     │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   │   └── libdd-trace-stats v6.0.0 (*)
                     │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
                     │   │   ├── libdd-trace-stats v6.0.0 (*)
                     │   │   └── libdd-trace-utils v9.0.0
                     │   │       ├── libdd-data-pipeline v7.0.0 (*)
                     │   │       ├── (dev) libdd-data-pipeline-ffi v38.0.0 (*)
                     │   │       ├── libdd-trace-obfuscation v5.0.0
                     │   │       │   └── libdd-trace-stats v6.0.0 (*)
                     │   │       ├── libdd-trace-stats v6.0.0 (*)
                     │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
                     │   ├── libdd-common-ffi v38.0.0
                     │   │   └── libdd-data-pipeline-ffi v38.0.0 (*)
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-dogstatsd-client v4.0.0 (*)
                     │   ├── libdd-shared-runtime v2.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   ├── libdd-trace-obfuscation v5.0.0 (*)
                     │   ├── libdd-trace-stats v6.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-data-pipeline v7.0.0 (*)
                     ├── (dev) libdd-ddsketch v1.1.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   ├── libdd-telemetry v6.0.0 (*)
                     │   └── libdd-trace-stats v6.0.0 (*)
                     ├── (dev) libdd-trace-normalization v3.0.0
                     │   ├── libdd-data-pipeline v7.0.0 (*)
                     │   └── libdd-trace-utils v9.0.0 (*)
                     ├── (dev) libdd-trace-obfuscation v5.0.0 (*)
                     ├── (dev) libdd-trace-stats v6.0.0 (*)
                     └── (dev) libdd-trace-utils v9.0.0 (*)

error[unsound]: Rand is unsound with a custom logger using `rand::rng()`
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:226:1
    │
226 │ rand 0.8.5 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ unsound advisory detected
    │
    ├ ID: RUSTSEC-2026-0097
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0097
    ├ It has been reported (by [@lopopolo](https://github.com/lopopolo)) that the `rand` library is [unsound](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library) (i.e. that safe code using the public API can cause Undefined Behaviour) when all the following conditions are met:
      
      - The `log` and `thread_rng` features are enabled
      - A [custom logger](https://docs.rs/log/latest/log/#implementing-a-logger) is defined
      - The custom logger accesses `rand::rng()` (previously `rand::thread_rng()`) and calls any `TryRng` (previously `RngCore`) methods on `ThreadRng`
      - The `ThreadRng` (attempts to) reseed while called from the custom logger (this happens every 64 kB of generated data)
      - Trace-level logging is enabled or warn-level logging is enabled and the random source (the `getrandom` crate) is unable to provide a new seed
      
      `TryRng` (previously `RngCore`) methods for `ThreadRng` use `unsafe` code to cast `*mut BlockRng<ReseedingCore>` to `&mut BlockRng<ReseedingCore>`. When all the above conditions are met this results in an aliased mutable reference, violating the Stacked Borrows rules. Miri is able to detect this violation in sample code. Since construction of [aliased mutable references is Undefined Behaviour](https://doc.rust-lang.org/stable/nomicon/references.html), the behaviour of optimized builds is hard to predict.
    ├ Announcement: https://github.com/rust-random/rand/pull/1763
    ├ Solution: Upgrade to >=0.10.1 OR <0.10.0, >=0.9.3 OR <0.9.0, >=0.8.6 (try `cargo update -p rand`)
    ├ rand v0.8.5
      ├── libdd-common v5.1.0
      │   ├── libdd-capabilities-impl v3.0.0
      │   │   ├── libdd-data-pipeline v7.0.0
      │   │   │   └── libdd-data-pipeline-ffi v38.0.0
      │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
      │   │   ├── libdd-shared-runtime v2.0.0
      │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-data-pipeline-ffi v38.0.0 (*)
      │   │   │   ├── libdd-dogstatsd-client v4.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0
      │   │   │   │       └── libdd-data-pipeline v7.0.0 (*)
      │   │   │   ├── libdd-telemetry v6.0.0
      │   │   │   │   ├── libdd-data-pipeline v7.0.0 (*)
      │   │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   │   └── libdd-trace-stats v6.0.0 (*)
      │   │   ├── (dev) libdd-telemetry v6.0.0 (*)
      │   │   ├── libdd-trace-stats v6.0.0 (*)
      │   │   └── libdd-trace-utils v9.0.0
      │   │       ├── libdd-data-pipeline v7.0.0 (*)
      │   │       ├── (dev) libdd-data-pipeline-ffi v38.0.0 (*)
      │   │       ├── libdd-trace-obfuscation v5.0.0
      │   │       │   └── libdd-trace-stats v6.0.0 (*)
      │   │       ├── libdd-trace-stats v6.0.0 (*)
      │   │       └── (dev) libdd-trace-utils v9.0.0 (*)
      │   ├── libdd-common-ffi v38.0.0
      │   │   └── libdd-data-pipeline-ffi v38.0.0 (*)
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-dogstatsd-client v4.0.0 (*)
      │   ├── libdd-shared-runtime v2.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   ├── libdd-trace-obfuscation v5.0.0 (*)
      │   ├── libdd-trace-stats v6.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-data-pipeline v7.0.0 (*)
      ├── (dev) libdd-ddsketch v1.1.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   ├── libdd-telemetry v6.0.0 (*)
      │   └── libdd-trace-stats v6.0.0 (*)
      ├── (dev) libdd-trace-normalization v3.0.0
      │   ├── libdd-data-pipeline v7.0.0 (*)
      │   └── libdd-trace-utils v9.0.0 (*)
      ├── (dev) libdd-trace-stats v6.0.0 (*)
      ├── libdd-trace-utils v9.0.0 (*)
      └── proptest v1.5.0
          └── (dev) libdd-tinybytes v1.1.1
              ├── libdd-data-pipeline v7.0.0 (*)
              ├── libdd-data-pipeline-ffi v38.0.0 (*)
              ├── (dev) libdd-tinybytes v1.1.1 (*)
              └── libdd-trace-utils v9.0.0 (*)

error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:296:1
    │
296 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── tracing-appender v0.2.3
          └── libdd-log v1.0.0
              └── (dev) libdd-data-pipeline v7.0.0
                  └── libdd-data-pipeline-ffi v38.0.0

advisories FAILED, bans ok, sources ok

Updated: 2026-07-30 12:07:22 UTC | Commit: ec4eee6 | dependency-check job results

pr-commenter · 2026-07-30T11:07:13Z

Benchmarks

Comparison

Candidate

Candidate benchmark details

Baseline

Baseline benchmark details

datadog-official · 2026-07-30T11:16:24Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 91.62%
• Overall Coverage: 74.82% (-0.02%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 1c4db0d | Docs | Datadog PR Page | Give us feedback!}

dd-octo-sts · 2026-07-30T11:37:21Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	8.01 MB	8.01 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	87.20 MB	87.25 MB	+.05% (+51.79 KB) 🔍

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	98.45 MB	98.51 MB	+.05% (+52.65 KB) 🔍
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.78 MB	10.78 MB	+.01% (+1.94 KB) 🔍

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	26.06 MB	26.07 MB	+.04% (+11.00 KB) 🔍
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	89.18 KB	90.34 KB	+1.29% (+1.15 KB) ⚠️
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	188.08 MB	188.17 MB	+.04% (+96.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	979.11 MB	979.59 MB	+.04% (+492.22 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.47 MB	8.47 MB	+.05% (+5.00 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	89.18 KB	90.34 KB	+1.29% (+1.15 KB) ⚠️
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	25.05 MB	25.06 MB	+.06% (+16.00 KB) 🔍
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	49.83 MB	49.87 MB	+.06% (+33.07 KB) 🔍

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	22.62 MB	22.63 MB	+.05% (+12.50 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	90.58 KB	91.75 KB	+1.29% (+1.17 KB) ⚠️
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	192.55 MB	192.67 MB	+.06% (+120.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	970.38 MB	970.62 MB	+.02% (+245.27 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.54 MB	6.54 MB	+.06% (+4.50 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	90.58 KB	91.75 KB	+1.29% (+1.17 KB) ⚠️
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	26.91 MB	26.93 MB	+.08% (+24.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	47.43 MB	47.46 MB	+.06% (+32.72 KB) 🔍

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	77.70 MB	77.75 MB	+.05% (+47.05 KB) 🔍
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	8.92 MB	8.92 MB	+.04% (+4.00 KB) 🔍

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	93.26 MB	93.31 MB	+.04% (+47.68 KB) 🔍
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.85 MB	10.86 MB	+.04% (+4.88 KB) 🔍

lloeki · 2026-07-30T11:49:02Z

Draft dd-trace-rb example consumer: DataDog/dd-trace-rb#6132.

lloeki added 2 commits July 29, 2026 18:47

lloeki added the AI Generated PR largely written by AI tools label Jul 30, 2026

github-actions Bot added the data-pipeline label Jul 30, 2026

lloeki mentioned this pull request Jul 30, 2026

Encode meta struct values natively DataDog/dd-trace-rb#6132

Draft

lloeki force-pushed the lloeki/native-meta-struct-m1 branch from 69926bd to 27ad1af Compare July 30, 2026 12:02

lloeki force-pushed the lloeki/native-meta-struct-m2 branch from c402ab6 to 1c4db0d Compare July 30, 2026 12:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(data-pipeline): encode structured values - #2304

feat(data-pipeline): encode structured values#2304
lloeki wants to merge 2 commits into
lloeki/native-meta-struct-m1from
lloeki/native-meta-struct-m2

lloeki commented Jul 30, 2026 •

edited by atlassian Bot

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Jul 30, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

dd-octo-sts Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

lloeki commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

lloeki commented Jul 30, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Documentation Check Results

📦 libdd-data-pipeline-ffi - 1336 warning(s)

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔒 Cargo Deny Results

📦 libdd-data-pipeline-ffi - 3 error(s)

Uh oh!

pr-commenter Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Candidate

Baseline

Uh oh!

datadog-official Bot commented Jul 30, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Artifact Size Benchmark Report

Uh oh!

lloeki commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lloeki commented Jul 30, 2026 •

edited by atlassian Bot

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading

📦 `libdd-data-pipeline-ffi` - 1336 warning(s)

github-actions Bot commented Jul 30, 2026 •

edited

Loading

📦 `libdd-data-pipeline-ffi` - 3 error(s)

pr-commenter Bot commented Jul 30, 2026 •

edited

Loading

datadog-official Bot commented Jul 30, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

dd-octo-sts Bot commented Jul 30, 2026 •

edited

Loading