Skip to content

Localize vendored dependency symbols in the prebuilt static archive - #28

Open
mightypirate1 wants to merge 1 commit into
LadybugDB:mainfrom
mightypirate1:localize-vendored-symbols
Open

Localize vendored dependency symbols in the prebuilt static archive#28
mightypirate1 wants to merge 1 commit into
LadybugDB:mainfrom
mightypirate1:localize-vendored-symbols

Conversation

@mightypirate1

Copy link
Copy Markdown

liblbug.a bundles its whole vendored C/C++ dependency tree (zstd, lz4, simsimd, CRoaring, mbedtls, yyjson, brotli...) with default GLOBAL symbol visibility. Any consumer whose link graph carries another copy of one of those libraries fails with duplicate-symbol errors — and on Rust >=1.82, where we link with +whole-archive, every one of those symbols is force-included regardless of whether lbug's own code reaches it, so the consumer has no way to avoid pulling them in.

Real case: a Rust workspace that links simsimd via another embedded database engine and ZSTD_* via async-compression/zstd-sys cannot link lbug into the same binary. The linker reports 140+ duplicate symbols before hitting its error cap; a full symbol-table cross-reference found 259 colliding names (173 zstd-family, 86 simsimd).

What this does: on the static-prebuilt, linux-gnu path, post-process the downloaded archive once per cache key: ld -r --whole-archive it into one relocatable object (binding its internal references first), then objcopy -w --localize-symbol every vendored prefix, then re-archive. Only lbug's own C API (the lbug_/connection_/query_result_/... names in include/lbug_rs.h and include/lbug_arrow.h) stays globally visible. The processed archive is cached beside the raw one and reused across builds; if ld/objcopy/ar are missing or any step fails, we fall back to the unprocessed archive with a cargo:warning, so this can never turn a build that works today into one that doesn't.

Tested:

  • This crate's own suite against the processed archive: cargo test --release (158 tests) and --release --features arrow (166 tests), plus doctests — all passing.
  • The consumer reproduction: the workspace above fails to link with the unpatched crate and links cleanly with this branch, into a binary that also carries the other engine and the zstd-linking gRPC stack; that workspace's graph conformance suite passes against the processed archive, so localization changed nothing functionally.
  • After localization, nm shows zero vendored symbols remaining global and all 179 lbug_* API symbols intact.

Scope: other platforms are untouched — this is a GNU binutils flow with no direct equivalent on Mach-O or MSVC; follow-ups there are possible. C++-mangled vendored symbols (antlr4, httplib, the vendored parquet reader) are deliberately not touched — some are already renamed into lbug_-prefixed namespaces upstream for exactly this reason, and localizing mangled RTTI symbols is a different, riskier problem than this plain-C-ABI fix.

The deeper fix — building the vendored tree with hidden visibility when producing the prebuilt itself — belongs in the main LadybugDB/ladybug release pipeline; this is the consumer-side mitigation until then.

liblbug.a bundles its whole vendored C/C++ dependency tree (zstd, lz4,
simsimd, CRoaring, mbedtls, yyjson, brotli...) with default GLOBAL symbol
visibility. Any consumer whose link graph carries another copy of one of
those libraries fails with duplicate-symbol errors - and on Rust >=1.82,
where we link with `+whole-archive`, every one of those symbols is
force-included regardless of whether lbug's own code reaches it, so the
consumer has no way to avoid pulling them in.

On the static-prebuilt, linux-gnu path, post-process the downloaded
archive once per cache key: `ld -r --whole-archive` it into one
relocatable object (binding its internal references first), then
`objcopy -w --localize-symbol` every vendored prefix, then re-archive.
Only lbug's own C API (the lbug_/connection_/query_result_/... names in
include/lbug_rs.h and include/lbug_arrow.h) stays globally visible. The
processed archive is cached beside the raw one and reused across builds;
if ld/objcopy/ar are missing or any step fails, we fall back to the
unprocessed archive with a cargo:warning, so this can never turn a build
that works today into one that doesn't.

Other platforms are untouched: this is a GNU binutils flow with no
direct equivalent on Mach-O or MSVC.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant