Localize vendored dependency symbols in the prebuilt static archive - #28
Open
mightypirate1 wants to merge 1 commit into
Open
Localize vendored dependency symbols in the prebuilt static archive#28mightypirate1 wants to merge 1 commit into
mightypirate1 wants to merge 1 commit into
Conversation
liblbug.a bundles its whole vendored C/C++ dependency tree (zstd, lz4, simsimd, CRoaring, mbedtls, yyjson, brotli...) with default GLOBAL symbol visibility. Any consumer whose link graph carries another copy of one of those libraries fails with duplicate-symbol errors - and on Rust >=1.82, where we link with `+whole-archive`, every one of those symbols is force-included regardless of whether lbug's own code reaches it, so the consumer has no way to avoid pulling them in. On the static-prebuilt, linux-gnu path, post-process the downloaded archive once per cache key: `ld -r --whole-archive` it into one relocatable object (binding its internal references first), then `objcopy -w --localize-symbol` every vendored prefix, then re-archive. Only lbug's own C API (the lbug_/connection_/query_result_/... names in include/lbug_rs.h and include/lbug_arrow.h) stays globally visible. The processed archive is cached beside the raw one and reused across builds; if ld/objcopy/ar are missing or any step fails, we fall back to the unprocessed archive with a cargo:warning, so this can never turn a build that works today into one that doesn't. Other platforms are untouched: this is a GNU binutils flow with no direct equivalent on Mach-O or MSVC.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
liblbug.a bundles its whole vendored C/C++ dependency tree (zstd, lz4, simsimd, CRoaring, mbedtls, yyjson, brotli...) with default GLOBAL symbol visibility. Any consumer whose link graph carries another copy of one of those libraries fails with duplicate-symbol errors — and on Rust >=1.82, where we link with +whole-archive, every one of those symbols is force-included regardless of whether lbug's own code reaches it, so the consumer has no way to avoid pulling them in.
Real case: a Rust workspace that links simsimd via another embedded database engine and ZSTD_* via async-compression/zstd-sys cannot link lbug into the same binary. The linker reports 140+ duplicate symbols before hitting its error cap; a full symbol-table cross-reference found 259 colliding names (173 zstd-family, 86 simsimd).
What this does: on the static-prebuilt, linux-gnu path, post-process the downloaded archive once per cache key: ld -r --whole-archive it into one relocatable object (binding its internal references first), then objcopy -w --localize-symbol every vendored prefix, then re-archive. Only lbug's own C API (the lbug_/connection_/query_result_/... names in include/lbug_rs.h and include/lbug_arrow.h) stays globally visible. The processed archive is cached beside the raw one and reused across builds; if ld/objcopy/ar are missing or any step fails, we fall back to the unprocessed archive with a cargo:warning, so this can never turn a build that works today into one that doesn't.
Tested:
Scope: other platforms are untouched — this is a GNU binutils flow with no direct equivalent on Mach-O or MSVC; follow-ups there are possible. C++-mangled vendored symbols (antlr4, httplib, the vendored parquet reader) are deliberately not touched — some are already renamed into lbug_-prefixed namespaces upstream for exactly this reason, and localizing mangled RTTI symbols is a different, riskier problem than this plain-C-ABI fix.
The deeper fix — building the vendored tree with hidden visibility when producing the prebuilt itself — belongs in the main LadybugDB/ladybug release pipeline; this is the consumer-side mitigation until then.