[VL][CI] Fix Spark 4.1 CI crash: isolate libgluten's bundled libstdc++

[jackylee-ch]

What changes were proposed in this pull request?

The spark-test-spark41 / spark-test-spark41-slow jobs in Velox Backend (x86) have been failing consistently (exit 134 / SIGABRT) since the mid-July GitHub-hosted runner image rollout (tracked in #12534). The forked Surefire JVM crashes with SIGSEGV during Velox backend initialization, before the first native test (ColumnarBatchTest) even runs:

# C  [libgluten.so+0xdb73a4]  std::__codecvt_utf16_base<char16_t>::do_unshift(...)+0x4
# siginfo: si_code: SEGV_MAPERR, si_addr: 0x0000000000000030
Java frames:
 org.apache.gluten.init.NativeBackendInitializer.initialize(...)
 org.apache.gluten.backendsapi.velox.VeloxListenerApi.onExecutorStart(...)
 org.apache.gluten.test.VeloxBackendTestBase.setup()

Root cause (verified with readelf / addr2line on the CI artifact)

libgluten.so and libvelox.so both statically link the gcc-13 libstdc++ (dev/vcpkg/toolchain.cmake: -static-libstdc++ -static-libgcc).

• libvelox has a local: * catch-all in its symbols.map, so its bundled libstdc++ symbols are demoted to STB_LOCAL and never take part in global symbol resolution.
• libgluten cannot use local: * — it must keep Arrow/substrait exported for libvelox to link against — so its ~5000 bundled libstdc++ symbols stay in the dynamic symbol table. Critically, the std::locale::id objects that index the facet table are emitted as STB_GNU_UNIQUE.

Inside the centos-9 image, the JVM pulls in the system gcc-11 libstdc++.so.6.0.29. The dynamic loader collapses libgluten's STB_GNU_UNIQUE locale ids into a single process-wide definition, so libgluten indexes its own gcc-13 facet table with the wrong id, reads a bogus num_put/codecvt facet pointer, and dereferences null+0x30.

The older centos-8 system libstdc++ (gcc-8) is not new enough to interpose, which is exactly why only the centos-9 spark41 jobs crashed while spark33…spark40 (centos-8), built from the very same native artifact, stayed green. This is not a code regression from any merged PR.

Fix

Pass -Wl,--exclude-libs for the bundled C++ runtime archives (libstdc++.a, libgcc.a, libgcc_eh.a) when linking libgluten. This demotes those symbols to STB_LOCAL — which also clears the STB_GNU_UNIQUE binding — giving libgluten the same self-contained libstdc++ isolation libvelox already has, without hiding the Arrow/substrait symbols libvelox resolves against.

-Bsymbolic alone does not work here, because STB_GNU_UNIQUE overrides it (verified in a separate CI run).

The CI workflow is unchanged — the Spark 4.1 jobs keep running on centos-9-jdk17. The fix is purely in the native link step, so it addresses the image itself rather than side-stepping it.

How was this patch tested?

Validated on a debug branch that trims the workflow to only the affected jobs and rebuilds the native libs with this change, keeping the container on centos-9-jdk17:

• spark-test-spark41 — the SIGSEGV is gone; ColumnarBatchTest, arrow batched python udf over parquet scan, and the full backends-velox / gluten-ut suites run
• spark-test-spark41-slow — success

Run: https://github.com/jackylee-ch/gluten/actions/runs/29676262109

Closes #12534

[Copilot]

Pull request overview

This PR adjusts the Velox backend (x86) GitHub Actions workflow to address consistent Spark 4.1 CI crashes occurring in the centos-9-jdk17 container by moving Spark 4.1 test jobs back to the centos-8-jdk17 image, and adds a Miniconda-based Python 3.11 setup for Spark 4.1 Python UDF requirements.

Changes:

• Switch spark-test-spark41 and spark-test-spark41-slow jobs from apache/gluten:centos-9-jdk17 to apache/gluten:centos-8-jdk17.
• Replace CentOS 9 dnf Python 3.11 installation with a Miniconda Python 3.11 install for the Spark 4.1 job prepare step.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jackylee-ch]

Resolved: this revision drops the miniconda approach entirely. The Spark 4.1 jobs stay on centos-9-jdk17 with the original dnf install python3.11 Prepare step (unchanged from main), so there is no longer any symlink mutation of /usr/bin/python3. The workflow file is now identical to main; the only change in this PR is the native link step in cpp/core/CMakeLists.txt.

[jackylee-ch]

Resolved: the miniconda download is gone in this revision. The Spark 4.1 jobs stay on centos-9-jdk17 and use the same dnf install python3.11 step as main, so there is no longer any curl|bash of an external installer. The only change in this PR is the native link step in cpp/core/CMakeLists.txt.

[philo-he]

@jackylee-ch, do you mean CentOS-7-built Velox static lib hitting an incompatibility issue on Centos-9 at runtime? The apache/gluten:centos-9-jdk17 docker has been used for Spark 4.1 for a long time. Curious what Velox code changes make this issue happen recently.

The PR looks good as a quick fix. Not sure if we still have strong reason to use Centos 9 for Spark 4.1 tests (introduced by #11519). cc @zhouyuan

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[jackylee-ch]

@philo-he It's not Velox code. libgluten's symbols.map has no local: *, so it exports its bundled gcc-13 libstdc++ symbols (locale facet ids/vtables) globally. When the process also loads the system gcc-11 libstdc++.so.6, those get interposed and the facet layout mismatches → crash. It's been latent; a recent image/runner change (system libstdc++ now new enough to interpose) exposed it. centos-8's older libstdc++ can't interpose, so it hid the bug.

I'm trying another way to fix the problem.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[jackylee-ch]

@philo-he @zhouyuan I would merge this for a quick fix, you can check and resolve it later.