wrap_sysconfig re-wraps without an idempotence guard, causing 2ⁿ slowdown that hangs Salt highstates indefinitely

[bchill]

wrap_sysconfig re-wraps without an idempotence guard, causing 2ⁿ slowdown that hangs Salt highstates indefinitely

Summary

relenv.runtime.wrap_sysconfig() reassigns its wrappers onto the same cached sysconfig module object every time it is called, with no check for whether that module has already been wrapped. Each call therefore adds another wrapper layer around sysconfig.get_config_var, get_config_vars and get_paths.

Because get_config_vars_wrapper calls the function it wraps twice per invocation, N accumulated layers cost 2ⁿ calls to the innermost function.

Anything that re-executes the relenv .pth — most notably importlib.reload(site) — adds a layer. Salt calls exactly that in salt/state.py: module_refresh(). On a long-running Salt job the layers accumulate until a single sysconfig.get_config_var() call effectively never returns, and the process spins at 100% CPU forever.

Measured on a wedged Salt minion: 46 layers. At that depth one get_config_var call requires roughly six years.

Versions

relenv	0.22.14
Salt	3008.2 (onedir)
Python	3.14
OS	Ubuntu 24.04 (arm64, Raspberry Pi 5) and Rocky 9 (x86_64, VM)

Short-lived processes and routine runs sit at the flat end of the curve and are genuinely unaffected. Only a long-lived process that accumulates enough refreshes crosses the knee — and when it does, it goes from fine to never-finishing within about eight layers.

Every Salt onedir installation is exposed to this; most simply never reach the depth where it becomes visible.

Suggested fix

Minimal, verified:

def wrap_sysconfig(name: str) -> ModuleType:
    module: ModuleType = importlib.import_module("sysconfig")
    mod = cast("Any", module)
# Already wrapped (e.g. after importlib.reload(site) re-ran the .pth):
# re-wrapping stacks another layer, and layers cost 2**n.
if getattr(mod.get_config_var, "__wrapped__", None) is not None:
    return module

mod.get_config_var = get_config_var_wrapper(mod.get_config_var)
...

functools.wraps already sets __wrapped__, so the marker exists.

Verified: with this guard in place, a host that had never once completed a highstate completed one in ~10 minutes, running through every SLS. Depth stays at 1 throughout. Same host, same states, no other change.

Worth considering beyond the minimal fix

1. bootstrap() installs other wrappers too — get_paths, the pip wrappers, build_ext.finalize_options. These are presumably accumulating layers by the same mechanism; only get_config_var was measured here. A guard inside bootstrap() would cover the whole class rather than one function.

2. The double func call in get_config_vars_wrapper looks unintentional. The first call's result is used to build config_vars, which is then assigned to mod._CONFIG_VARS, and the second call presumably re-reads it. Making it single-call would remove the exponential factor independently of the idempotence guard — defence in depth.

3. RELENV_BUILDENV short-circuits the doubling, but it is a build-time facility (it also gates include-dir injection for C extensions and RPATH relocation in install_wheel_wrapper, and flips get_config_vars away from returning the system toolchain's AR/CC/CFLAGS). It should not be suggested as a workaround.

Workaround for anyone hitting this now

Patch wrap_sysconfig as above in /opt/saltstack/salt/lib/python3.14/site-packages/relenv/runtime.py, then restart the minion — job children are forked from the long-lived parent and inherit its already-imported module objects, so an on-disk edit has no effect until the parent re-imports.

Note the patch lives inside the onedir and is silently reverted by any salt-minion package upgrade.