Merge forward 3006.x into 3007.x - #70063
Open
dwoz wants to merge 112 commits into
Open
Conversation
…ltstack#69758) zypper returns exit code 104 when a search matches nothing. The search-style calls in zypperpkg (search, _get_visible_patterns, _get_patches, and Wildcard._get_available_versions) run through cmd.run_all, whose default success_retcodes is [0], so the 104 was logged as a command failure even though the search itself completed normally and correctly reported that nothing was found. Add an ignore_not_found option to the _Zypper wrapper that whitelists 104 via success_retcodes for those search calls, so the exit code is treated as success and no ERROR is emitted. The found case (0) and all genuine error codes are left untouched. Fixes saltstack#58551
…saltstack#67975) (saltstack#69813) * Support dnf5 group list/info in pkg.group_list and pkg.group_info (saltstack#67975) Backport of the dnf5 group support from saltstack#67975 to 3006.x, grafted onto the current group functions so the saltstack#60276 multi-word member-group @-fallback is preserved. dnf5 changed the "group list" and "group info" output formats, which the yum/dnf parser did not understand, so the group functions (and pkg.group_installed) returned empty or wrong data on Fedora 41+ and RHEL/AlmaLinux 10. - group_list: parse the dnf5 "group list --hidden" table by tokenizing each row (ID plus the trailing yes/no Installed column) instead of a regex, so a group name that contains or ends in the word "yes"/"no" is not mistaken for the status column, and a row with trailing whitespace is not dropped. - group_info: parse the dnf5 "group info" format, where each package section carries its first member inline after the colon. That inline member is pulled into a dedicated variable rather than mutating the loop's line, blank members are skipped, and section lines are fully stripped on both ends. dnf5 environment and language groups live under a separate "environment" subcommand and are out of scope here, so those keys stay empty on dnf5. Validated end to end against live dnf5 5.4.2.1 on Fedora 44: group_list parses all 157 groups and group_info parses every package with none dropped. Co-authored-by: Greg Oster <oster@fween.ca> * Drop the spurious empty optional package from the group_info test The empty-member guard added in this PR correctly stops a blank line in the dnf/yum "groupinfo" output from being recorded as an empty-string package, so the existing test_group_info expectation no longer includes the leading "" in the gnome-desktop optional list. --------- Co-authored-by: Greg Oster <oster@fween.ca>
Update relenv to 0.22.18
Several fixes for `x509_v2`
…ooks Two coupled fixes that keep grouped pip-updates PRs from failing on 3006.x: 1. Python 3.9 caps. Dependabot raises the shared floor of several packages to a release that no longer supports Python 3.9 (jaraco.functools 4.5.0, jaraco.context 6.1.2, msgpack 1.2.1, more-itertools 11.0.0, pycparser 3.0, pythonnet 3.1.0, virtualenv 21.5.1, xmldiff 3.0, zipp 4.1.0). With a single unmarked floor the py3.9 lock targets become unresolvable. Split each into a python_version < '3.10' branch capped at the last 3.9-compatible release plus an open py>=3.10 branch, mirroring the existing cryptography/aiohttp/urllib3 splits. (cryptography and pyopenssl already carry all-Python caps here, so they need no split.) 2. Malformed pip-compile hooks. The Py3.13 ZeroMQ hooks (linux/freebsd/ darwin/windows) and the docs hook were missing their `- id: pip-compile` line, so YAML folded them into the preceding Py3.14 blocks as duplicate keys and the Py3.14 CI locks never regenerated. Restore the missing id lines, correct the py3.14->py3.13 file globs, and regenerate the affected Py3.14 locks.
The zenoss.monitored state set ret["changes"] to None on the already-monitored and failed-add code paths. The state output validator (OutputUnifier.content_check) requires changes to be a dictionary and raises "'Changes' should be a dictionary.", so both paths failed with result=False and that exception comment. Return an empty dict, which correctly means "no changes" and passes validation. Fixes saltstack#53966
_gen_keep_files filtered requisites with `"file" in comp`, which for a bare-string requisite ID degraded to a substring match. Any ID containing the substring "file" then hit `comp["file"]` and raised "TypeError: string indices must be integers". Guard the membership test with an isinstance check so only dict requisites are considered; bare strings are ignored instead of crashing. Fixes saltstack#53692 and Fixes saltstack#61042
The '^' list-override marker was only stripped when the target dict already contained a matching list to override. When a class defined an override list with no prior list present, dict_merge plain-assigned the value (including the nested "pillars" dict) by reference and never descended to the list, leaving a literal '^' element in the merged pillar. Descend into dicts and honour a leading '^' on a list in the key-absent branch as well. Fixes saltstack#50755
salt.utils.network.is_reachable_host only caught socket.gaierror, but socket.getaddrinfo raises UnicodeError (an idna "label too long" error, which is not a subclass of gaierror) when a name contains a DNS label longer than 63 characters. A long salt-ssh -E/--pcre target triggers this, so _expand_target crashed instead of treating the target as not a reachable host. Also catch UnicodeError and return False. Fixes saltstack#57207
Minion.pillar_refresh compiles a fresh pillar and rebinds self.opts["pillar"] to a new dict. A proxy minion packs the pillar into its proxy module loader once, at init, by reference (self.proxy.pack["__pillar__"] = self.opts["pillar"]). The rebind orphans the dict that pack still aliases, so already-loaded proxy modules keep serving the pillar they were first packed with until the proxy restarts. Re-pack the loader with the freshly compiled pillar after the rebind, guarded so a regular (non-proxy) minion is unaffected. This mirrors the existing deltaproxy __grains__ re-pack and avoids reload_modules(), so a module's connection state and __context__ are preserved. Deltaproxy is covered by the same change: handle_event dispatches each sub-proxy's pillar_refresh to that sub-proxy instance, so the re-pack runs per sub-proxy against its own loader and opts. Out of scope: __opts__, __opts__["pillar"] and values a module copied out of pillar during init() are load-time snapshots and stay stale under any non-reload fix. Fixes saltstack#58197
Minion.pillar_refresh calls module_refresh() at the top, rebuilding the execution-module loaders (functions/returners/executors/utils) from self.opts, and only afterwards compiles the new pillar and rebinds self.opts["pillar"]. Each exec loader snapshots opts["pillar"] by value when it is built, so those freshly rebuilt loaders capture the OLD pillar. A regular minion masks this: every job rebuilds the loaders through gen_modules() before running. A proxy minion's metaproxy job path never does, so an exec module keeps serving the previous refresh's __pillar__ until the next refresh_pillar -- the "run the same command twice, get two answers" symptom. Re-run module_refresh() after the rebind so the exec loaders are rebuilt against the freshly compiled pillar. It sits next to the saltstack#58197 proxy re-pack in the success branch and is gated on opts["proxy"], so a regular minion pays no extra loader rebuild and a failed compile triggers none. Fixes saltstack#59393
pillar_refresh is a coroutine method, so io_loop.run_sync can call it directly; the no-arg wrapper lambda tripped pylint's unnecessary-lambda (W0108) in the lint-tests job.
Pillar compilation loads execution modules (for salt[...] calls in pillar templates) with the incoming opts when file_client is "local". On the master that branch is always taken, and the incoming opts carry the master's id, so salt["match.compound"] and friends matched against the master rather than the minion the pillar was being compiled for (saltstack#58407). Load them with self.opts, which __gen_opts stamps with the target minion_id, matching the non-local branch. Masterless is unaffected (opts["id"] already equals the minion_id there). Fixes saltstack#58407
…ted example The saltstack#58407 fix makes execution modules called during pillar rendering resolve against the target minion, so the documented match.filter_by example (which omits the minion_id argument) now correctly matches the minion and gets the db role. The parametrized expectation still asserted the pre-fix web* roles, which contradicted the fix and failed in CI.
Adds functional coverage for salt.sdb.env (set/get via environment variables, including the setdefault no-overwrite behavior) and salt.sdb.yaml (get with nested/colon traversal, multi-file merge, missing keys, and the read-only set raising NotImplemented). These are the two sdb backends that require no external service. Refs saltstack#61260
salt.utils.data.filter_by ran every lookup_dict key through fnmatch.fnmatchcase, so a literal key containing glob metacharacters (the "[" and "]" in GPU/PCI model strings such as "GP104GL [Quadro P4000]") was parsed as a character class and never matched its own value, falling through to the default (saltstack#60976). Try an exact string comparison before the fnmatch fallback. The change is additive: literal keys now match and existing glob patterns are unaffected. This also covers grains.filter_by, pillar.filter_by, and match.filter_by, which all delegate here. Fixes saltstack#60976
wheel.key.gen passed keysize straight to salt.crypt.gen_keys, which feeds it to rsa.generate_private_key (key_size must be an int). The salt-api rest_cherrypy POST /keys endpoint passes keysize as a string (cherrypy form values are always strings), so a request specifying a keysize raised TypeError and returned a 500 (saltstack#56425). Coerce keysize to an int and enforce the 2048-bit minimum that gen's docstring already documents. Fixes saltstack#56425
The file.comment, file.append and file.prepend states read the target file as bytes and decode it with the system encoding purely to build a diff. A strict decode raised UnicodeDecodeError and aborted the state when the file contained bytes invalid in that encoding. Decode with errors="replace" since the result is only used for the diff. Fixes saltstack#50903
…ack#69139) The LazyLoader put Salt's own source directories on the global sys.path while a module body executed (__populate_sys_path, and the fpath_dirname append in _load_module). Any bare import issued while that module ran could then resolve to a same-named single-file Salt module and get cached in sys.modules for the life of the process. That is what broke napalm on modern Salt. Loading salt/utils/napalm.py runs its top-level `import napalm`, which reaches ncclient.transport, which does a bare `import ssh` to detect the optional ssh-python/libssh package. With salt/utils on sys.path that bound to salt/utils/ssh.py (a plain module, not the ssh-python package), so ncclient's `from ssh.channel import Channel` raised "'ssh' is not a package", `import napalm` failed, HAS_NAPALM was False, and the napalm proxy/execution modules never passed their __virtual__ gate. This is the root cause behind the "Proxymodule napalm is missing an init()" reports in saltstack#69139 (which saltstack#69330 only improved the error message for). Salt-internal modules are imported via their fully-qualified salt.* names, so they never needed sys.path; only external/custom module dirs do, so a custom module's bare sibling imports keep resolving. Skip appending any directory under SALT_BASE_PATH in both __populate_sys_path and the fpath_dirname append. As a side effect, a module whose optional same-named dependency is not installed no longer "loads" by importing itself.
…ack#69139) The unit tests monkeypatch SALT_BASE_PATH and use a synthetic shadow file. This drives the real minion_mods + utils loaders against the real SALT_BASE_PATH and asserts, from inside a module body executed mid-load, that no Salt-internal directory is ever placed on sys.path -- the name-independent guarantee that protects every salt/utils and salt/modules collision (dns, napalm, git, pip, consul, ...), not just ssh. The concrete salt/utils/ssh.py shadow is also asserted when a real top-level ssh is absent.
…race Serialize concurrent NAPALM calls on a shared device connection (saltstack#55332)
Fix handling of GeneralNames and basicConstraints in x509_v2
…adroom-configurable Add opt-in cgroup-aware minion_memory_headroom / minion_memory_max (saltstack#69884)
…-08-14-26 # Conflicts: # .github/workflows/ci.yml # .github/workflows/nightly.yml # .github/workflows/scheduled.yml # .github/workflows/staging.yml # .pre-commit-config.yaml # cicd/shared-gh-workflows-context.yml # pkg/windows/nsis/installer/Salt-Minion-Setup.nsi # requirements/base.txt # requirements/static/ci/py3.14/darwin.lock # requirements/static/ci/py3.14/docs.lock # requirements/static/ci/py3.14/freebsd.lock # requirements/static/ci/py3.14/windows.lock # salt/fileclient.py # salt/loader/lazy.py # salt/minion.py # salt/states/file.py # salt/transport/tcp.py # salt/utils/event.py # salt/utils/http.py # tests/pytests/unit/loader/test_loader.py # tests/pytests/unit/utils/test_http.py
The revert of PR saltstack#69622 (`2746c4549a0`) removed the mis-merged 3008.x content from 3007.x's diff, but `git revert -m 1` does not rewrite history — the 3008.x commits, including the `v3008.1` release commit, remain reachable in the graph. `salt.version.__discover_version` uses `git describe --tags --long --match v[0-9]*`, which then picks up `v3008.1` and reports the branch as `3008.1+714.g<sha>`. That in turn breaks docs (rst_prolog's ``|current_release_doc|`` substitutes ``/topics/releases/3008.1``, a file that does not exist on 3007.x, and the man build fails with 13 sphinx warnings-as-errors) and cascades into every downstream build job that depends on the tarball. Constrain the match to `v3007.*` on this branch so tag detection only considers 3007.x releases. `git describe` now returns `v3007.14-<ahead>-g<sha>` as intended.
The three merge-forwarded tests carry 3006.x's `import salt.ext.tornado.*` which was removed from 3007.x. Rewrite to plain `tornado.*` so lint-tests passes and the tests can be collected on 3007.x. Same fix that was in the previous merge-forward attempt, dropped when we reset to the clean (reverted) 3007.x baseline.
Bump the bundled onedir Python from 3.10.20 to 3.11.15 on 3007.x, matching the 3006.x change from saltstack#69526. Python 3.10 reaches end of security support in October 2026, while Salt 3007.x must continue shipping security fixes past that date. Aligning the bundled interpreter with 3.11 keeps Salt's security release cadence on a supported CPython. The change is also load-bearing for CI on this merge-forward: relenv 0.22.18 (picked up from 3006.x) regressed the Windows CPython 3.10 native build, which surfaces as a libzmq zmq_assert abort on every salt-call subprocess started from the onedir (STATUS_FATAL_APP_EXIT out of libzmq's zmq.cpp:988 with EINVAL/EAGAIN). The 3006.x nightly proves relenv 0.22.18 + Python 3.11.15 is healthy on Windows. The cicd shared context drives every workflow's python-version input; the regenerated ci/nightly/scheduled/staging workflows pick up the new value via tools/precommit/workflows.py.
The py3.11 CI lockfile pulls jsonschema 4.26.0, which regressed several
message strings the ConfigTestCase suite asserts on:
* ``is not one of`` (oneOf/anyOf enum failure) -> generic
``is not valid under any of the given schemas``
* ``is too short`` (minItems) -> ``should be non-empty``
* ``is not allowed for`` (not) -> ``should not be valid under ...``
* ``'X' is a required property`` when nested inside anyOf -> generic
``is not valid under any of the given schemas``
The existing ``JSONSCHEMA_VERSION >= 3.0.0`` branches assumed the 3.x
wording carried forward; narrow them to the 3.x range and let 4.x fall
through to the pre-3.x generic message, and add a new gated branch for
the ``not``/``minItems`` cases.
The hostname and date-time format-checker tests can't run under jsonschema
4.x with only the deps we ship in CI: 4.x delegates ``hostname`` to the
optional ``fqdn`` package and ``date-time`` to ``rfc3339-validator`` /
``isoduration``, and our lockfiles only carry the legacy ``strict-rfc3339``
that 3.x used. Skip them on 4.x with a clear reason rather than pull new
optional deps into the CI matrix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.