build: lock Python runtime and development dependencies with hashes

[m-khan-97]

What changed

I am taking the dependency-reproducibility slice of #304. Production currently installs test tools and resolves transitive dependencies afresh; this makes a successful CI run less useful as evidence for the next deployment.

• Separate direct runtime, development and lock-tool inputs.
• Commit fully pinned SHA-256 locks, with development constrained to the production package versions.
• Require hashes and binary wheels in the container, native Render build commands, deployment workflow and relevant CI installs.
• Verify input/lock consistency within the existing rule-validation gate. The check operates in a temporary directory and preserves compatible pins rather than upgrading on every CI run.
• Add 12 regression checks, including offline good/bad wheel hashes, stale-lock handling, runtime/dev separation and CI wiring.
• Document the update process and declare Python 3.11 as the repository's native-build default, matching CI and Docker.

Existing exact direct runtime pins are preserved. Previously open ranges and transitive dependencies now have explicit versions. No scanner features, live cloud settings, review-policy changes or new vulnerability exceptions are included.

Validation

• Clean Python 3.11/Linux development installation with --require-hashes --only-binary=:all: and pip check passed.
• Separate clean runtime installation and API/scanner imports passed; pytest, coverage, ruff and pip-tools are absent from that environment.
• The runtime environment also built the container's BM25 artifact successfully (984 chunks). A subsequent targeted hallucination-guard run passed 2 tests and skipped 2 requiring an AI API key; no live LLM was called.
• Input/lock consistency check passed.
• Full local pytest suite: 994 passed, 9 skipped; 86.50% API/scanner coverage, above the existing 80% gate. Skips were three missing-BM25-index tests and six PostgreSQL-dependent cases. One existing pytest iterator-deprecation warning remains.
• All 12 dependency-lock tests passed, including rejection of a mismatched wheel hash without network access.
• Repository-wide ruff check and format check passed; git diff --check passed.
• Runtime pip-audit with the existing CI exception arguments reported no known vulnerabilities. No exceptions were added or relaxed.

Limits and review notes

Docker is unavailable locally, so the existing container runtime/scan CI job must validate the image. Live Azure, PostgreSQL integration and deployment were not exercised locally. Before deployment, verify that a Render dashboard PYTHON_VERSION does not override .python-version.

This does not claim a fully reproducible container: base-image/OS packages and the existing pip/setuptools/wheel bootstrap remain separate work. Security-scanner tooling also retains its existing independent environments.

Refs #304 — intentionally not closing it. Deployment topology, release provenance, backup/restore drills and operations acceptance criteria remain open.

Dependencies

Depends-On: none

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Scorecard details

Package	Version	Score	Details
pip/pip	26.1.2	Unknown	Unknown
pip/build	1.6.0	Unknown	Unknown
pip/click	8.5.0	Unknown	Unknown
pip/packaging	26.3	Unknown	Unknown
pip/pip-tools	7.5.3	Unknown	Unknown
pip/pyproject-hooks	1.2.0	Unknown	Unknown
pip/setuptools	83.0.0	Unknown	Unknown
pip/wheel	0.46.3	Unknown	Unknown
pip/azure-common	1.1.28	🟢 6.9	Details Check Score Reason Maintained 🟢 10 30 commit(s) and 4 issue activity found in the last 90 days -- score normalized to 10 Code-Review 🟢 9 Found 22/23 approved changesets -- score normalized to 9 CII-Best-Practices 🟢 5 badge detected: Passing Security-Policy 🟢 10 security policy file detected Packaging ⚠️ -1 packaging workflow not detected License 🟢 10 license file detected Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Signed-Releases ⚠️ -1 no releases found Branch-Protection 🟢 5 branch protection is not maximal on development and all release branches Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0 Binary-Artifacts 🟢 8 binaries present in source code Pinned-Dependencies 🟢 5 dependency not pinned by hash detected -- score normalized to 5 Fuzzing 🟢 10 project is fuzzed
pip/azure-core	1.41.0	Unknown	Unknown
pip/azure-mgmt-core	1.6.0	🟢 6.9	Details Check Score Reason Maintained 🟢 10 30 commit(s) and 4 issue activity found in the last 90 days -- score normalized to 10 Code-Review 🟢 9 Found 22/23 approved changesets -- score normalized to 9 CII-Best-Practices 🟢 5 badge detected: Passing Security-Policy 🟢 10 security policy file detected Packaging ⚠️ -1 packaging workflow not detected License 🟢 10 license file detected Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Signed-Releases ⚠️ -1 no releases found Branch-Protection 🟢 5 branch protection is not maximal on development and all release branches Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0 Binary-Artifacts 🟢 8 binaries present in source code Pinned-Dependencies 🟢 5 dependency not pinned by hash detected -- score normalized to 5 Fuzzing 🟢 10 project is fuzzed
pip/blinker	1.9.0	Unknown	Unknown
pip/cffi	2.1.1	Unknown	Unknown
pip/charset-normalizer	3.5.1	Unknown	Unknown
pip/click	8.5.0	Unknown	Unknown
pip/greenlet	3.5.5	Unknown	Unknown
pip/idna	3.19	Unknown	Unknown
pip/isodate	0.7.2	Unknown	Unknown
pip/itsdangerous	2.2.0	Unknown	Unknown
pip/jinja2	3.1.6	Unknown	Unknown
pip/mako	1.4.1	Unknown	Unknown
pip/markupsafe	3.0.3	Unknown	Unknown
pip/msal	1.38.0	🟢 7.2	Details Check Score Reason Packaging ⚠️ -1 packaging workflow not detected Maintained 🟢 10 19 commit(s) and 2 issue activity found in the last 90 days -- score normalized to 10 Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Token-Permissions 🟢 9 detected GitHub workflow tokens with excessive permissions Code-Review 🟢 10 all changesets reviewed Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 Binary-Artifacts 🟢 10 no binaries found in the repo Fuzzing 🟢 10 project is fuzzed CII-Best-Practices ⚠️ 2 badge detected: InProgress License 🟢 9 license file detected Signed-Releases ⚠️ -1 no releases found Security-Policy ⚠️ 0 security policy file not detected SAST 🟢 9 SAST tool is not run on all commits -- score normalized to 9 Branch-Protection ⚠️ 1 branch protection is not maximal on development and all release branches
pip/msal-extensions	1.3.1	Unknown	Unknown
pip/oauthlib	3.3.1	🟢 8	Details Check Score Reason Code-Review 🟢 7 Found 7/10 approved changesets -- score normalized to 7 Maintained 🟢 9 11 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 9 Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Security-Policy 🟢 10 security policy file detected Token-Permissions 🟢 9 detected GitHub workflow tokens with excessive permissions Binary-Artifacts 🟢 10 no binaries found in the repo CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Fuzzing 🟢 10 project is fuzzed Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection 🟢 6 branch protection is not maximal on development and all release branches Packaging 🟢 10 packaging workflow detected SAST 🟢 8 SAST tool is not run on all commits -- score normalized to 8
pip/packaging	26.3	Unknown	Unknown
pip/prometheus-client	0.26.0	Unknown	Unknown
pip/pycparser	3.0	Unknown	Unknown
pip/python-json-logger	4.2.0	Unknown	Unknown
pip/pyyaml	6.0.3	Unknown	Unknown
pip/requests-oauthlib	2.0.0	🟢 4.3	Details Check Score Reason Maintained ⚠️ 0 0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Code-Review 🟢 4 Found 4/9 approved changesets -- score normalized to 4 Security-Policy 🟢 10 security policy file detected Packaging ⚠️ -1 packaging workflow not detected Binary-Artifacts 🟢 10 no binaries found in the repo Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Fuzzing ⚠️ 0 project is not fuzzed License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ -1 internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/sentry-sdk	2.69.0	🟢 6	Details Check Score Reason Maintained 🟢 10 30 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10 Code-Review 🟢 10 all changesets reviewed Binary-Artifacts 🟢 10 no binaries found in the repo Packaging ⚠️ -1 packaging workflow not detected Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Pinned-Dependencies 🟢 10 all dependencies are pinned Fuzzing ⚠️ 0 project is not fuzzed License 🟢 10 license file detected Security-Policy 🟢 10 security policy file detected Branch-Protection 🟢 4 branch protection is not maximal on development and all release branches Signed-Releases ⚠️ 0 Project has not signed or included provenance with any releases. SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/six	1.17.0	🟢 3.7	Details Check Score Reason Code-Review ⚠️ 2 Found 7/30 approved changesets -- score normalized to 2 Binary-Artifacts 🟢 10 no binaries found in the repo Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Maintained ⚠️ 0 0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 Packaging ⚠️ -1 packaging workflow not detected Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Security-Policy ⚠️ 0 security policy file not detected Fuzzing 🟢 10 project is fuzzed License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/sqlalchemy	2.0.52	Unknown	Unknown
pip/typing-extensions	4.16.0	Unknown	Unknown
pip/urllib3	2.7.0	Unknown	Unknown
pip/werkzeug	3.1.8	Unknown	Unknown

Scanned Files

• requirements-lock.txt
• requirements.txt

[m-khan-97]

CI caught an integration mistake in my lock-check setup: creating .lock-tools inside the checkout caused the existing credential scan to inspect pip/setuptools placeholder strings. I have moved the tool environment into RUNNER_TEMP and added a regression assertion for that boundary. The credential scan and its patterns are unchanged. All 19 focused dependency/container-configuration tests and lint/format checks pass locally. The new commit triggers a fresh CI run.

[parthrohit22]

Reviewed 9358488. The three-tier split (runtime / dev constrained to -c requirements.txt / isolated lock tooling in requirements-lock.*) is the right shape, and the enforcement is consistent across the surfaces that matter: --require-hashes --only-binary=:all: in the Dockerfile, all four render.yaml build commands, the CI installs, and deploy.yml, each followed by pip check.

Verified locally:

• requirements.txt pins 66 packages, every one carrying --hash=sha256: lines; every direct dependency in requirements.in is present as an == pin in the lock.
• All 12 tests/test_dependency_locks.py pass, including the offline mismatched-wheel-hash rejection.
• scripts/lock_dependencies.py --check seeds the temp resolve with the committed .txt files, so only a genuine input change (not a new upstream release) makes the gate fail — that's the usual pip-tools-in-CI failure mode handled well.
• All 21 PR checks are green, including Container Scan (Trivy), SCA (pip-audit) with the existing exception args, and Rule & Compliance Validation (which runs the lock check).

Non-blocking:

1. The Install ruff lint step now installs the full requirements-dev.txt (runtime + ~40 Azure SDKs + pytest) under --require-hashes just to get ruff. It works, but a ruff-only pinned+hashed input would keep the lint job lean and independent of the runtime resolution.
2. lock_dependencies.py --check still needs PyPI reachable to re-resolve changed inputs; a registry outage fails that step even when the locks are correct. The seed-with-committed-locks logic keeps this to genuine input diffs, so acceptable — a one-line note in the CI step would set expectations.
3. Dockerfile's pip==26.1.2 setuptools==83.0.0 wheel==0.46.3 bootstrap is still unhashed. The PR body scopes this out explicitly; flagging only so it's a recorded boundary.
4. .python-version (3.11) is enforced only by lock_dependencies.py, and only for lock generation. The Dockerfile FROM and CI setup-python still name their versions independently. Consistent today; an assertion that the Dockerfile base tracks .python-version would harden against future drift.

None of these block. Approving.