Skip to content

Automate ScanCode license category updates - #178

Open
elrayle wants to merge 2 commits into
elr/license-categories-data-onlyfrom
elr/license-categories-scancode-updates
Open

elrayle wants to merge 2 commits into
elr/license-categories-data-onlyfrom
elr/license-categories-scancode-updates

Conversation

@elrayle

@elrayle elrayle commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fourth and final layer of native stack #179, based directly on #177 (elr/license-categories-data-only, verified parent d3b0f4df730fbdd79bf6903d88205c42af10d8b4). Stack order and PR bases remain #175 → #176 → #177 → #178.

  • Extend the existing daily Fetch Licenses workflow at 04:05 UTC to check SPDX and ScanCode independently, regenerate npm data after SPDX extraction, and retain one serialized auto-update-licenses PR against main.
  • Add -update-scancode: resolve upstream main once to a full immutable commit, download that revision once, compute SHA-256, and generate from the same bytes. Advance pin/provenance even for upstream documentation-only changes; unchanged revisions still verify the recorded checksum.
  • Validate Go, checked-in generated data, and npm tests before the scheduled updater creates its PR. Include the pin/checksum and generated npm data in that PR; distinguish category payload changes from provenance-only changes in its description.
  • Preserve manual overrides, exclusions, Unknown warnings, unsupported-category failures, source provenance, -verify, and layer 3's literal-data-only publication policy. Verification cannot be combined with pin updates.

No version bumps, auto-merges, publication changes, remote workflow dispatches, registry/configuration changes, or SPDX identifier expansion. The checked-in ScanCode pin and generated data remain unchanged in this implementation PR.

File scope

  • .github/workflows/fetch-licenses.yaml: independent ScanCode refresh, npm regeneration/diff inclusion, pre-PR validation, serialization, and explicit update paths.
  • cmd/license-categories/main.go: integrate refresh mode and rename defaultScancodeSource to currentScancodeSource without replacing parent provenance/verification logic.
  • cmd/license-categories/update_scancode.go and update_scancode_test.go: immutable API/archive refresh, AST-based pin replacement, and offline success/error/checksum/verification-mode cases.
  • cmd/pr_body.go and pr_body_test.go: ScanCode commits and category-payload summary in the existing update PR description.
  • packages/license-categories/docs/operations.md: concise cron operations documentation; retain existing publication/override guidance.

Validation

Passed again after syncing the corrected parent:

  • go test -race ./...
  • go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.11.4 run ./... (exact CI version, 0 issues)
  • npm test --prefix packages/license-categories (all 26 parent cases pass)
  • make verify-license-categories
  • go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 .github/workflows/*.yaml
  • Changed-Go-only formatting and git diff --check

The initial CI run exposed two distinct causes: CI's newer gosec reported G703 on the updater test's fixed fixture write under t.TempDir; a one-line, narrowly scoped annotation documents why this test path is safe without weakening production scanning. Package validation separately inherited a HEAD~1 assumption from layer 3's Actions-entrypoint test, which fails in shallow checkouts. That test was fixed in #177 using an isolated Git fixture and known mismatching tag; this layer incorporates that corrected parent without changing its test or checkout policy.

Also repeated offline updater/PR-body cases under the race detector. A disposable isolated checkout refreshed to live immutable ScanCode commit f10c818a2fbf0cd57a7c1b6cadd86a863408183c, recorded archive SHA-256 047188d4a4f2e62c497e63e74f8ebdeccbf889d2bee200fca60b18cd52bd6ed2, verified generated data, and successfully reran with the unchanged pin. That live update changed provenance only; the unchanged layer-3 comparator confirmed identical literal license data. The fixture was removed without changing this branch's reviewed pin.

Local validation used Go 1.26.4 and Node 26.4.0; Actions retain their declared Go version and Node 24. Existing missing-metadata warnings remain explicit. Scheduled execution and GitHub-token PR creation were not run remotely; repository Actions settings may need to permit workflow-created PRs as documented.

Copilot AI adaptive review requested due to automatic review settings November 11, 2026 17:44
@elrayle
elrayle added this pull request to stack #179 November 11, 2026 17:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The workflow counts checkout-HEAD stamp changes as updates, causing recurring header-only pull requests even when the license data and ScanCode pin are unchanged.

1 open issue
What changed in this PR

This PR completes the license-category update stack by integrating ScanCode refreshes and npm data generation into the existing daily SPDX update workflow.

Changes:

  • Adds immutable ScanCode revision resolution, archive checksums, and pin updates while preserving verification and category overrides.
  • Serializes automated updates and validates Go, generated data, and npm tests before creating a PR.
  • Extends PR descriptions and operations documentation to distinguish category-data changes from provenance-only updates.
File Description
packages/​license-categories/​docs/​operations.md Documents scheduled updates, validation, and operational requirements.
cmd/​pr_body.go Adds ScanCode commit and category-payload summaries.
cmd/​pr_body_test.go Tests payload and provenance-only summaries.
cmd/​license-categories/​update_scancode.go Resolves immutable revisions, computes checksums, and replaces pins.
cmd/​license-categories/​update_scancode_test.go Adds offline refresh, failure, checksum, and verification-mode tests.
cmd/​license-categories/​main.go Integrates refresh mode with generation and verification.
.github/​workflows/​fetch-licenses.yaml Adds serialized ScanCode updates, npm regeneration, and pre-PR validation.

Review effort: Balanced (auto)

genfiles_changed=true
# --quiet: Exits with 1 if there are changes; otherwise, exits with 0
git diff --quiet -- spdxexp/spdxlicenses/*.go && genfiles_changed=false
git diff --quiet -- spdxexp/spdxlicenses/*.go cmd/license-categories/main.go packages/license-categories/src/generated.ts && genfiles_changed=false
@elrayle
elrayle force-pushed the elr/license-categories-scancode-updates branch from 7eed4fa to b8541a8 Compare November 11, 2026 17:50
@elrayle
elrayle force-pushed the elr/license-categories-data-only branch from d3b0f4d to 2238d31 Compare November 11, 2026 18:43
@elrayle
elrayle force-pushed the elr/license-categories-scancode-updates branch from b8541a8 to 216e82c Compare November 11, 2026 18:44
@elrayle
elrayle removed this pull request from stack #179 November 11, 2026 18:44
@elrayle
elrayle added this pull request to stack #184 November 11, 2026 18:45
@elrayle
elrayle force-pushed the elr/license-categories-data-only branch from 2238d31 to a991934 Compare November 11, 2026 21:03
@elrayle
elrayle force-pushed the elr/license-categories-scancode-updates branch from 216e82c to c373e53 Compare November 11, 2026 21:04
@elrayle
elrayle force-pushed the elr/license-categories-data-only branch from a991934 to a0b1264 Compare November 11, 2026 22:28
elrayle and others added 2 commits November 11, 2026 17:28
Extend the existing daily license updater to refresh the ScanCode pin and checksum, regenerate npm data, validate both outputs, and maintain one reviewable update PR. Preserve provenance verification and data-only publication policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Scope G703's path-traversal exclusion to the fixed fixture write under t.TempDir. The CI-pinned golangci-lint v2.11.4 reports this controlled test path; keep production scanning unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@elrayle
elrayle force-pushed the elr/license-categories-scancode-updates branch from c373e53 to 50de9b3 Compare November 11, 2026 22:29
@elrayle
elrayle removed this pull request from stack #184 November 12, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants