Skip to content

test(agent): cover policy management end to end - #1981

Open
Benoît Cortier (CBenoit) wants to merge 13 commits into
cbenoit-policy-audit-eventsfrom
cbenoit-policy-e2e-support
Open

Benoît Cortier (CBenoit) wants to merge 13 commits into
cbenoit-policy-audit-eventsfrom
cbenoit-policy-e2e-support

Conversation

@CBenoit

Copy link
Copy Markdown
Member

Exercises Agent policy management through protected standard-user and LocalSystem clients without weakening executable, ACL, path, or process-identity checks.

Covers read and validation behavior, standard-user write denial, Create, Update, Repair, stale conflict rejection, exact ConfirmOverwrite, restart persistence, durable managed authority, and observable audit outcomes. The Windows CI job runs both identities and the feature-gated route authorization tests.

Local PsExec execution could not install PSEXESVC, so the identity scenarios are delegated to CI; local builds, workspace tests, Clippy, formatting, runner parsing, and range validation pass.

Extracted from the production-shaped E2E support in #1963.

@CBenoit

Copy link
Copy Markdown
Member Author

Implementation notes:

  • Extends the existing production-shaped Agent policy tester without adding production bypasses or changing store, authentication, installer, or audit implementation.
  • Stages the tester under protected LocalSystem-owned ACLs before running either a restricted standard-user token or LocalSystem.
  • Covers validation, standard-user denial, Create, Update, Repair, stale Reject, exact ConfirmOverwrite, restart persistence, durable managed authority, and audit trace observability while retaining existing unsafe-path and watcher-token regressions.
  • CI runs both identity modes and the feature-gated package-broker authorization tests.
  • Local PsExec could not install PSEXESVC, so identity scenarios await Windows CI. Full serial workspace tests, Clippy, formatting, locked builds, runner parsing, and range checks pass locally.

Note

Human-tuned, LLM-assisted content.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The standard-user scenario runs both client and Agent under the same restricted identity, leaving the production cross-identity authorization boundary untested.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds end-to-end Agent policy-management coverage across restricted and LocalSystem execution modes.

Changes:

  • Expands lifecycle, authorization, persistence, and audit tests.
  • Adds protected unelevated test orchestration.
  • Runs identity and feature-gated route tests in Windows CI.

Documentation-specific review was not applicable.

File summaries
File Description
crates/agent-policy-tester/src/windows.rs Adds identity modes and policy lifecycle scenarios.
crates/agent-policy-tester/run-unelevated.ps1 Runs the tester with a restricted token.
crates/agent-policy-tester/run-as-system.ps1 Selects elevated test mode.
crates/agent-policy-tester/Cargo.toml Adds Windows identity dependencies.
Cargo.lock Records dependency updates.
.github/workflows/ci.yml Runs both identities and route tests.
Review details
  • Files reviewed: 5/6 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread crates/agent-policy-tester/src/windows.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The security-sensitive Windows identity and ACL orchestration requires confirmation from the Windows CI execution.

Review details
  • Files reviewed: 5/6 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The psexec -l client runs at Low integrity and may be denied by the named pipe before testing the intended authorization response.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 5/6 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread crates/agent-policy-tester/run-unelevated.ps1 Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failed readiness validation can leave LocalSystem test processes running into later CI steps.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 5/6 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread crates/agent-policy-tester/run-unelevated.ps1 Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Exercise policy management through a protected standard-user client and a LocalSystem client without weakening executable or path checks.

Cover validation and write denial, managed Create, Update, Repair, stale conflicts, confirmed overwrite, restart persistence, durable managed authority, and observable audit outcomes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run the protected policy tester under a restricted standard-user token before the existing LocalSystem lifecycle.

Keep feature-gated route authorization tests in the same Windows job so the development signature feature cannot hide them from the default workspace suite.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the Agent and test server under LocalSystem while a distinct restricted process exercises the named-pipe management endpoints.

Coordinate readiness and shutdown through a protected, read-only test directory so authorization regressions cannot pass by inspecting the server token.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Treat PsExec's detached-process PID as diagnostic output and use bounded, validated readiness as the authoritative launch result.

Publish readiness atomically, preserve launch diagnostics, and keep shutdown and cleanup idempotent so orchestration errors remain actionable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Launch policy authorization requests from a unique temporary standard-user account instead of a Low-integrity PsExec token.

Require the exact account SID and Medium mandatory integrity level, keep credentials out of arguments and logs, and remove the account and profile after the bounded run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Own LocalSystem test-server shutdown as soon as detached launch is attempted, even when readiness validation fails.

Signal the unique protected stop marker with a direct fallback, preserve the original scenario error, and bound status collection before cleanup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Cover the current policy contract, receipt invalidation, warnings, and
interrupted Repair recovery. Exercise actual installer conversion and
managed authority under the hosted LocalSystem gate without accepting
skipped privileged tests or incomplete server completion status.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pass the restored package root explicitly to the LocalSystem test
invocation so generated imports discover the test SDK and produce its
required privileged-test result.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the restored NuGet imports and the product Agent filename
when running the privileged installer lifecycle tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run migration lifecycle probes through the staged trusted test client
instead of the hosted .NET test process, whose executable image cannot
be retained by the package broker.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove installer migration and retired-policy compatibility coverage while
preserving the canonical policy management lifecycle and its security
boundaries.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exercise the final canonical policy rule shapes and validation semantics
through the end-to-end policy management harness.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Three moderate test-coverage gaps must be addressed before approval.

Review details

Suppressed comments (3)

crates/agent-policy-tester/run-unelevated.ps1:350

  • The new self-test does not exercise the cleanup guarantee it claims to cover. $launchAttempted is assigned true immediately before each Wait-ServerReadiness call and is only tested against itself; no code invokes the surrounding finally or verifies that Signal/status cleanup runs after failed readiness. A regression that removes or relocates $serverLaunchAttempted = $true could therefore pass SelfTest. Refactor the orchestration into a testable function with a fake launcher, or add an integration assertion that failed readiness still signals and observes server shutdown.
        $launchAttempted = $true
        Set-Content -LiteralPath $ready -Value (
            @{
                Nonce = "wrong-nonce"
                PipeName = "\\.\pipe\test"
                ServerPid = 100
                ServerSid = "S-1-5-18"
                AgentPid = 200
                AgentSid = "S-1-5-18"
            } | ConvertTo-Json -Compress
        )
        try {
            Wait-ServerReadiness -Path $ready -ServerStatusPath $status -ExpectedNonce "nonce" `
                -TimeoutMilliseconds 50 -LaunchValue 6256 -LaunchDiagnostics "started detached process" | Out-Null
            throw "Mismatched-readiness simulation unexpectedly succeeded"
        } catch {
            if ($_ -notmatch "readiness nonce mismatch" -or -not $launchAttempted) {

crates/agent-policy-tester/src/windows.rs:921

  • The standard-user leg only exercises /v1/policy/management and /v1/policy/validate; it never calls GET /v1/policy, even though the policy-management contract keeps reads available to an authenticated standard user. A regression that makes active-policy reads administrator-only would therefore pass this scenario. Add a GET assertion here (the policy is missing in this harness, so expect HTTP 404).
    let management = policy_management_by_pipe(pipe_name).await?;
    ensure!(management["State"] == "Missing", "expected a missing policy");

crates/agent-policy-tester/src/windows.rs:1129

  • This ConfirmOverwrite request uses current_token from the stale-conflict response, so the token is already fresh. It never verifies the contract that a stale token remains rejected when ConflictHandling is ConfirmOverwrite; a regression that turns this mode into an unconditional stale-token bypass would still pass. Preserve stale_token, submit it with ConfirmOverwrite, and assert 409/StalePolicyStoreToken before retrying with current_token.
    let confirmed = replace_policy_response(
        &agent,
        "Update",
        "ConfirmOverwrite",
        current_token.clone(),
  • Files reviewed: 5/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

Exercise standard-user read access, stale overwrite rejection, and
LocalSystem shutdown verification after failed readiness.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@CBenoit

Copy link
Copy Markdown
Member Author

Addressed the exact-head Copilot review coverage gaps in ff4031ba:

  • The standard-user path now verifies the canonical missing-policy GET /v1/policy response.
  • ConfirmOverwrite now explicitly rejects a stale store token before the current-token retry.
  • The runner SelfTest now simulates failed readiness with malformed server status and proves the production shutdown helper signals, consumes status, and preserves failure.

No inline review threads were created by the review to resolve.

Note

Human-tuned, LLM-assisted content.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The extensive multi-process Windows identity orchestration and security-boundary coverage require final human validation.

Review details
  • Files reviewed: 5/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants