Add test coverage messages, coverage threshold checks, and summary

[james-newell-forge]

Summary

Adds the RFC 019 code-coverage message and consumer model to Microsoft.Testing.Platform. Coverage producers can publish counts-based measurements, scope-aware threshold evaluations, and report references through the data-consumer pipeline. The platform correlates them into ITestCoverageResult, renders them in the terminal summary, and returns exit code 14 when a threshold fails.

Note that this PR was assisted with Claude Opus 4.8 and GitHub Copilot.

Public API

• CoverageMetric provides the seeded append-only metric vocabulary, including Custom.
• CoverageScope and CoverageScopeLevel identify overall, module, assembly, namespace, type, and file scopes.
• TestCoverageMessage reports CoveredCount and CoverableCount; Percentage is derived.
• TestCoverageThresholdMessage reports scope, metric, aggregation, actual/required percentages, no-data policy, and a derived Passed value.
• TestCoverageReportMessage references a typed rich coverage artifact.
• ITestCoverageResult exposes correlated scope summaries, thresholds, report references, and HasThresholdFailure for third-party consumers.

Terminal output

Test run summary: Passed!
  total: 22
  failed: 0
  succeeded: 22
  skipped: 0
  duration: 1s 288ms

  Code Coverage Summary:
    Total - Line: 96.6%
    Total - Branch: 77.7%
    Total - Method: 91.7%

  Coverage Threshold Results:
    Total - Line (Total over Module): 96.6% >= 70.0% threshold
    Total - Branch (Total over Module): 77.7% >= 70.0% threshold
    Total - Method (Total over Module): 91.7% >= 70.0% threshold

Scope names and custom metric names are normalized before terminal rendering. No-data policy results are rendered explicitly, and near-threshold comparisons retain enough precision to remain truthful.

Platform behavior

• A single TestCoverageResult correlates data by session, producer, scope, metric, and custom metric name; duplicate full keys use last-write-wins while preserving stable order.
• Coverage is registered for both in-process framework and out-of-process controller pipelines.
• Threshold failures override only an otherwise-successful run with CoverageThresholdFailed (14), while honoring --ignore-exit-code 14.
• Hot-reload sessions reset accumulated coverage state between runs.

Tests

• Unit coverage for message/read-model validation, correlation, deduplication, ordering, reset, no-data policy, precision, localization-safe labels, and terminal control-character normalization.
• Restored all unrelated terminal reporter regression coverage from main.
• Acceptance coverage for in-process and controller exit-code, terminal rendering, and ignore-exit-code behavior.

Coverage capabilities

• Extensions can resolve ITestCoverageCapabilities and inspect EnabledProducerUids from a session/data-consumer callback. This lets a report generator detect whether another enabled extension declares coverage messages and emit a warning when none is available.
• JSON-RPC initialize advertises testing.supportsTestCoverageMessages.
• Remote capabilities advertise protocol support only; enabled producer UIDs remain application-local because framework and per-session extensions may not exist yet during initialize.

[Copilot]

Pull request overview

Adds first-class coverage messages, terminal summaries, threshold tracking, and threshold-specific exit handling to Microsoft.Testing.Platform.

Changes:

• Introduces public coverage and threshold message contracts.
• Registers consumers and renders coverage results in terminal summaries.
• Returns exit code 14 when coverage thresholds fail.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/.../TerminalTestReporterTests.cs	Tests coverage-summary rendering.
.../Services/TestCoverageResult.cs	Tracks coverage and failed thresholds.
.../Services/ITestCoverageResult.cs	Defines the internal result contract.
.../PublicAPI/PublicAPI.Unshipped.txt	Declares new public APIs.
.../TerminalOutputDevice.SessionLifecycle.cs	Renders coverage after sessions.
.../TerminalOutputDevice.DataConsumption.cs	Consumes coverage messages.
.../TerminalOutputDevice.cs	Registers message types and buffers.
.../TerminalTestReporter.Summary.cs	Formats coverage summaries.
.../Messages/TestCoverageMessages.cs	Adds coverage message contracts.
.../InternalAPI/InternalAPI.Unshipped.txt	Records new internal APIs.
.../Hosts/TestHostControllersTestHost.cs	Handles controller threshold exits.
.../Hosts/TestHostBuilder.Framework.cs	Registers the framework consumer.
.../Hosts/TestHostBuilder.CommonServices.cs	Registers the coverage service.
.../Hosts/ConsoleTestHost.cs	Handles in-process threshold exits.
.../Helpers/ExitCodes.cs	Adds exit code 14.
...Extensions.TrxReport/...Unshipped.txt	Updates internal API baseline.
...Extensions.Retry/...Unshipped.txt	Updates internal API baseline.
...Extensions.MSBuild/InternalAPI.Unshipped.txt	Updates internal API baseline.
...Extensions.JUnitReport/...Unshipped.txt	Updates internal API baseline.
...Extensions.HtmlReport/...Unshipped.txt	Updates internal API baseline.
...Extensions.HangDump/...Unshipped.txt	Updates internal API baseline.
...Extensions.GitHubActionsReport/...Unshipped.txt	Updates internal API baseline.
...Extensions.CtrfReport/...Unshipped.txt	Updates internal API baseline.
...Extensions.AzureDevOpsReport/...Unshipped.txt	Updates internal API baseline.

[Evangelink]

Hi @james-newell-forge — thanks for the contribution! I did a security pass and a fairly deep code review. Sharing findings below.

Prompt-injection / security check

No prompt injection found. The only AI-related text is the benign "this PR was assisted with Claude Opus 4.8" note in the description. Nothing in the diff attempts to manipulate reviewers or automation. 👍

Before going further: goal + tracking issue

A couple of process points before we dig into the code:

1. What's the concrete goal here? Could you describe the end-to-end scenario you're trying to enable? Right now the PR adds message contracts, a consumer, terminal rendering, and a new exit code — but nothing in the codebase actually produces TestCoverageMessage / TestCoverageThresholdMessage. Your own note ("Changes would need to be made to mimic the coverage output in the SDK via IPC") confirms the producer side is missing, so this is effectively a public API + dead consumption path with no way to exercise it today. Knowing the real target scenario will help us judge the shape of the API.

2. Please get an approved tracking issue first. This introduces a brand-new public API surface to Microsoft.Testing.Platform (two message types, three enums, a new exit code, a PublicAPI.Unshipped.txt contract). Per our contributing guidelines we want new features — especially new public API — to have an approved design/issue before implementation, since public API is essentially permanent. Could you open a Feature issue (use the native GitHub Feature issue type, not a type/feature label) describing the design so the team can sign off on the surface? That avoids you investing more effort in an API shape we might want to change.

I'll hold off requesting detailed changes until we align on the above, but here's the technical review so it's all in one place.

Design-level concerns

• No producer / can't be tested end-to-end. Because nothing emits these messages, there are no acceptance/integration tests exercising the real path (terminal output + exit code 14). The added unit tests only call AppendCoverageSummary directly. This makes it hard to validate the feature actually works as a whole.
• Duplicated state. Coverage entries are buffered twice: once in TerminalOutputDevice (_coverageEntries / _coverageThresholdEntries) and once in TestCoverageResult. Two independent consumers accumulating the same data is easy to get out of sync. Consider having the terminal device read from the single ITestCoverageResult service rather than maintaining its own lists.
• Public API naming. TestCoverageThresholdMessage.Stat (of type CoverageThresholdStat) is an unclear abbreviation for a permanent public property. Something like Statistic / AggregationKind reads better. Same for the enum name.
• PropertyBagData with an empty bag. Both messages derive from PropertyBagData but never populate the property bag — consumers that inspect the bag generically will see nothing. Is PropertyBagData the right base, or should these carry their data as bag properties?

Correctness / consistency issues

• Missing localization (guideline violation). The user-facing terminal strings — "Code Coverage Summary:", "Coverage Threshold Results:", and the "Test coverage" / "Reports code coverage..." display names — are hardcoded English. Every other terminal summary string in TerminalTestReporter goes through TerminalResources.resx (e.g. TerminalResources.OutOfProcessArtifactsProduced). New strings must be added to the .resx and the .xlf files regenerated via /t:UpdateXlf.
• Culture-sensitive number formatting. {entry.Value:F1} uses the current culture, so on some locales you'll get 96,6%. Terminal diagnostics elsewhere generally format with CultureInfo.InvariantCulture.
• Thread safety. ConsumeAsync does an unguarded List.Add. Please confirm the data-consumer pump guarantees serialized calls per consumer; if not, these lists need synchronization.

Minor

• The exit-code check is duplicated verbatim in ConsoleTestHost and TestHostControllersTestHost; a shared helper would avoid drift.
• The 9 InternalAPI.Unshipped.txt edits across extension projects are just re-baselining the new ExitCode value — expected, but worth a sanity check that they were regenerated rather than hand-edited.

Happy to help once there's an approved issue and we know the target scenario. Thanks again!

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

[Evangelink]

Blocking PR until my previous points are addressed

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 4 comments.

[james-newell-forge]

The goal on this change is to allow coverage extensions like Microsoft's coverage or coverlet to add threshold coverage checks (which would address microsoft/codecoverage#160). In addition, coverage summaries can be displayed as a first-class citizen. Will add a tracking issue.

[james-newell-forge]

Thread safety. ConsumeAsync does an unguarded List.Add. Please confirm the data-consumer pump guarantees serialized calls per consumer; if not, these lists need synchronization.

TestCoverageResult.ConsumeAsync thread-safety

Verdict: List<T>.Add isn't thread-safe, but the usage here is — the message bus serializes ConsumeAsync per consumer.

Why it's safe

• No concurrent Add. Each IDataConsumer gets one AsyncConsumerDataProcessor with a single-reader channel (SingleReader = true) and one background loop that awaits each ConsumeAsync before reading the next. Many threads may PublishAsync, but dispatch is serialized → two .Adds never overlap.
• Safe reads. All readers run only after DrainDataAsync() (+ DisableAsync()):
  • the hosts read HasCoverageThresholdFailure for the exit code, and
  • TerminalOutputDevice.DisplayAfterSessionEndRunAsync reads CoverageEntries / ThresholdEntries to render the summary.
    The drain marker (awaited TaskCompletionSource) is a happens-before barrier, so all writes are visible and no read races an in-flight Add.

Caveats

• Relies on the contract that only the bus calls ConsumeAsync (true here).
• Would race if any code read the collections before the drain (none does today — both the host exit-code check and the terminal summary run post-drain).

Recommendation

No change needed — consistent with the platform's other built-in consumers.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 45 changed files in this pull request and generated 4 comments.

[james-newell-forge]

Please get an approved tracking issue first. This introduces a brand-new public API surface to Microsoft.Testing.Platform (two message types, three enums, a new exit code, a PublicAPI.Unshipped.txt contract). Per our contributing guidelines we want new features — especially new public API — to have an approved design/issue before implementation, since public API is essentially permanent. Could you open a Feature issue (use the native GitHub Feature issue type, not a type/feature label) describing the design so the team can sign off on the surface? That avoids you investing more effort in an API shape we might want to change.

Tracking Issue created: #9943

[Copilot]

Review details

• Files reviewed: 92/92 changed files
• Comments generated: 1
• Review effort level: Medium

[Copilot]

Review details

• Files reviewed: 93/93 changed files
• Comments generated: 1
• Review effort level: Medium

[Copilot]

Review details

• Files reviewed: 95/95 changed files
• Comments generated: 0 new
• Review effort level: Medium

[Copilot]

Review details

• Files reviewed: 95/95 changed files
• Comments generated: 1
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (1)

src/Platform/Microsoft.Testing.Platform/IPC/Serializers/TestHostCompletedRequestSerializer.cs:21

• Catching EndOfStreamException also treats a payload containing 1–3 bytes of a truncated second field as a valid legacy payload, silently accepting a corrupt frame. Legacy payloads have exactly zero bytes after ExitCode; check for that case explicitly and let ReadInt reject partial fields.

• Files reviewed: 95/95 changed files
• Comments generated: 11
• Review effort level: Medium

[Copilot]

Review details

• Files reviewed: 95/95 changed files
• Comments generated: 2
• Review effort level: Medium

[Copilot]

Pull request overview

Copilot reviewed 95 out of 95 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 90 out of 90 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 90 out of 90 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 90 out of 90 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 90 out of 90 changed files in this pull request and generated no new comments.

[Copilot]

Review details

• Files reviewed: 90/90 changed files
• Comments generated: 0 new
• Review effort level: Medium