Add webhook-injector propagation alerts#26
Merged
Conversation
Adds five new alerts covering the propagation-path metrics that
webhook-injector exposes. The two existing alerts (WebhookCertificate
NearExpiry, WebhookCertificateAboutToExpire) cover the cert expiry
side; these five cover the apply / stamp / GRM-bridge side that is
just as important for caBundle freshness reaching the shoot.
Alerts:
- WebhookTargetStampErrors (critical, 10m for): rate of target_stamps
_total{result="error"} > 0. MWC/VWC apply on the target cluster
is failing — the freshly rotated caBundle is NOT reaching the
admission webhooks and subsequent CR writes may be rejected with
TLS errors.
- WebhookManagedResourceStampErrors (critical, 10m for): rate of
managed_resource_stamps_total{result="error"} > 0. MR Secret
stamping on the hosting cluster is failing — embedded CRDs will not
refresh in the shoot via GRM, and conversion webhooks for those
CRDs will fail TLS once the old cert is pruned.
- WebhookManagedResourceUnhealthy (warning, 15m for): rate of
managed_resource_unhealthy_total > 0. A labeled MR has
ResourcesApplied or ResourcesHealthy != True. GRM is unable to
apply or keep resources healthy in the shoot — caBundle propagation
is degraded.
- WebhookManagedResourceKeepObjectsViolation (warning, 15m for): rate
of keep_objects_violations_total > 0. A labeled MR ships CRDs
without spec.keepObjects=true — destructive shape: deleting that
MR (intentionally or accidentally) would cascade-delete shoot-side
CRDs and every custom resource of those kinds.
- WebhookManagedResourceForceReconcilePatchErrors (warning, 15m for):
rate of force_reconcile_total{result="error"} > 0. Patching
gardener.cloud/operation=reconcile on MRs is failing. GRM will
still pick up the change on its own watch/sync cadence, but with
extra latency.
Style follows the existing webhook cert alerts and the rest of this
file: Helm dig templating for for/severity/service/support_group,
disabled toggle, additionalRuleLabels include, playbook link. Each
alert can be individually disabled via .Values.prometheusRules.disabled.
Verified:
- helm lint charts/controlplane-operations: clean
- promtool check rules: SUCCESS, 11 rules found (6 existing + 5 new)
There was a problem hiding this comment.
Pull request overview
Adds propagation-path Prometheus alerts for webhook-injector metrics in the controlplane-remote PrometheusRule, complementing the existing certificate-expiry alerts to detect failures in caBundle propagation to target clusters / ManagedResources.
Changes:
- Add 5 new Prometheus alert rules covering target stamping errors, ManagedResource stamping errors, ManagedResource unhealthy state, keepObjects violations, and force-reconcile patch errors.
- Introduce per-alert disable toggles and standard labels (severity/service/support_group/additionalRuleLabels) consistent with existing rules in this file.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Patch bump for the additive webhook-injector propagation alerts. Follows the convention from #23 (1.1.9 → 1.1.10 for the previous additive alerts PR).
Addresses Copilot review feedback on #26: Using rate(metric[N]) > 0 together with for: N is a known Prometheus gotcha — a single error increment can fall out of the trailing N-minute rate window right as the for clause would otherwise satisfy, so a sustained but low-frequency error stream fails to fire the alert. Fix: keep the for: durations (which encode the persistence threshold we actually want — 10m for critical apply failures, 15m for warning-severity MR conditions) but shrink the rate window to a fixed [5m]. That ensures the rate observation continues to evaluate > 0 across the full for period for any error stream that's persistent at all. Same change applied to all five new alerts: - WebhookTargetStampErrors - WebhookManagedResourceStampErrors - WebhookManagedResourceUnhealthy - WebhookManagedResourceKeepObjectsViolation - WebhookManagedResourceForceReconcilePatchErrors Patch-bump chart + plugin to 1.1.12. Verified: - helm lint charts/controlplane-operations: clean - promtool check rules: SUCCESS, 11 rules found.
defo89
reviewed
Jun 26, 2026
Annotations on the five propagation alerts now identify the affected
shoot and the per-metric labels in summary/description, following the
pattern from controlplane-gardener.yaml ({{` $labels.<name> `}}).
Switch sum() → sum by (...) so the labels survive aggregation:
- WebhookTargetStampErrors: sum by (namespace, kind)
→ summary names {{ $labels.kind }} (Mutating/Validating) and
{{ $labels.namespace }} (the shoot's control-plane namespace).
- WebhookManagedResourceStampErrors: sum by (namespace)
→ identifies the shoot whose MR Secret stamp failed.
- WebhookManagedResourceUnhealthy: sum by (namespace, condition, status)
→ summary spells out which condition (ResourcesApplied/Healthy) is
at which status (False/Progressing/Unknown).
- WebhookManagedResourceKeepObjectsViolation: sum by (namespace)
→ identifies the shoot whose MR ships CRDs without keepObjects=true.
- WebhookManagedResourceForceReconcilePatchErrors: sum by (namespace)
→ identifies the shoot.
All descriptions now suggest checking webhook-injector logs in the
affected namespace, since the exact MR name is not a metric label
(by design — cardinality-bounded) but is in the log line.
Cardinality bound is small: target_stamps has 2×3 series max,
managed_resource_stamps and force_reconcile have 3, unhealthy has
2×3, keep_objects_violations has 1 — all per-Pod. {result="error"}
narrows further.
Patch-bump chart + plugin to 1.1.13.
Verified:
- helm lint: clean
- helm template + promtool check rules: SUCCESS, 11 rules found
- rendered output contains literal {{ $labels.namespace }} etc. as
Prometheus expects, not Helm template artifacts.
Co-authored-by: Dmitri Fedotov <13087245+defo89@users.noreply.github.com> Signed-off-by: Vladimir Videlov <vladimir.videlov@sap.com>
defo89
approved these changes
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds five new alerts to
controlplane-remote.yamlcovering the propagation-path metrics that webhook-injector exposes.The two existing alerts added in #23 (
WebhookCertificateNearExpiry,WebhookCertificateAboutToExpire) cover the cert expiry side. These five cover the apply / stamp / GRM-bridge side that is just as important for caBundle freshness reaching the shoot — if the cert rotates but the caBundle doesn't propagate, every TLS call from the shoot apiserver to the webhook server still breaks.What's added
WebhookTargetStampErrorstarget_stamps_total{result="error"}rate > 0 — MWC/VWC apply on the target cluster is failing. The freshly rotated caBundle is not reaching the admission webhooks; CR writes may be rejected with TLS errors.WebhookManagedResourceStampErrorsmanaged_resource_stamps_total{result="error"}rate > 0 — MR Secret stamping on the hosting cluster is failing. Embedded CRDs will not refresh in the shoot via GRM. Conversion webhooks for those CRDs will fail TLS once the old cert is pruned.WebhookManagedResourceUnhealthymanaged_resource_unhealthy_totalrate > 0 — a labeled MR hasResourcesAppliedorResourcesHealthycondition withStatus != True. GRM can't apply or keep the resources healthy in the shoot.WebhookManagedResourceKeepObjectsViolationkeep_objects_violations_totalrate > 0 — a labeled MR ships CRDs withoutspec.keepObjects=true. Destructive shape: deleting that MR would cascade-delete every shoot CRD it carries and every CR of those kinds.WebhookManagedResourceForceReconcilePatchErrorsforce_reconcile_total{result="error"}rate > 0 — patchinggardener.cloud/operation=reconcileon MRs is failing. GRM still picks up the change on its own watch/sync cadence, but with extra latency.The metric names and the "this counter is the alertable signal" framing come directly from the webhook-injector docs — each metric here is one the docs explicitly flag as alertable.
Style
Follows the exact convention of the existing alerts in this file (Argora + webhook cert):
{{ if not .Values.prometheusRules.disabled.<AlertName> }}toggle per alert{{ dig "<AlertName>" "for"|"severity"|"service"|"support_group" "<default>" .Values.prometheusRules }}for all four labels{{ include "controlplane-operations.additionalRuleLabels" . }}for global labels#TODO: add playbookcommentexpr: |matching the existing webhook-cert alertsVerification
All 11 alerts (6 existing + 5 new) parse correctly. PromQL syntax + label keys + rule structure validated.
Disabling
Each alert can be individually disabled via the existing pattern: