Skip to content

MON-4619: Declare KubeStateMetricsTimezonePanic risk for 4.20.z - #10074

Merged
openshift-merge-bot[bot] merged 2 commits into
openshift:masterfrom
shahsahil264:OCPBUGS-87957-kube-state-metrics-timezone-panic
Jul 31, 2026
Merged

MON-4619: Declare KubeStateMetricsTimezonePanic risk for 4.20.z#10074
openshift-merge-bot[bot] merged 2 commits into
openshift:masterfrom
shahsahil264:OCPBUGS-87957-kube-state-metrics-timezone-panic

Conversation

@shahsahil264

Copy link
Copy Markdown
Contributor

Summary

Adds conditional update risk for all 4.20.z GA releases (4.20.0 through 4.20.30) to warn clusters upgrading from 4.19.z that kube-state-metrics may crash loop if CronJobs with .spec.timeZone are present.

Root cause: /usr/share/zoneinfo/ is missing from the kube-state-metrics container image. The statically-compiled Go binary panics when time.LoadLocation() fails for timezone-aware CronJobs.

  • Affected: 4.20.z, 4.21.z, 4.22.z (confirmed by Benjamin Affolter and Simon Pasquier)
  • Not affected: 4.19.z
  • Blocked-edges for 4.20.z only: upgrading from 4.20→4.21 or 4.21→4.22 does not increase exposure (affected→affected)
  • No fixedIn: fix PRs openshift/kube-state-metrics#147 and #150 still open
  • type: Always: no PromQL metric for CronJob timezone usage

Bug

Test plan

  • ./hack/validate-blocked-edges.py passes

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 22, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@shahsahil264: This pull request references Jira Issue OCPBUGS-87957, which is invalid:

  • expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "4.22.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Adds conditional update risk for all 4.20.z GA releases (4.20.0 through 4.20.30) to warn clusters upgrading from 4.19.z that kube-state-metrics may crash loop if CronJobs with .spec.timeZone are present.

Root cause: /usr/share/zoneinfo/ is missing from the kube-state-metrics container image. The statically-compiled Go binary panics when time.LoadLocation() fails for timezone-aware CronJobs.

  • Affected: 4.20.z, 4.21.z, 4.22.z (confirmed by Benjamin Affolter and Simon Pasquier)
  • Not affected: 4.19.z
  • Blocked-edges for 4.20.z only: upgrading from 4.20→4.21 or 4.21→4.22 does not increase exposure (affected→affected)
  • No fixedIn: fix PRs openshift/kube-state-metrics#147 and #150 still open
  • type: Always: no PromQL metric for CronJob timezone usage

Bug

Test plan

  • ./hack/validate-blocked-edges.py passes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 77414213-471b-41dd-b6dc-d20add630ada

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@shahsahil264

Copy link
Copy Markdown
Contributor Author

@wking Can you PTAL!

Thanks

@openshift-ci
openshift-ci Bot requested review from DavidHurta and hongkailiu July 22, 2026 16:56
@sdodson

sdodson commented Jul 22, 2026

Copy link
Copy Markdown
Member

/hold

I don't want us to block all updates to 4.20 until such a time that we fix this. Please look closer at 4.20.28 and 4.20.29, /usr/share/zoneinfo exists in the former but not the latter. Why?

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 22, 2026
@sdodson
sdodson self-requested a review July 22, 2026 21:17

@sdodson sdodson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…d versions

Adds conditional update risk for versions where /usr/share/zoneinfo is
missing from the kube-state-metrics container image: 4.20.29-30,
4.21.24-25, and 4.22.0-6. Clusters upgrading from unaffected versions
with CronJobs using .spec.timeZone will experience kube-state-metrics
crash loops, breaking monitoring and potentially stalling upgrades.

First affected versions verified by Scott Dodson via rpm -Va tzdata:
4.20.29, 4.21.24, 4.22.0. Earlier versions (4.20.28, 4.21.23) have
zoneinfo present and are not affected.

Bug: https://redhat.atlassian.net/browse/OCPBUGS-87957
Impact: https://redhat.atlassian.net/browse/MON-4619

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Signed-off-by: Sahil Shah <sahshah@redhat.com>
@shahsahil264
shahsahil264 force-pushed the OCPBUGS-87957-kube-state-metrics-timezone-panic branch from fe6078e to 890e4f3 Compare July 23, 2026 14:23
@shahsahil264

Copy link
Copy Markdown
Contributor Author

Updated, narrowed to 4.20.29-30, 4.21.24-25, 4.22.0-6 per your findings
Thanks!

@shahsahil264
shahsahil264 requested a review from sdodson July 23, 2026 14:55
@DavidHurta

DavidHurta commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

/retitle MON-4619: Declare KubeStateMetricsTimezonePanic risk for 4.20.z

@openshift-ci openshift-ci Bot changed the title OCPBUGS-87957: Declare KubeStateMetricsTimezonePanic risk for 4.20.z MON-4619: Declare KubeStateMetricsTimezonePanic risk for 4.20.z Jul 31, 2026
@openshift-ci-robot openshift-ci-robot removed jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 31, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 31, 2026

Copy link
Copy Markdown

@shahsahil264: This pull request references MON-4619 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Adds conditional update risk for all 4.20.z GA releases (4.20.0 through 4.20.30) to warn clusters upgrading from 4.19.z that kube-state-metrics may crash loop if CronJobs with .spec.timeZone are present.

Root cause: /usr/share/zoneinfo/ is missing from the kube-state-metrics container image. The statically-compiled Go binary panics when time.LoadLocation() fails for timezone-aware CronJobs.

  • Affected: 4.20.z, 4.21.z, 4.22.z (confirmed by Benjamin Affolter and Simon Pasquier)
  • Not affected: 4.19.z
  • Blocked-edges for 4.20.z only: upgrading from 4.20→4.21 or 4.21→4.22 does not increase exposure (affected→affected)
  • No fixedIn: fix PRs openshift/kube-state-metrics#147 and #150 still open
  • type: Always: no PromQL metric for CronJob timezone usage

Bug

Test plan

  • ./hack/validate-blocked-edges.py passes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

…4.21.26, 4.22.7

Declare fixedIn 4.20.32, 4.21.27, and 4.22.8 respectively.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci

openshift-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@shahsahil264: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@wking wking left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 31, 2026
@openshift-ci

openshift-ci Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: shahsahil264, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 31, 2026
@sdodson

sdodson commented Jul 31, 2026

Copy link
Copy Markdown
Member

/hold cancel

Version ranges have been cleared up and represent actual regressed versions. Kube-state-metrics actually fixed their issue in 4.22.7 but we don't know what other coponents are broken and no one has noticed yet. As a result I am happy to wait for the more complete fix of restoring /usr/share/zoneinfo in https://issues.redhat.com/browse/OCPBUGS-99767 and cloned bugs be the one that clears this risk

@openshift-ci openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 31, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit d4c2fb6 into openshift:master Jul 31, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants