Add a metric and an alert for conditional risk condtions by hongkailiu · Pull Request #1318 · openshift/cluster-version-operator

hongkailiu · 2026-02-15T02:14:45Z

The new metic is cluster_version_risk_conditions{name, condition, risk} and the alert is firing if some risk applies to the cluster.

coderabbitai · 2026-02-15T02:14:59Z

Walkthrough

Adds a new PrometheusRule alert (RiskApplies) and a new Prometheus GaugeVec metric cluster_version_risk_conditions with collection logic and tests to expose cluster version conditional update risk conditions.

Changes

Cohort / File(s)	Summary
Service Monitor Configuration `install/0000_90_cluster-version-operator_02_servicemonitor.yaml`	Inserted a new PrometheusRule alert `RiskApplies` in the `cluster-version` group that fires when `cluster_version_risk_conditions{condition="Applies"}` is true for 10m; includes summary, description, severity, and runbook_url.
Metric Implementation `pkg/cvo/metrics.go`	Added `clusterVersionRiskConditions *prometheus.GaugeVec` to `operatorMetrics`, initialized it, exposed its descriptor in `Describe`, added `collectConditionalUpdateRisks` to convert `ConditionalUpdateRisk` entries into labeled 0/1 metrics, and wired the collector into the per-ClusterVersion and main `Collect` flows.
Metric Tests `pkg/cvo/metrics_test.go`	Added `Test_collectConditionalUpdateRisks` to validate metric emission for multiple `ConditionalUpdateRisk` scenarios (no risks, unknown, Applies true/false/unknown). Note: the test was added twice (duplicate definitions present).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

No actionable comments were generated in the recent review. 🎉

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci · 2026-02-15T02:15:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongkailiu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [hongkailiu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@install/0000_90_cluster-version-operator_02_servicemonitor.yaml`:
- Around line 86-95: The runbook_url for the alert RiskApplies points to a
non-existent path; update the runbook_url field in the RiskApplies alert stanza
to use the cluster-version-operator runbook path used by similar alerts (e.g.,
match the URL used by ClusterVersionOperatorDown) — i.e., replace
https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/RiskApplies.md
with
https://github.com/openshift/runbooks/blob/master/alerts/cluster-version-operator/RiskApplies.md
or alternatively create the missing runbook at the original path so the current
URL remains valid.

In `@pkg/cvo/metrics_test.go`:
- Line 976: The test function named Test_collectConditionalUpdates is testing
collectConditionalUpdateRisks (not collectConditionalUpdates), which is
misleading and conflicts with TestCollectUnknownConditionalUpdates; rename the
test to something like Test_collectConditionalUpdateRisks (or
TestCollectConditionalUpdateRisks) and update any references so the test name
clearly matches the function under test (collectConditionalUpdateRisks) and does
not collide with the existing TestCollectUnknownConditionalUpdates that targets
collectConditionalUpdates.

coderabbitai · 2026-02-15T02:18:19Z

install/0000_90_cluster-version-operator_02_servicemonitor.yaml

+    - alert: RiskApplies
+      annotations:
+        summary: The cluster has been exposed to the conditional update risk for 10 minutes.
+        description: The conditional update risk {{ "{{ $labels.risk }}" }} applies to the cluster, and the cluster update to a version exposed to the risk is not recommended. For more information refer to 'oc adm upgrade'.
+        runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/RiskApplies.md
+      expr: |
+        max by (namespace, name, risk) (cluster_version_risk_conditions{job="cluster-version-operator", name="version", condition="Applies"} == 1)
+      for: 10m
+      labels:
+        severity: warning


⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# Check the actual file content around lines 86-95 cat -n install/0000_90_cluster-version-operator_02_servicemonitor.yaml | sed -n '80,100p'

Repository: openshift/cluster-version-operator

Length of output: 1817

🏁 Script executed:

# Look at all runbook_url entries in the file to understand the pattern rg "runbook_url:" install/0000_90_cluster-version-operator_02_servicemonitor.yaml -A 1 -B 1

Repository: openshift/cluster-version-operator

Length of output: 2565

🏁 Script executed:

# Check the structure around line 47 to compare ClusterVersionOperatorDown cat -n install/0000_90_cluster-version-operator_02_servicemonitor.yaml | sed -n '40,55p'

Repository: openshift/cluster-version-operator

Length of output: 1279

🏁 Script executed:

# Try to access the runbooks repository to check file existence # Since we can't clone, we'll use a different approach to check GitHub # Check if the cluster-monitoring-operator path exists curl -s -I "https://raw.githubusercontent.com/openshift/runbooks/master/alerts/cluster-monitoring-operator/RiskApplies.md" | head -1 # Check if the cluster-version-operator path exists curl -s -I "https://raw.githubusercontent.com/openshift/runbooks/master/alerts/cluster-version-operator/RiskApplies.md" | head -1

Repository: openshift/cluster-version-operator

Length of output: 102

🏁 Script executed:

# Verify if the ClusterVersionOperatorDown runbook exists (which is referenced at line 47) curl -s -I "https://raw.githubusercontent.com/openshift/runbooks/master/alerts/cluster-version-operator/ClusterVersionOperatorDown.md" | head -1 # Also check the cluster-monitoring-operator directory structure curl -s -I "https://raw.githubusercontent.com/openshift/runbooks/master/alerts/cluster-monitoring-operator/ClusterOperatorDown.md" | head -1

Repository: openshift/cluster-version-operator

Length of output: 102

The runbook URL references a non-existent file.

The runbook_url at line 90 points to a non-existent path. Neither cluster-monitoring-operator/RiskApplies.md nor cluster-version-operator/RiskApplies.md exist in the runbooks repository.

Since RiskApplies is a cluster-version-operator-specific alert (it monitors cluster_version_risk_conditions), it should follow the pattern of other CVO alerts like ClusterVersionOperatorDown at line 47, which uses:
https://github.com/openshift/runbooks/blob/master/alerts/cluster-version-operator/RiskApplies.md

Either create the runbook at that location or update the URL to reference an existing runbook.

🤖 Prompt for AI Agents

In `@install/0000_90_cluster-version-operator_02_servicemonitor.yaml` around lines 86 - 95, The runbook_url for the alert RiskApplies points to a non-existent path; update the runbook_url field in the RiskApplies alert stanza to use the cluster-version-operator runbook path used by similar alerts (e.g., match the URL used by ClusterVersionOperatorDown) — i.e., replace https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/RiskApplies.md with https://github.com/openshift/runbooks/blob/master/alerts/cluster-version-operator/RiskApplies.md or alternatively create the missing runbook at the original path so the current URL remains valid.

I am working on it.

@hongkailiu, thank you for confirming! I'll wait for the update to the runbook URL.

pkg/cvo/metrics_test.go

Follow up [1]. [1]. openshift#1284 (comment)

hongkailiu · 2026-02-15T15:49:30Z

Testing with 129bc80:

launch 4.22,openshift/cluster-version-operator#1318 aws,techpreview

Then

$ oc patch clusterversion/version --patch '{"spec":{"upstream":"https://fauxinnati-fauxinnati.apps.ota-stage.q2z4.p1.openshiftapps.com/api/upgrades_info/graph"}}' --type=merge

$ oc adm upgrade channel risks-always

The alert is Pending.

and

openshift-ci · 2026-02-15T19:16:56Z

@hongkailiu: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

hongkailiu · 2026-02-16T13:58:08Z

/hold

This feature has to be gated by TP.

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 15, 2026

coderabbitai bot reviewed Feb 15, 2026

View reviewed changes

hongkailiu force-pushed the cluster_version_risk_condition branch from 05082f4 to 3b6c582 Compare February 15, 2026 13:51

hongkailiu added 2 commits February 15, 2026 08:52

Introduce a new metric cluster_version_risk_conditions

636802e

Follow up [1]. [1]. openshift#1284 (comment)

Add alert/RiskApplies

129bc80

hongkailiu force-pushed the cluster_version_risk_condition branch from 3b6c582 to 129bc80 Compare February 15, 2026 13:52

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a metric and an alert for conditional risk condtions#1318

Add a metric and an alert for conditional risk condtions#1318
hongkailiu wants to merge 2 commits intoopenshift:mainfrom
hongkailiu:cluster_version_risk_condition

hongkailiu commented Feb 15, 2026

Uh oh!

coderabbitai bot commented Feb 15, 2026 •

edited

Loading

Uh oh!

openshift-ci bot commented Feb 15, 2026

Uh oh!

coderabbitai bot left a comment

Uh oh!

coderabbitai bot Feb 15, 2026 •

edited

Loading

Uh oh!

hongkailiu Feb 15, 2026

Uh oh!

coderabbitai bot Feb 15, 2026

Uh oh!

Uh oh!

hongkailiu commented Feb 15, 2026

Uh oh!

openshift-ci bot commented Feb 15, 2026

Uh oh!

hongkailiu commented Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

hongkailiu commented Feb 15, 2026

Uh oh!

coderabbitai bot commented Feb 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

openshift-ci bot commented Feb 15, 2026

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot Feb 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hongkailiu Feb 15, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot Feb 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hongkailiu commented Feb 15, 2026

Uh oh!

openshift-ci bot commented Feb 15, 2026

Uh oh!

hongkailiu commented Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai bot commented Feb 15, 2026 •

edited

Loading

coderabbitai bot Feb 15, 2026 •

edited

Loading