Skip to content

OCPBUGS-62626: Auto fix Skip machine scaling test on two-node clusters with 0 worker replicas#31002

Open
ropatil010 wants to merge 1 commit intoopenshift:mainfrom
ropatil010:OCPBUGS-62626-autofix
Open

OCPBUGS-62626: Auto fix Skip machine scaling test on two-node clusters with 0 worker replicas#31002
ropatil010 wants to merge 1 commit intoopenshift:mainfrom
ropatil010:OCPBUGS-62626-autofix

Conversation

@ropatil010
Copy link
Copy Markdown

Hi Team,

This fixes a test failure on two-node/compact clusters where worker machineSets exist but have 0 replicas.

Problem:
The test "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously" was failing on two-node baremetal clusters with:
Error: not all machines have a node reference: map[]

Failure logs:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-serial-techpreview-3of3/2043511398402625536/artifacts/e2e-metal-ovn-two-node-fencing-serial-techpreview/baremetalds-e2e-test/artifacts/e2e.log

Root Cause:
On two-node clusters, worker machineSets may have spec.replicas=0 because workers are handled differently (masters also serve as workers). When the test attempted to scale from 0→1:

  1. A new machine was created
  2. The machine never got a nodeRef due to limited hardware resources
  3. getNodesFromMachineSet() failed waiting for the nodeRef

Solution:
Add a pre-flight check to skip the test if any worker machineSet has 0 replicas. This check is performed BEFORE any scaling operations, preventing the failure.

The fix also moves the Machine API and machineSet checks to the beginning of the test for better early validation.

Tested on:

  • Two-node cluster with 0 worker machineSets: SKIP (expected)
  • Two-node cluster with worker machineSet having 0 replicas: SKIP (expected)

This fixes a test failure on two-node/compact clusters where worker
machineSets exist but have 0 replicas.

Problem:
The test "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed
cluster should grow and decrease when scaling different machineSets
simultaneously" was failing on two-node baremetal clusters with:
  Error: not all machines have a node reference: map[]

Root Cause:
On two-node clusters, worker machineSets may have spec.replicas=0
because workers are handled differently (masters also serve as workers).
When the test attempted to scale from 0→1:
1. A new machine was created
2. The machine never got a nodeRef due to limited hardware resources
3. getNodesFromMachineSet() failed waiting for the nodeRef

Solution:
Add a pre-flight check to skip the test if any worker machineSet has
0 replicas. This check is performed BEFORE any scaling operations,
preventing the failure.

The fix also moves the Machine API and machineSet checks to the
beginning of the test for better early validation.

Tested on:
- Two-node cluster with 0 worker machineSets: SKIP (expected)
- Two-node cluster with worker machineSet having 0 replicas: SKIP (expected)

Related: OCPBUGS-62626
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Apr 13, 2026
@openshift-ci-robot
Copy link
Copy Markdown

@ropatil010: This pull request references Jira Issue OCPBUGS-62626, which is invalid:

  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Hi Team,

This fixes a test failure on two-node/compact clusters where worker machineSets exist but have 0 replicas.

Problem:
The test "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously" was failing on two-node baremetal clusters with:
Error: not all machines have a node reference: map[]

Failure logs:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-serial-techpreview-3of3/2043511398402625536/artifacts/e2e-metal-ovn-two-node-fencing-serial-techpreview/baremetalds-e2e-test/artifacts/e2e.log

Root Cause:
On two-node clusters, worker machineSets may have spec.replicas=0 because workers are handled differently (masters also serve as workers). When the test attempted to scale from 0→1:

  1. A new machine was created
  2. The machine never got a nodeRef due to limited hardware resources
  3. getNodesFromMachineSet() failed waiting for the nodeRef

Solution:
Add a pre-flight check to skip the test if any worker machineSet has 0 replicas. This check is performed BEFORE any scaling operations, preventing the failure.

The fix also moves the Machine API and machineSet checks to the beginning of the test for better early validation.

Tested on:

  • Two-node cluster with 0 worker machineSets: SKIP (expected)
  • Two-node cluster with worker machineSet having 0 replicas: SKIP (expected)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 13, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 715f282f-da65-4f32-a344-f9cb233cc1e4

📥 Commits

Reviewing files that changed from the base of the PR and between 1a93dad and a7aedfd.

📒 Files selected for processing (1)
  • test/extended/machines/scale.go

Walkthrough

Reorders and extends preconditions in a machine scaling test by moving operator verification and machineSet fetching earlier in the test body. Adds a conditional skip when any worker MachineSet reports zero replicas, indicating a two-node cluster scenario.

Changes

Cohort / File(s) Summary
Test Precondition Refactoring
test/extended/machines/scale.go
Reorders initial test preconditions by moving operator check and machineSet discovery earlier. Adds conditional skip for zero-replica scenarios (two-node clusters). Maintains existing scaling, node verification, and health check logic unchanged.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci bot requested review from deads2k and p0lyn0mial April 13, 2026 10:37
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 13, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ropatil010
Once this PR has been reviewed and has the lgtm label, please assign sosiouxme for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ropatil010
Copy link
Copy Markdown
Author

/assign @atiratree
cc: @mrda

@ropatil010
Copy link
Copy Markdown
Author

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-serial-techpreview-3of3

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 13, 2026

@ropatil010: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-serial-techpreview-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/e8983490-3726-11f1-848f-a4914750a001-0

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@ropatil010
Copy link
Copy Markdown
Author

/cc @mrda

@openshift-ci openshift-ci bot requested a review from mrda April 13, 2026 11:51
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 13, 2026

@ropatil010: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-ovn a7aedfd link true /test e2e-gcp-ovn
ci/prow/e2e-vsphere-ovn a7aedfd link true /test e2e-vsphere-ovn
ci/prow/e2e-aws-ovn-fips a7aedfd link true /test e2e-aws-ovn-fips
ci/prow/e2e-vsphere-ovn-upi a7aedfd link true /test e2e-vsphere-ovn-upi
ci/prow/e2e-metal-ipi-ovn-ipv6 a7aedfd link true /test e2e-metal-ipi-ovn-ipv6

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-trt
Copy link
Copy Markdown

openshift-trt bot commented Apr 13, 2026

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: a7aedfd

  • "Find all of the input images from ocp/5.0 and tag them into the stable-initial stream" [Total: 11, Pass: 11, Fail: 0, Flake: 0]

@ropatil010
Copy link
Copy Markdown
Author

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-serial-techpreview-3of3

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 13, 2026

@ropatil010: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-serial-techpreview-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/2f20f770-375a-11f1-861d-2e5141bd6f7b-0

@ropatil010
Copy link
Copy Markdown
Author

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants