OCPBUGS-72387: Update the downloads manifest

[jhadvig]

Follow up of #1099

Problem

1. Cluster upgrades (4.20→4.21) fail with node drain timeouts:
2. PDB blocks eviction of unhealthy pods: Cannot evict pod as it would violate the pod's disruption budget
   Downloads pod eviction times out: global timeout reached: 1m30s

Root Causes:

1. PDB blocking: maxUnavailable: 1 prevents eviction of unhealthy/stuck pods (fixed in OCPBUGS-72387: Fix PDB blocking node drains during cluster upgrades #1099)
2. Slow termination: terminationGracePeriodSeconds: 0 causes SIGTERM/SIGKILL race, and sys.exit(0) attempts slow cleanup of 100 HTTP threads
3. Binary archives are created in series which is blocking start of the server

Downloads Deployment Fix:

• Prevents SIGTERM/SIGKILL race condition
• Forces instant termination of all 100 worker threads
• Pod exits in <1s, well within eviction timeout
• Create binary archives asynchronously

/assign @TheRealJon @cajieh

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-72387, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @yapei

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Follow up of #1099

Problem

1. Cluster upgrades (4.20→4.21) fail with node drain timeouts:
2. PDB blocks eviction of unhealthy pods: Cannot evict pod as it would violate the pod's disruption budget
   Downloads pod eviction times out: global timeout reached: 1m30s

Root Causes:

1. PDB blocking: maxUnavailable: 1 prevents eviction of unhealthy/stuck pods (fixed in OCPBUGS-72387: Fix PDB blocking node drains during cluster upgrades #1099)
2. Slow termination: terminationGracePeriodSeconds: 0 causes SIGTERM/SIGKILL race, and sys.exit(0) attempts slow cleanup of 100 HTTP threads

Downloads Deployment Fix:

• Prevents SIGTERM/SIGKILL race condition
• Forces instant termination of all 100 worker threads
• Pod exits in <1s, well within eviction timeout

/assign @TheRealJon @cajieh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[cajieh]

Looks good 👍, but I will defer to @TheRealJon for a second pair of eyes.

[jhadvig]

/retest

[openshift-ci]

@jhadvig: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[yanpzhan]

Successfully upgraded 4.21 ibmcloud cluster to image built from the pr.
/verified by yanpzhan

[openshift-ci-robot]

@yanpzhan: This PR has been marked as verified by yanpzhan.

Details

In response to this:

Successfully upgraded 4.21 ibmcloud cluster to image built from the pr.
/verified by yanpzhan

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-72387, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @yapei

Details

In response to this:

Follow up of #1099

Problem

1. Cluster upgrades (4.20→4.21) fail with node drain timeouts:
2. PDB blocks eviction of unhealthy pods: Cannot evict pod as it would violate the pod's disruption budget
   Downloads pod eviction times out: global timeout reached: 1m30s

Root Causes:

1. PDB blocking: maxUnavailable: 1 prevents eviction of unhealthy/stuck pods (fixed in OCPBUGS-72387: Fix PDB blocking node drains during cluster upgrades #1099)
2. Slow termination: terminationGracePeriodSeconds: 0 causes SIGTERM/SIGKILL race, and sys.exit(0) attempts slow cleanup of 100 HTTP threads
3. Binary archives are created in series which is blocking start of the server

Downloads Deployment Fix:

• Prevents SIGTERM/SIGKILL race condition
• Forces instant termination of all 100 worker threads
• Pod exits in <1s, well within eviction timeout
• Create binary archives asynchronously

/assign @TheRealJon @cajieh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jhadvig]

Adding the verify label back since I've need to remove unnecessary logs.
@TheRealJon PTAL

[jhadvig]

/payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-upgrade-fips
/payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-upgrade-fips-nat-instance

[openshift-ci]

@jhadvig: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-upgrade-fips

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/468d9930-fc55-11f0-9f54-8d960f04dd9d-0

[TheRealJon]

I'm no python expert, but the addition of async archive building seems like a solid idea to help make pods ready up faster!
/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jhadvig, TheRealJon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [TheRealJon,jhadvig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@jhadvig: Jira Issue OCPBUGS-72387: All pull requests linked via external trackers have merged:

• openshift/console-operator#1099
• openshift/console-operator#1101

Jira Issue OCPBUGS-72387 has been moved to the MODIFIED state.

Details

In response to this:

Follow up of #1099

Problem

1. Cluster upgrades (4.20→4.21) fail with node drain timeouts:
2. PDB blocks eviction of unhealthy pods: Cannot evict pod as it would violate the pod's disruption budget
   Downloads pod eviction times out: global timeout reached: 1m30s

Root Causes:

1. PDB blocking: maxUnavailable: 1 prevents eviction of unhealthy/stuck pods (fixed in OCPBUGS-72387: Fix PDB blocking node drains during cluster upgrades #1099)
2. Slow termination: terminationGracePeriodSeconds: 0 causes SIGTERM/SIGKILL race, and sys.exit(0) attempts slow cleanup of 100 HTTP threads
3. Binary archives are created in series which is blocking start of the server

Downloads Deployment Fix:

• Prevents SIGTERM/SIGKILL race condition
• Forces instant termination of all 100 worker threads
• Pod exits in <1s, well within eviction timeout
• Create binary archives asynchronously

/assign @TheRealJon @cajieh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.