Skip to content

rhdh: add cluster health check and increase login timeout for hibernating clusters #76597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gustavolira wants to merge 2 commits into
base: main
Choose a base branch
from
+54 −9
Open

rhdh: add cluster health check and increase login timeout for hibernating clusters #76597

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
947a0fc
rhdh: increase login timeout and add cluster health check for hiberna…
gustavolira Mar 20, 2026
11b5c34
rhdh: harden login and health check in helm step scripts
gustavolira Mar 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions ...redhat-developer/rhdh/ocp/helm/nightly/redhat-developer-rhdh-ocp-helm-nightly-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,16 +64,31 @@ else
exit 1
fi

timeout --foreground 5m bash <<-"EOF"
# Disable tracing to protect credentials from leaking into CI logs
set +x 2>/dev/null
Copy link
Contributor

@zdrapela zdrapela Mar 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use set anywhere in the CI scripts, so the set +x is present by default. If you want to configure it explicitly, it makes sense to do it at the beginning of the commands.sh bash script? Because this is not the only sensitive data here.

if ! timeout --foreground 10m bash <<-"EOF"; then
while ! oc login "$OPENSHIFT_API" -u "$OPENSHIFT_USERNAME" -p "$OPENSHIFT_PASSWORD" --insecure-skip-tls-verify=true; do
sleep 20
echo "Login failed, retrying in 30s..."
sleep 30
done
EOF
if [ $? -ne 0 ]; then
echo "Timed out waiting for login"
exit 1
fi

echo "========== Cluster Health Check =========="
echo "Verifying cluster API server is fully responsive..."
Copy link
Contributor

@zdrapela zdrapela Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more of a check for nodes to be ready, so why not echo what it actually is?
Maybe also oc wait --for=condition=Ready nodes --all --timeout=XXXs may be handy here?

if ! timeout --foreground 5m bash <<-"EOF"; then
while ! oc get nodes 2>&1; do
echo "API server not ready, retrying in 15s..."
sleep 15
done
EOF
echo "Timed out waiting for cluster API server to become ready"
exit 1
fi
echo "Cluster API server is ready"

echo "========== Cluster Service Account and Token Management =========="
export K8S_CLUSTER_URL K8S_CLUSTER_TOKEN
K8S_CLUSTER_URL=$(oc whoami --show-server)
Expand Down
21 changes: 18 additions & 3 deletions ...r/step-registry/redhat-developer/rhdh/ocp/helm/redhat-developer-rhdh-ocp-helm-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,16 +85,31 @@ else
exit 1
fi

timeout --foreground 5m bash <<-"EOF"
# Disable tracing to protect credentials from leaking into CI logs
set +x 2>/dev/null
if ! timeout --foreground 10m bash <<-"EOF"; then
while ! oc login "$OPENSHIFT_API" -u "$OPENSHIFT_USERNAME" -p "$OPENSHIFT_PASSWORD" --insecure-skip-tls-verify=true; do
sleep 20
echo "Login failed, retrying in 30s..."
sleep 30
done
EOF
if [ $? -ne 0 ]; then
echo "Timed out waiting for login"
exit 1
fi

echo "========== Cluster Health Check =========="
echo "Verifying cluster API server is fully responsive..."
if ! timeout --foreground 5m bash <<-"EOF"; then
while ! oc get nodes 2>&1; do
echo "API server not ready, retrying in 15s..."
sleep 15
done
EOF
echo "Timed out waiting for cluster API server to become ready"
exit 1
fi
echo "Cluster API server is ready"

echo "========== Cluster Service Account and Token Management =========="
export K8S_CLUSTER_URL K8S_CLUSTER_TOKEN
K8S_CLUSTER_URL=$(oc whoami --show-server)
Expand Down
21 changes: 18 additions & 3 deletions .../rhdh/ocp/helm/upgrade/nightly/redhat-developer-rhdh-ocp-helm-upgrade-nightly-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,16 +64,31 @@ else
exit 1
fi

timeout --foreground 5m bash <<-"EOF"
# Disable tracing to protect credentials from leaking into CI logs
set +x 2>/dev/null
if ! timeout --foreground 10m bash <<-"EOF"; then
while ! oc login "$OPENSHIFT_API" -u "$OPENSHIFT_USERNAME" -p "$OPENSHIFT_PASSWORD" --insecure-skip-tls-verify=true; do
sleep 20
echo "Login failed, retrying in 30s..."
sleep 30
done
EOF
if [ $? -ne 0 ]; then
echo "Timed out waiting for login"
exit 1
fi

echo "========== Cluster Health Check =========="
echo "Verifying cluster API server is fully responsive..."
if ! timeout --foreground 5m bash <<-"EOF"; then
while ! oc get nodes 2>&1; do
echo "API server not ready, retrying in 15s..."
sleep 15
done
EOF
echo "Timed out waiting for cluster API server to become ready"
exit 1
fi
echo "Cluster API server is ready"

echo "========== Cluster Service Account and Token Management =========="
export K8S_CLUSTER_URL K8S_CLUSTER_TOKEN
K8S_CLUSTER_URL=$(oc whoami --show-server)
Expand Down