Fix NSG Reconciler Flake #4572
Open
Fix NSG Reconciler Flake #4572
+75
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mociarain
requested review from
bennerv,
cadenmarchese,
hawkowl,
hlipsig,
jharrington22,
kimorris27,
mrWinston,
rogbas,
sankur-codes,
tiguelu,
tsatam,
wanghaoran1988 and
yjst2012
as code owners
bizz001
reviewed
Jan 29, 2026
bizz001
reviewed
Jan 29, 2026
The original code used r.instance which was fetched once at reconciliation start. If another process modified the Cluster resource during the reconciliation loop, the patch would fail with a conflict error (HTTP 409) due to stale resourceVersion. This change wraps the annotation update in retry.RetryOnConflict, which is the standard Kubernetes pattern for optimistic concurrency control. Each retry iteration fetches a fresh copy of the Cluster resource, ensuring we have the latest resourceVersion before patching. Trade-offs: - Extra API call: Each retry does a Get before Patch, adding latency but ensuring correctness - Retry limits: retry.DefaultRetry uses exponential backoff with ~5 retries; in extreme contention scenarios this could still fail - No r.instance sync: The in-memory r.instance is not updated, which is fine since it's only used for reading during reconciliation and the annotation update is terminal Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I observed several example where our E2E tests were failing due to this test. After a little pairing we came to the conclusion that a likely cause is the Update made to the cluster CR from the reconciler failing. We believe this fails because:
This PR moves to a Patch rather than an update which should avoid this problem.
On a wider note the only reason I can see that we add the annotation to the CR is so it can be picked up by the monitor and written as a metric... This seems brittle and I'd be in favour of just writing to the in-cluster prometheus from the ARO-Operator directly assuming https://github.com/openshift-online/architecture/pull/57 is accepted @alcasim in particular.