Skip to content

Conversation

@arjan-bal
Copy link
Contributor

@arjan-bal arjan-bal commented Feb 2, 2026

Fixes: #8046

Problem

The test is flaky due to a race condition in the driver script. Currently, the script waits for the client to exit and immediately validates the server output:

if ! timeout 20 go run ${example}/*client/*.go $client_args &> $CLIENT_LOG; then
fail "client failed to communicate with server
got server log:
$(cat $SERVER_LOG)
got client log:
$(cat $CLIENT_LOG)
"
else
pass "client successfully communicated with server"
fi
# Check server log for expected output if expecting an
# output
if [ -n "${EXPECTED_SERVER_OUTPUT[$example]}" ]; then
if ! grep -q "${EXPECTED_SERVER_OUTPUT[$example]}" $SERVER_LOG; then
fail "server log missing output: ${EXPECTED_SERVER_OUTPUT[$example]}
got server log:
$(cat $SERVER_LOG)
got client log:
$(cat $CLIENT_LOG)
"
else
pass "server log contains expected output: ${EXPECTED_SERVER_OUTPUT[$example]}"
fi
fi

In the graceful shutdown example, the expected log is written only after the shutdown is complete:

log.Println("Server stopped gracefully.")

If the test driver validates the server output after the client stream closes but before the server logs the expected message, the test fails.

Fix

This PR updates the test driver to poll for the expected log in the server output, instead of checking it only once. Additionally, the shutdown of the test server is blocked until either the server logs a failure or a success message.

Validation

To reproduce the failure, I added a 1s sleep just before the line log.Println("Server stopped gracefully.") in the server code. I verified that the test no longer fails with this fix, even with the artificial delay.

RELEASE NOTES: N/A

@arjan-bal arjan-bal added this to the 1.80 Release milestone Feb 2, 2026
@arjan-bal arjan-bal added Type: Testing Area: Testing Includes tests and testing utilities that we have for unit and e2e tests within our repo. labels Feb 2, 2026
@codecov
Copy link

codecov bot commented Feb 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.28%. Comparing base (3e475d7) to head (6840670).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8871      +/-   ##
==========================================
+ Coverage   82.47%   83.28%   +0.80%     
==========================================
  Files         414      414              
  Lines       32751    32751              
==========================================
+ Hits        27011    27276     +265     
+ Misses       4089     4071      -18     
+ Partials     1651     1404     -247     

see 30 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@arjan-bal arjan-bal merged commit b7b1cce into grpc:master Feb 3, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: Testing Includes tests and testing utilities that we have for unit and e2e tests within our repo. Type: Testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky Test: extras/features/gracefulstop

3 participants