[bpf-ci-bot] sched_ext rt_stall test still flaky: measurement includes pre-sleep CPU time

[kernel-patches-review-bot]

Summary

The sched_ext rt_stall selftest remains flaky in CI despite the synchronization fix in commit 0b82cc331d2e. The test fails because it measures total CPU time since fork rather than the delta during the measurement window, causing pre-sleep CPU accumulation to skew the ratio below the 4% threshold.

Failure Details

• Test / Component: selftests/sched_ext rt_stall
• Frequency: Most sched_ext CI runs on x86_64 (observed in 3+ independent PRs over 3 days)
• Failure mode: Flaky — ratio drops to 3.0-3.9% instead of expected ~5%
• Affected architectures: x86_64
• CI runs observed:
  • https://github.com/kernel-patches/bpf/actions/runs/22834153226 (bpf-next_test)
  • https://github.com/kernel-patches/bpf/actions/runs/22803463271 (fix constant blinding bypass)
  • https://github.com/kernel-patches/bpf/actions/runs/22788551155 (perf_link: avoid failures)

Root Cause Analysis

The rt_stall test verifies that the sched_ext deadline server prevents RT tasks from starving EXT/FAIR tasks. It forks two children pinned to the same CPU — one EXT/FAIR and one SCHED_FIFO — then measures their CPU time ratio after sleep(5).

Commit 0b82cc331d2e added pipe-based synchronization so children complete their setup before the parent starts sleep(RUN_TIME). However, children start busy-looping immediately after signal_ready(), while the parent still needs to process both pipe reads before calling sleep(). During this gap, both children accumulate CPU time — with the RT child dominating.

The measurement reads total CPU time from /proc/pid/stat (utime + stime), which includes this pre-sleep time. This inflates the RT denominator:

Run	Failing iteration	EXT/FAIR	RT	Ratio	Note
bpf-next_test	i=2 (FAIR)	0.180s	4.740s	3.66%
constant blinding	i=3 (EXT)	0.180s	5.690s	3.07%	RT > RUN_TIME proves pre-sleep accumulation
perf_link	i=3 (EXT)	0.190s	4.740s	3.85%

The RT task getting 5.69s of CPU time in a 5-second sleep window is conclusive evidence: ~0.69s of RT time was accumulated before the measurement started.

The failure tends to occur in later iterations (i=2 or i=3) because the parent has more work to do between iterations (destroying/re-attaching the sched_ext link), giving children more time to accumulate pre-sleep CPU time.

Relevant code: tools/testing/selftests/sched_ext/rt_stall.c:sched_stress_test()

Proposed Fix

Take before/after snapshots of CPU time around the sleep(RUN_TIME) window and compute deltas, rather than using total CPU time since fork. This eliminates the pre-measurement bias regardless of how long the gap between signal_ready() and sleep() takes.

See attached patch: 0001-selftests-sched_ext-Fix-rt_stall-flaky-measurement-w.patch

The fix adds:

1. A pre-sleep snapshot of both children's CPU times via get_process_runtime()
2. Subtraction of the pre-sleep snapshot from the post-sleep reading
3. Error handling for the new snapshot reads

Impact

Without this fix, the sched_ext test job fails in most CI runs, blocking unrelated PRs from passing CI. The sched_ext job is not marked continue_on_error, so any sched_ext test failure fails the entire workflow.

References

• Prior fix: 0b82cc331d2e ("selftests/sched_ext: Fix rt_stall flaky failure")
• Original test: be621a76341c ("selftests/sched_ext: Add test for sched_ext dl_server")
• Related vmtest issue: [bpf-ci-bot] The sched_ext numa selftest is flaky #453 (covers numa test, not rt_stall)

[kernel-patches-review-bot]

0001-selftests-sched_ext-Fix-rt_stall-flaky-measurement-w.patch

From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: BPF CI Bot <bot+bpf-ci@kernel.org>
Date: Sun, 9 Mar 2026 00:00:00 +0000
Subject: [PATCH] selftests/sched_ext: Fix rt_stall flaky measurement window

The rt_stall test still flakes in CI despite the pipe synchronization
fix in commit 0b82cc331d2e ("selftests/sched_ext: Fix rt_stall flaky
failure"). The test reads total CPU time from /proc/pid/stat, which
includes time children accumulated between signal_ready() and the
parent's sleep(RUN_TIME) call.

In CI, the RT child can accumulate more CPU time than the sleep
duration itself (e.g., 5.69s of CPU time in a 5s sleep window),
proving that pre-measurement time inflates the denominator and pushes
the measured ratio below the 4% threshold.

Fix by taking before/after snapshots of CPU time around sleep() and
computing deltas. This isolates the measurement to exactly the
intended window, regardless of scheduling latency between pipe
synchronization and sleep().

Fixes: 0b82cc331d2e ("selftests/sched_ext: Fix rt_stall flaky failure")
Generated-by: BPF CI Bot (claude-opus-4-6-v1) <bot+bpf-ci@kernel.org>
---
 tools/testing/selftests/sched_ext/rt_stall.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/tools/testing/selftests/sched_ext/rt_stall.c b/tools/testing/selftests/sched_ext/rt_stall.c
index 81ea9b4883e5..7c98de53c2f0 100644
--- a/tools/testing/selftests/sched_ext/rt_stall.c
+++ b/tools/testing/selftests/sched_ext/rt_stall.c
@@ -197,6 +197,17 @@ static bool sched_stress_test(bool is_ext)
 	wait_ready(ext_ready[0]);
 	wait_ready(rt_ready[0]);

+	/*
+	 * Snapshot CPU times before the measurement window. Children
+	 * may have accumulated CPU time between signal_ready() and now;
+	 * using deltas ensures we only measure time during sleep().
+	 */
+	float ext_before = get_process_runtime(ext_pid);
+	float rt_before = get_process_runtime(rt_pid);
+
+	if (ext_before == -1 || rt_before == -1)
+		ksft_exit_fail_msg("Error getting pre-sleep runtime\n");
+
 	/* Let the processes run for the specified time */
 	sleep(RUN_TIME);

@@ -205,6 +216,7 @@ static bool sched_stress_test(bool is_ext)
 	if (ext_runtime == -1)
 		ksft_exit_fail_msg("Error getting runtime for %s task (PID %d)\n",
 				   class_str, ext_pid);
+	ext_runtime -= ext_before;
 	ksft_print_msg("Runtime of %s task (PID %d) is %f seconds\n",
 		       class_str, ext_pid, ext_runtime);

@@ -212,6 +224,7 @@ static bool sched_stress_test(bool is_ext)
 	rt_runtime = get_process_runtime(rt_pid);
 	if (rt_runtime == -1)
 		ksft_exit_fail_msg("Error getting runtime for RT task (PID %d)\n", rt_pid);
+	rt_runtime -= rt_before;
 	ksft_print_msg("Runtime of RT task (PID %d) is %f seconds\n", rt_pid, rt_runtime);

 	/* Kill the processes */
--
2.43.0