🏥 CI FailureIntegration Tests: multiple suite failures after "fix issues, remove workflows" commit

[github-actions]

Summary

The Integration Tests workflow failed on commit 3aa408c ("fix issues, remove workflows" by @dsyme) with 43 test failures across 10 suites in 3 parallel jobs.

---

Failing Jobs

Job	Test Suites Failed	Tests Failed
Protocol & Security Tests	3 of 5	20 of 55
Container & Ops Tests	5 of 7	16 of 66
API Proxy Tests	2 of 3	7 of 27
Domain & Network Tests	cancelled (upstream failure)	—

---

Failure Analysis

1. container-workdir.test.ts — Chroot Mode WORKDIR Mismatch

Error:

expect(result.stdout.trim()).toContain('/workspace');
                              ^
at container-workdir.test.ts:39
```

**Root Cause:** The test expects `pwd` to return `/workspace` (the Dockerfile's default `WORKDIR`), but in chroot mode the working directory is set to the host user's home (`/home/runner`). The test was written assuming non-chroot mode behavior. The entrypoint logs confirm:
```
[entrypoint] Chroot working directory: /home/runner
/home/runner   ← actual pwd output
```

**Suites affected:** `container-workdir`

---

### 2. `protocol-support.test.ts` — curl Returns 403 from api.github.com

**Error:**
```
curl: (22) The requested URL returned error: 403
[DEBUG] Agent exit code: 22
expect(result).toSucceed();  ← at protocol-support.test.ts:128
```

**Root Cause:** A test (likely "should pass User-Agent header") that runs `curl -fsS -H "Accept: application/json" https://api.github.com/zen` is receiving a 403. This indicates either a GitHub API rate-limit on the CI runner, or a proxy/header configuration issue introduced by the recent commit.

**Suites affected:** `protocol-support`, `credential-hiding`, `one-shot-tokens` (and possibly others)

---

### 3. `api-proxy-rate-limit.test.ts` — `rate_limit_rejected_total` Missing from Metrics

**Error:**
```
{"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"},"request_id":"req_011CYZS1sooSU88L27xeq8HX"}
expect(result.stdout).toContain('rate_limit_rejected_total');
                       ^
at api-proxy-rate-limit.test.ts:237
```

**Root Cause:** The rate limit test is getting a genuine authentication error (`invalid x-api-key`) from the upstream API rather than being rate-limited by the proxy, so the `rate_limit_rejected_total` counter is never incremented. This could be a test logic issue (the test may need a fake/mock API key) or the proxy is not returning the rejection before forwarding to the upstream.

**Suites affected:** `api-proxy-rate-limit`, `api-proxy-observability`

---

### 4. Secondary: Cleanup Permission Errors

Multiple `rm: cannot remove` errors during cleanup of chroot home directories:
```
rm: cannot remove '/tmp/awf-cmd-1.sh': Operation not permitted
rm: cannot remove '/tmp/awf-lib/one-shot-token.so': Permission denied
rm: cannot remove '/tmp/awf-1772234676021-chroot-home/.ssh/id_rsa': Permission denied

These are non-fatal (cleanup script continues) but indicate files owned by root are not being cleaned up properly.

---

Recommended Actions

1. container-workdir.test.ts: Fix the test to either:

   • Run in non-chroot mode when testing container WORKDIR behavior, OR
   • Update the assertion to match chroot mode behavior (working dir = host user's home ~/)
   • The test comment says "Default WORKDIR in Dockerfile is /workspace" — this is not the right expectation for chroot mode

2. protocol-support.test.ts: Investigate whether the recent commit changed proxy header forwarding. Re-run to check if this is a transient GitHub API rate-limit or a real regression. Check if --ssl-bump or header modification logic changed.

3. api-proxy-rate-limit.test.ts: The test relies on an auth rejection to trigger rate limiting behavior, but the counter only increments for proxy-level rate limit rejections. Verify the test setup properly triggers the rate limiter before forwarding upstream.

4. Cleanup permissions: The cleanup.sh script should use sudo rm -rf for chroot home directories since some files are created as root.

---

Related

• Commit: 3aa408c — "fix issues, remove workflows"
• Workflow run: 22507491192

---

🏥 Automatically investigated by CI Doctor

Generated by CI Doctor

[github-actions]

Same Failures Persist in Run #53

The integration test failures documented here continue to reproduce on the latest main branch commit 0f68cbda ("ci: inject skip-cleanup flag to workflow awf calls").

Updated Failure Counts

Job	Suites Failed	Tests Failed
Protocol & Security Tests	2 of 5	19 of 55
Container & Ops Tests	5 of 7	16 of 66
API Proxy Tests	2 of 3	7 of 27
Domain & Network Tests	cancelled	—

Confirmed Same Root Causes

All four previously identified issues reproduced identically:

1. container-workdir.test.ts — result.stdout contains entrypoint logs including chroot working directory /home/runner, not /workspace
2. no-docker.test.ts — which docker exits 0 (docker-stub.sh is present at /usr/bin/docker); test was written expecting docker to be absent
3. credential-hiding.test.ts — oauth_token found in stdout because the chroot command reads the real ~/.config/gh/hosts.yml on the CI runner
4. api-proxy-rate-limit.test.ts — rate_limit_rejected_total counter never incremented because upstream returns invalid x-api-key before the proxy can reject it

Note on Triggering Commit

The commit 0f68cbda only modifies scripts/ci/postprocess-smoke-workflows.ts (smoke workflow injection) and does not affect integration test behavior. These failures are pre-existing since the earlier commit 3aa408c ("fix issues, remove workflows").

Generated by CI Doctor

[github-actions]

Recurrence: Same failures observed on commit 65f71c5

The same pre-existing integration test failures recurred on run 22732504683 triggered by "fix(deps): resolve minimatch ReDoS and ajv vulnerabilities (#1152)" by @Mossaka.

Failure Comparison

Job	Suites Failed	Tests Failed	vs. Issue #1102
Protocol & Security Tests	2 of 5	19 of 55	Previously 3/5, 20/55 — slight improvement
API Proxy Tests	2 of 3	7 of 27	Same
Container & Ops Tests	5 of 7	16 of 66	Same
Domain & Network Tests	cancelled	—	cancelled (upstream)

Key Details

• container-workdir.test.ts (line 39): Still failing — pwd returns /home/runner instead of /workspace in chroot mode
• api-proxy-rate-limit.test.ts (line 237): Still failing — rate_limit_rejected_total metric not present; upstream returns authentication_error: invalid x-api-key instead of triggering the rate-limiter counter
• credential-hiding.test.ts (line 292): Failing — output contains oauth_token when it shouldn't (grep found credentials in the CI runner's host files via chroot)
• Cleanup permission errors: Still present (rm: cannot remove '/tmp/awf-cmd-1.sh': Operation not permitted, same chroot home permission denied errors)

Conclusion

These failures are not caused by the dependency update in commit 65f71c5. The identical failure counts for Container & Ops and API Proxy jobs confirm these are the same pre-existing bugs tracked here.

---

🏥 Automatically investigated by CI Doctor — run 22733653607

Generated by CI Doctor

[github-actions]

Same failures recurring on commit 8c6047f

The Integration Tests run #22732913573 shows identical failure counts on commit 8c6047f (fix(squid): run Squid container as non-root user):

Job	Suites Failed	Tests Failed
Protocol & Security Tests	2 of 5	19 of 55
API Proxy Tests	2 of 3	7 of 27
Container & Ops Tests	5 of 7	16 of 66
Domain & Network Tests	cancelled	—

These numbers nearly match the original report, confirming the failures are pre-existing and not caused by this commit.

Confirmed failures from logs

Protocol & Security — credential-hiding.test.ts Test 14:

expect(received).not.toContain('oauth_token')
// oauth_token found in output (likely from /tmp/awf-lib/one-shot-token.so binary in chroot)

API Proxy — api-proxy-rate-limit.test.ts:

{"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}
expect(result.stdout).toContain('rate_limit_rejected_total');
// API proxy auth error prevents rate_limit_rejected_total counter from incrementing
```

**Container & Ops** — `no-docker.test.ts`:
```
Expected awf to fail, but it succeeded
// docker binary accessible (via chroot host filesystem mount?)
```
**Container & Ops** — `container-workdir.test.ts`:
```
Expected substring: "/workspace"
// Chroot mode sets workdir to /home/runner, not /workspace
```

### Note on cleanup errors

All 3 failing jobs also show non-fatal cleanup errors:
```
rm: cannot remove '/tmp/awf-cmd-1.sh': Operation not permitted
rm: cannot remove '/tmp/awf-lib/one-shot-token.so': Permission denied

These are root-owned files from inside containers that the runner user can't delete.

The new Squid non-root commit is unaffected by these failures. The root causes remain as described in the original issue above.

Generated by CI Doctor

[github-actions]

New Failure on commit 62787461 — same pattern persists

The Integration Tests workflow run #93 failed again on 2026-03-05 after merging commit 62787461 ("fix(security): secure SSL Bump key wiping via tmpfs (#1156)").

Note: Run #91 on the prior commit (76de95b8) shows the same 3 jobs failing, confirming these failures are pre-existing and not introduced by the SSL Bump commit.

Failure Summary

Job	Suites Failed	Tests Failed
Protocol & Security Tests	2	19
API Proxy Tests	2	7
Container & Ops Tests	5	16
Domain & Network Tests	cancelled	—

Key Failure Details

1. credential-hiding.test.ts (Protocol & Security)

• Test 14: "grep for tokens in hidden files finds nothing" — FAIL
• expect(output).toBe('') at line 272 fails; expect(output).not.toContain('oauth_token') at line 292 fails
• Root cause: result.stdout includes full entrypoint log output (container startup messages including [entrypoint] One-shot token library copied...) which contains the string "oauth_token" within the word context, making the grep result non-empty
• Active fix branch: fix/052-credential-hiding-tests (PR in progress) and fix/054-token-logging-leak

2. api-proxy-rate-limit.test.ts (API Proxy)

• expect(result.stdout).toContain('rate_limit_rejected_total') fails at line 237
• Same authentication error (invalid x-api-key) prevents the counter from being incremented — identical to original issue analysis

3. container-workdir.test.ts (Container & Ops)

• expect(result.stdout.trim()).toContain('/workspace') fails at line 39
• Chroot mode sets working directory to /home/runner instead of /workspace
• Identical to the root cause identified in original issue

4. docker-in-docker-removal.test.ts (Container & Ops)

• "docker command should not be available" — FAIL
• expect(result).toFail() fails — docker command unexpectedly succeeds

Cleanup Permission Errors (non-fatal, still present)

rm: cannot remove '/tmp/awf-cmd-1.sh': Operation not permitted
rm: cannot remove '/tmp/awf-lib/one-shot-token.so': Permission denied

Active Fix Branches

Multiple branches are actively working on fixes:

• fix/052-credential-hiding-tests — Integration Tests run safe-inputs not working with firewall #104 in progress
• fix/054-token-logging-leak
• fix/042-chroot-escape-test-gaps
• fix/038-toctou-test-coverage
• fix/104-ip-tls-bypass
• fix/037-toctou-ssl-bump

Generated by CI Doctor

[github-actions]

🔄 Still failing — run 22733686235

Triggered by commit b8596445: fix(security): eliminate TOCTOU race conditions in ssl-bump.ts (#1159)

The same pre-existing failures persist (this commit only touched src/ssl-bump.ts):

Job	Result	Failures
Container & Ops Tests	❌ failure	5 suites, 16 tests
API Proxy Tests	❌ failure	2 suites, 7 tests
Protocol & Security Tests	❌ failure	2 suites, 19 tests
Domain & Network Tests	⚠️ cancelled	—

Confirmed failure details

1. container-workdir.test.ts — still failing with the same chroot WORKDIR mismatch:

Expected substring: "/workspace"
Received: "[entrypoint] Chroot working directory: /home/runner\n/home/runner"
```

**2. `credential-hiding.test.ts` (Test 14)** — `oauth_token` found where it shouldn't be:
```
expect(received).not.toContain("oauth_token")
```

**3. `api-proxy-rate-limit.test.ts`** — authentication error instead of rate limit rejection:
```
{"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}
expect(result.stdout).toContain('rate_limit_rejected_total')  ← fails
```

**4. `no-docker.test.ts`** — docker CLI is available but test expects it to fail:
```
Expected awf to fail, but it succeeded
// Should fail because docker-cli is not installed

Note

A branch fix/052-credential-hiding-tests is currently in progress (run 22734391307) that may address some of these. Recent main-branch runs have all been failing:

• Run 22733573480 — fix(security): stop logging partial token values
• Run 22733226039 — fix(security): secure SSL Bump key wiping via tmpfs
• Run 22732951286 — fix(security): disable IPv6 via sysctl when ip6tables unavailable

Generated by CI Doctor