Intercept HTTPS CONNECT tunnels in mocked backend mode

[cbeauchesne]

Motivation

In mocked_backend mode, the agent sends HTTPS requests to backend (trace.agent.datadoghq.com, api.datadoghq.com, etc.) using HTTP_PROXY, which means it opens HTTPS CONNECT tunnels through the proxy. When the real backend is unreachable, mitmproxy fails the tunnel at the TLS setup, and request() is not fired.

And because request() never fired, the existing mock logic (which returns synthetic 202/200 responses) was bypassed entirely for these flows.

Furthermore, if any of those requests is a connectivity checks for the agent, it then consider that the backend is down, and does not try to send any data. In conequence, any assertion on agent side fails.

Changes

• Stub TLS server (running() hook): when mocked_backend=True, starts a minimal asyncio TLS server on 127.0.0.1:11112 using the mitmproxy CA cert. It accepts connections and keeps them alive. Mitmproxy uses it purely to complete the TLS handshake leg; no actual data is forwarded through it.
• http_connect() hook: logs every CONNECT before any tunnel is established. For mocked agent flows, redirects the CONNECT destination to the stub server and saves the original (host, port) in _original_connects, keyed by client_conn.id.
• response(): restores the original (host, port) from _original_connects when building the saved JSON, so the
  file correctly reflects the real backend destination rather than 127.0.0.1.
• mitmproxy options: upstream_cert=False (generate TLS certs from SNI without fetching the upstream cert) and ssl_insecure=True (don't verify the stub server's self-signed cert) when in mocked mode.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

utils/proxy/core.py                                                     @DataDog/system-tests-core
utils/proxy/ports.py                                                    @DataDog/system-tests-core

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d682ab8ceb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Use exact match for validate endpoint detection

In mocked backend mode this condition uses substring membership (path in "/api/v1/validate") instead of an exact path comparison, so any path that happens to be a substring (for example "/" or "/api/v1") will incorrectly get the validate-specific 200 {"valid": true} response rather than the default mocked response. This can silently mask unexpected agent requests and alter test behavior for non-validate endpoints on the agent proxy port.

Useful? React with 👍 / 👎.