appsec: add smoke tests for apm standalone by florentinl · Pull Request #6181 · DataDog/system-tests

florentinl · 2026-02-02T14:02:30Z

Motivation

Test that AppSec is correctly enabled and functional in APM Standalone mode (with infra disabled at the agent level).

To do so, I setup a testing class that performs AAP smoke tests using only data intercepted at the agent level. This testing class can then be used through inheritance to create different test suites linked to different features and manifest entries.

Changes

Add appsec helpers in interfaces.agent to perform assertions on traces intercepted after the agent
Add smoke tests to check:
- Threat detection
- RASP
- Remote configuration of rules
- Telemetry emission
- API Security
  => This ensures that all communication protocols used by AAP (traces, trace stats, telemetry and RC) keep getting proxied by the agent.
Add two scenarios and a feature for APM Standalone and ASM Standalone + APM Standalone

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-02-02T14:03:01Z

CODEOWNERS have been resolved as:

tests/appsec/smoke_tests/__init__.py                                    @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/smoke_tests/test_apm_standalone.py                         @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/smoke_tests/utils.py                                       @DataDog/asm-libraries @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
docs/edit/agent-interface-validation-methods.md                         @DataDog/system-tests-core
manifests/agent.yml                                                     @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/interfaces/_agent.py                                              @DataDog/system-tests-core
utils/proxy/_deserializer.py                                            @DataDog/system-tests-core
utils/proxy/traces/trace_v1.py                                          @DataDog/system-tests-core

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0326ddddad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

manifests/dotnet.yml

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a6383aa34

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

manifests/ruby.yml

tests/appsec/smoke_tests/utils.py

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c6321e6888

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

utils/_context/_scenarios/__init__.py

…s flaky Separate test classes for each scenario so they can have independent manifest declarations: - Test_AppSecAPMStandalone_*: APPSEC_APM_STANDALONE (tracing enabled) - Test_AppSecStandaloneAPMStandalone_*: APPSEC_STANDALONE_APM_STANDALONE (tracing disabled — traces without appsec data silently dropped) Both sets use identical clean base classes with no workarounds. The Java standalone variants for Threats, Rasp, ApiSecurity, and UserEvents are marked flaky (APPSEC-60872): the Java tracer drops traces from the first request to each code path when DD_APM_TRACING_ENABLED=false. Updated manifests: java, golang, nodejs, php, ruby, cpp_nginx. File-level entries (dotnet, python, agent) already cover both class sets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f357eb7060

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tests/appsec/smoke_tests/utils.py

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from 8b69cf1 to e67144b Compare February 2, 2026 14:17

This comment has been minimized.

Sign in to view

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch 2 times, most recently from 20a2e93 to 8a60060 Compare March 5, 2026 10:55

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch 4 times, most recently from fd4926c to 8679f7a Compare March 26, 2026 14:15

florentinl marked this pull request as ready for review March 27, 2026 14:59

florentinl requested review from a team as code owners March 27, 2026 14:59

florentinl requested review from jandro996, manuel-alvarez-alvarez, r1viollet and taegyunkim and removed request for a team March 27, 2026 14:59

chatgpt-codex-connector bot reviewed Apr 8, 2026

View reviewed changes

manifests/dotnet.yml Outdated Show resolved Hide resolved

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch 3 times, most recently from 6888a95 to 2a6383a Compare April 9, 2026 15:47

chatgpt-codex-connector bot reviewed Apr 9, 2026

View reviewed changes

manifests/ruby.yml Show resolved Hide resolved

florentinl requested a review from cbeauchesne April 10, 2026 09:41

cbeauchesne reviewed Apr 10, 2026

View reviewed changes

tests/appsec/smoke_tests/utils.py Show resolved Hide resolved

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from 99c6401 to 755553f Compare April 13, 2026 07:31

florentinl requested a review from cbeauchesne April 13, 2026 11:57

cbeauchesne approved these changes Apr 13, 2026

View reviewed changes

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from 755553f to c6321e6 Compare April 13, 2026 12:53

chatgpt-codex-connector bot reviewed Apr 13, 2026

View reviewed changes

utils/_context/_scenarios/__init__.py Show resolved Hide resolved

florentinl and others added 16 commits April 13, 2026 16:34

appsec: add smoke tests for apm standalone

534e17f

comment to fill things

acbf3bb

enable tests for the different tracers

9a7a5a4

fix format

8c5b6d1

v1 trace payload support + small refactor

d707593

linter

633a695

move metastruct deserialization to the proxy

208c1e2

add extra coverage

dd331c2

update manifests

3e042b6

update docs

d863eb8

move files + renaming

5a55307

fix ruby manifest

f88f9ae

Fix docstring

f818831

tiny reformatting

7105502

Add APM standalone RFC

f357eb7

florentinl force-pushed the florentin.labelle/APPSEC-60883/infra-less-system-tests branch from c6321e6 to f357eb7 Compare April 13, 2026 14:34

chatgpt-codex-connector bot reviewed Apr 13, 2026

View reviewed changes

tests/appsec/smoke_tests/utils.py Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

appsec: add smoke tests for apm standalone#6181

appsec: add smoke tests for apm standalone#6181
florentinl wants to merge 16 commits intomainfrom
florentin.labelle/APPSEC-60883/infra-less-system-tests

florentinl commented Feb 2, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 2, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

florentinl commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

florentinl commented Feb 2, 2026 •

edited

Loading

github-actions bot commented Feb 2, 2026 •

edited

Loading