Add automated pipeline testing framework for tw pipelines#286
Draft
debasishbsws wants to merge 2 commits intochainguard-dev:mainfrom
Draft
Add automated pipeline testing framework for tw pipelines#286debasishbsws wants to merge 2 commits intochainguard-dev:mainfrom
debasishbsws wants to merge 2 commits intochainguard-dev:mainfrom
Conversation
Introduces a Go-based test runner that auto-generates Melange test configs from declarative YAML test cases, eliminating manual test file duplication. Key additions: - pipeline-runner/: Go CLI tool that parses test cases, generates Melange configs, executes tests, and reports failures - Test case YAML format with expect_pass for positive/negative testing - Initial test suites for contains-files, docs, emptypackage, and metapackage pipelines - Debug and verbose logging modes for troubleshooting - Generated configs auto-gitignored to keep source control clean Benefits: - 1:1 mapping: one test case = one package + one expected outcome - Real package testing against actual Wolfi packages - Automated validation with detailed failure reporting - Reduces test maintenance burden from N config files to single test suite definitions See tests/AUTOGEN-TESTS.md for complete documentation on architecture, usage, and test creation guidelines. Signed-off-by: Debasish Biswas <debasishbsws.dev@gmail.com>
aborrero
reviewed
Jan 28, 2026
| @@ -0,0 +1,23 @@ | |||
| name: Contains-files pipeline validation tests | |||
Contributor
There was a problem hiding this comment.
ok, so assuming we have 2 flavors of test: manual and autogenerated, maybe it makes sense to separate them in different directores:
test/manual/testcases/
test/autogen/testcases/
We can put the pipeline-runner code inside test/autogen/ as well, so everything is self-contained.
Member
Author
There was a problem hiding this comment.
I am thinking about this structure
tests/
├── manual/ # Manual tests (full melange YAML)
│ └── header-check.yaml
├── docs.yaml # Suite tests (declarative format)
├── contains-files.yaml
├── emptypackage.yaml
├── metapackage.yaml
├── runner/ # Test runner tool (infrastructure)
│ ├── main.go
│ ├── go.mod
│ └── pipeline-runner
├── packages/ # Build artifacts (gitignored)
├── generated/ # Auto-generated configs (gitignored)
├── README.md
└── .gitignore
Member
Author
There was a problem hiding this comment.
the normal autogen is the default test resides inside the tests folder and the manual is in a seperate folder
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces a Go-based test runner that auto-generates Melange test configs from declarative YAML test cases, eliminating manual test file duplication.
Key additions:
Benefits:
This will not replace the manual test infra we are adding here, but complements it for easier configuration for normal everyday package test.