feat: Add comprehensive pipeline validation test infrastructure

[debasishbsws]

Claude code description:

Introduce a new testing framework for validating pipeline checkers with both positive and negative test cases. This ensures pipeline validators work correctly before release and prevents regressions.

What's New:

1. Test Files

• tests/docs-test.yaml: Validates docs pipeline
  • Positive test: giflib-doc (valid Wolfi package with docs)
  • Negative tests: glibc (no docs), binaries, empty packages
  • All negative tests capture output for debugging
• tests/README.md: Documents test structure and best practices
• .gitignore: Ignore .DS_Store and tests/packages/ (build artifacts)

2. Test Targets
Reorganized into three categories:

• make test-melange: Tests main tw package
• make test-projects: Tests individual project tools (e.g., make test-project/gosh)
• make test-pipelines: Tests pipeline validators (new)
• make test-all: Runs complete test suite

3. Configuration Updates

• Set provider-priority: 0 for proper Wolfi precedence
• Added TEST_DIR and TEST_OUT_DIR for test package isolation
• Fixed MELANGE_TEST_OPTS with proper repository and keyring configuration
• Added architecture and pipeline-dirs settings

4. Pattern Rule Changes

• Renamed test-% to test-project/% to prevent naming conflicts
• Uses slash separator for clarity (e.g., test-project/gosh)

5. CI/CD Updates

• Added test-pipelines step to validate checkers
• Added granular targets: build-pipeline-tests, run-pipeline-tests
• Added informative echo statements for visibility
• Updated workflow to run all three test types

Verified Working:

• ✓ make test-melange
• ✓ make test-projects
• ✓ make test-pipelines
• ✓ make test-all
• ✓ make test-project/package-type-check

[aborrero]

I like the idea of using real world packages to verify pipelines are working as expected. However, the proposed infrastructure results in a significant amount of code duplication (melange yamls for the test infra).

I propose a slightly different approach: to autogenerate them at test time.

Imagine this testcase1.yaml file:

name: my positive test for test/tw/docs
packages:
 - glibc-doc
 - someother-doc
 - yetanother-doc
 pipelines:
   - uses: test/tw/docs
exit_code: 0

Imagine this other testcase2.yaml file:

name: my negative test for test/tw/docs
packages:
 - somerandom-nodocs-package
 - somerandom-nodocs-package2
 pipelines:
  - uses: test/tw/docs
exit_code: 1

We could have a number of yaml files like these.

Then, we will have a runner script (in python, go, whatever) that for each package, generates the same melange yamls as you envisioned, then copies the pipelines as defined in the test case, executes the tests, and verifies the exit code.

What do you think?

We could have several pipelines in a single test case:

name: testing multiple pipelines on the same test case
packages:
 - somerandom-package
 - somerandom-package2
 pipelines:
  - uses: test/tw/contains-files
    with:
      files: |
         /somefile1
         /somefile2
  - uses: test/tw/virtualpackage
    with:
       virtual-pkg-name: something
 exit_code: 0

The behavior is the same, our runner will generate the melange yamls for each package and copy all the pipelines, test, and verify the exit_code.

[aborrero]

good work! some comments inline.

[aborrero]

some cleanups are still required.

[aborrero]

maybe have a prompt here so is the same format as the other commands.

[debasishbsws]

Why manual tests are required:

1. Edge cases require synthetic packages: Testing the validator's
   behavior with zero headers, headers in wrong paths, or partially
   invalid headers requires constructing artificial packages. Real packages
   would never be built this way.

2. Testing the validator itself, not packages: These tests verify
   the header-check tool behaves correctly when encountering malformed
   or edge-case packages. The goal is to ensure the validator catches
   problems.

3. Negative test cases: Several test cases (no headers, wrong paths)
   are designed to fail. Standard package tests expect success.

The header-check-test.yaml file provides three critical test scenarios:

• Packages with zero headers (tests error detection) BUG we found while testing.
• Headers in non-standard paths like /opt/include (tests path validation)
• Mix of valid and invalid headers (tests partial failure reporting)

---

I have found and fix(header-check): Fail when no headers found instead of false positive
ca52d2a
Previously, header-check would exit 0 (success) when a package contained
zero headers, reporting '0/0 passed'. This created a false positive where
packages without any headers would incorrectly pass validation.

This fix adds a HEADERS_TESTED counter to track whether any headers were
actually tested. If zero headers are found, the script now exits with an
error message indicating no headers were found in the specified packages
or files.

[debasishbsws]

@aborrero as this Pr was created first for the manual tests as describe in the pr description. I am letting this as it was previously and creating a new Pr with the autogen test infra.

[debasishbsws]

@aborrero the draft Pr is #286. I will start working on that after this gets merged.

[aborrero]

LGTM.