[Feature]: API Test Coverage Report & OpenAPI Diff Highlights (per endpoint/method/path)

[wnghdcjfe]

Current Situation/Problem

When using itdoc, there is no built-in way to understand:

How much of my API surface is covered by tests generated/executed by itdoc, and

What changed between the previous and current API versions (added/removed/modified endpoints), highlighted in a human-readable way.

This creates friction for CI/CD and release reviews:

We can’t quickly answer “Which endpoints lack scenarios?” or “Did coverage regress?”

Reviewers can’t see “what changed this release” without running separate diff tools.

Teams with monorepos and multiple services struggle to keep a single, consistent view of coverage and changes over time.

Proposed Feature

Add a Coverage Report and OpenAPI Diff Highlights feature, surfaced both as:

• CLI flags (for CI pipelines), and
• Markdown/HTML artifacts (for human review).

1) Coverage Report (per endpoint/method/path)

• Compute “endpoints with ≥1 scenario” / “endpoints with 0 scenarios” and overall coverage ratio.

• Break down by HTTP method and by path with scenario counts (happy/edge/error).

• Export as:

  • Markdown table (for PR comments / docs),
  • JSON (for machine processing),
  • CLI summary (stdout).

CLI

itdoc build \
  --out docs/api \
  --coverage \
  --coverage-json docs/api/coverage.json \
  --coverage-md docs/api/coverage.md

Sample CLI Output

Coverage Summary
Endpoints total: 42
Covered (≥1 scenario): 31
Uncovered: 11
Coverage: 73.8%

Top uncovered:
  - GET /v1/reports/{id}
  - POST /v1/uploads
  - DELETE /v1/users/{id}
...

Sample Markdown Table (coverage.md)

### Endpoint Coverage

| Path                  | Method | Scenarios | Notes              |
|-----------------------|--------|-----------|--------------------|
| /v1/users             | GET    | 3         | happy, auth, error |
| /v1/users             | POST   | 2         | happy, validation  |
| /v1/users/{id}        | GET    | 0         | **UNCOVERED**      |
| /v1/reports/{id}      | GET    | 0         | **UNCOVERED**      |
| /v1/uploads           | POST   | 1         | happy              |

**Totals:** 31/42 endpoints covered (73.8%)

Sample JSON (coverage.json)

{
  "summary": {
    "endpoints_total": 42,
    "endpoints_covered": 31,
    "endpoints_uncovered": 11,
    "coverage_ratio": 0.738
  },
  "by_path_method": [
    { "path": "/v1/users", "method": "GET", "scenarios": ["happy","auth","error"] },
    { "path": "/v1/users", "method": "POST", "scenarios": ["happy","validation"] },
    { "path": "/v1/users/{id}", "method": "GET", "scenarios": [] }
  ],
  "uncovered": [
    { "path": "/v1/reports/{id}", "method": "GET" },
    { "path": "/v1/uploads", "method": "POST" }
  ]
}

2) OpenAPI Diff Highlights (added/removed/modified)

• Compare current OpenAPI against a baseline (previous release or main branch artifact).

• Produce high-signal changelog:

  • Added endpoints (new paths/methods),
  • Removed endpoints,
  • Modified (params, requestBody schema, responses, status codes, security).

CLI

itdoc build \
  --out docs/api \
  --diff --diff-base artifacts/openapi-prev.json \
  --diff-json docs/api/openapi-diff.json \
  --diff-md docs/api/openapi-diff.md

Sample Markdown (openapi-diff.md)

### API Changes (v1.12 → v1.13)

#### Added
- `POST /v1/uploads` (multipart/form-data)
- `GET /v1/health` (200 OK)

#### Removed
- `DELETE /v1/users/{id}`

#### Modified
- `GET /v1/users`:
  - response `200` schema: added field `role`
  - query param `status`: enum extended (`active`, `inactive`, `pending`)
- Security:
  - `POST /v1/auth/refresh` now requires `BearerAuth`

Sample JSON (openapi-diff.json)

{
  "added": [
    { "path": "/v1/uploads", "method": "POST" },
    { "path": "/v1/health", "method": "GET" }
  ],
  "removed": [
    { "path": "/v1/users/{id}", "method": "DELETE" }
  ],
  "modified": [
    {
      "path": "/v1/users",
      "method": "GET",
      "changes": {
        "responses": { "200": { "schema": { "addedProperties": ["role"] } } },
        "parameters": { "query:status": { "enumAdded": ["pending"] } }
      }
    },
    {
      "path": "/v1/auth/refresh",
      "method": "POST",
      "changes": { "security": { "added": ["BearerAuth"] } }
    }
  ]
}

3) Combined “Release View”

Optionally generate a single HTML/MD “Release Report” that contains:

• Coverage summary + tables,
• Diff highlights (added/removed/modified),
• “New endpoints without scenarios” callout.

itdoc build \
  --out docs/api \
  --coverage --diff --diff-base artifacts/openapi-prev.json \
  --release-report docs/api/release-report.md

---

Alternatives (Optional)

1. External tools (e.g., standalone OpenAPI diff utilities, custom scripts)

   • Pros: Flexible, already exist.
   • Cons: Fragmented pipeline; no shared config; harder for teams to adopt consistently.

2. CI plugins (custom GitHub Actions)

   • Pros: Can be built today.
   • Cons: Loses the “single-tool” value of itdoc; harder to guarantee consistent formatting and schema alignment.

3. Editor extensions for on-the-fly diff/coverage

   • Useful for local dev, but CI/reporting still needed.

---

Nice-to-Have / Acceptance Criteria

• --coverage produces JSON+MD; prints a one-line summary to stdout with non-zero exit if coverage threshold (e.g., --coverage-min=0.8) not met.
• --diff --diff-base <file> produces JSON+MD with clear Added/Removed/Modified sections; non-zero exit on breaking changes if --fail-on-breaking set.
• Works in monorepos; supports path filters (e.g., --include /v1/*).
• Output sizes remain small; runtime ≤ a few seconds on medium APIs.
• Documented examples + sample CI workflow in README.

감사합니다!

Alternatives (Optional)

No response

Additional Information

No response