Add an SDK Tier Audit skill and agentic workflow#1503
Open
jeffhandley wants to merge 7 commits intomodelcontextprotocol:mainfrom
Open
Add an SDK Tier Audit skill and agentic workflow#1503jeffhandley wants to merge 7 commits intomodelcontextprotocol:mainfrom
jeffhandley wants to merge 7 commits intomodelcontextprotocol:mainfrom
Conversation
Add a weekly agentic workflow that performs SEP-1730 tier audits of the C# MCP SDK using the mcp-sdk-tier-audit skill from the conformance repository. The workflow: - Runs weekly on Thursdays (fuzzy ~6:30am EST) and on manual dispatch - Supports two scopes: Conformance + Repo Health (default) and Repo Health - Allows overriding the C# SDK and conformance repo/branch targets - Files issues with tier results, auto-closing previous audit issues - Uses a dedicated AUDIT_PAT pool for Copilot engine authentication - Skips scheduled runs on forks; allows manual dispatch anywhere The audit logic is extracted into a reusable skill at .github/skills/sdk-tier-audit/SKILL.md with cross-platform instructions (macOS/Linux and Windows) so it can also be invoked locally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add 'output' input: 'Create Issue' (default) or 'Action Summary'
When 'Action Summary', the report is written to the step summary
and uploaded as an artifact but no issue is created.
- Condense repo/branch inputs into owner/repo:branch format:
csharp_sdk: 'modelcontextprotocol/csharp-sdk:main'
conformance: 'modelcontextprotocol/conformance:main'
Parsed with bash parameter expansion (${var%%:*} / ${var#*:})
- Add post-steps to upload /tmp/audit-report.md as an artifact
(90-day retention, ignore if missing)
- Executive summary now uses bullet points instead of a paragraph
- Audit report always written to $GITHUB_STEP_SUMMARY regardless
of output mode, so the summary page always shows results
- Agent writes a single /tmp/audit-report.md combining executive
summary + assessment + remediation, used for both issue body
and action summary
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add tools.github.min-integrity: approved for content guard policy - Add pull-requests: read permission (required by default toolsets) - Add FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true env var to opt into Node 24 early and suppress the Node 20 deprecation warning - Add max: 1 to create-issue safe output for explicit limit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two issues fixed: 1. Tier logic: The prompt was telling the agent to 'Apply tier logic' itself (Step 2.4). Now the prompt explicitly says: 'Do not apply your own tier logic or scoring — use only the conformance skill's thresholds, rules, and templates.' All tier determination is delegated to the conformance repo's mcp-sdk-tier-audit skill. 2. Action summary: The report was not appearing on the workflow summary page because the agent wasn't writing to $GITHUB_STEP_SUMMARY reliably. Now Step 3 is restructured with explicit requirements: - MUST write /tmp/audit-report.md (artifact) - MUST cat it to $GITHUB_STEP_SUMMARY (action summary) - Both required BEFORE Step 4 (publish) - 'Action Summary' mode now explicitly calls noop - Issue body must be identical to the action summary content Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeffhandley
force-pushed
the
jeffhandley/conformance-workflow
branch
2 times, most recently
from
9007907 to
2e5451c
Compare
Contributor
Author
|
@mikekistler / @ericstj I'm contemplating whether we should:
If we go with a discussions section/categories approach each report could just be posted as a new entry without worrying about closing old issues or having those issues themselves show up in our issue counts. |
ericstj
reviewed
Apr 6, 2026
| safe-outputs: | ||
| create-issue: | ||
| title-prefix: "[C# SDK Tier Audit] " | ||
| labels: [automation] |
Contributor
There was a problem hiding this comment.
Should it assign a priority?
Contributor
What action do you expect people to take as a result of these? Is it just informational, or might they need to take action to get things back in SLA? If the latter, then an issue might make more sense. Close it when no more action is needed. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an automated “SDK Tier Audit” capability to the C# SDK repo via a local skill and a GitHub Agentic Workflows (gh-aw) workflow that can run on a schedule or manually, and publish results (issue or action summary).
Changes:
- Introduces an agentic workflow (
sdk-tier-audit) that runs tier-check + an AI-assisted evaluation flow and can publish results via Safe Outputs. - Adds a local
sdk-tier-auditskill wrapper plus a reusable composite action to select a Copilot PAT from a secret pool. - Adds repo configuration/docs updates to support generated workflow lockfiles and release process guidance.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
.vscode/settings.json |
Enables Copilot for markdown in VS Code workspace settings. |
.github/workflows/sdk-tier-audit.md |
gh-aw source workflow definition and agent prompt for running the audit and publishing results. |
.github/workflows/sdk-tier-audit.lock.yml |
Generated GitHub Actions workflow compiled from the .md source. |
.github/skills/sdk-tier-audit/SKILL.md |
Local skill wrapper instructions for running the audit and delegating tier logic to conformance repo skill. |
.github/release-readme.md |
Documents the manual release process and related skills. |
.github/aw/actions-lock.json |
Locks selected GitHub Actions tags to SHAs for gh-aw compilation. |
.github/agents/agentic-workflows.agent.md |
Adds an agent “dispatcher” doc for working with gh-aw in this repo (including PAT pool guidance). |
.github/actions/select-copilot-pat/README.md |
Documents PAT pool usage and integration guidance. |
.github/actions/select-copilot-pat/action.yml |
Composite action that selects a random non-empty PAT secret index (0–9). |
.gitattributes |
Marks *.lock.yml workflows as generated and sets merge strategy to avoid conflicts. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce an SDK Tier Audit skill and agentic workflow to automate weekly runs of the audit, producing an issue in this repo to capture the audit report.
The workflow offers execution modes for:
Other notes:
Example results: