Add live API monitoring to detect schema drift #87
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements daily scheduled tests that make real API calls to detect schema drift and breaking changes from upstream Mapbox APIs. **Why this exists:** Mapbox APIs don't follow strict semantic versioning - responses can change without notice. This monitoring system provides early warning when schemas drift, preventing production failures. **How it works:** - GitHub Actions runs daily at midnight UTC - Tests call Mapbox APIs with representative queries - Responses are validated against our Zod output schemas - Failures are saved to test/failures/ for analysis - GitHub issues are created automatically with failure details - Test artifacts are uploaded for investigation **Monitored tools:** - SearchAndGeocodeTool (6 test queries) - CategorySearchTool (5 test categories) - ReverseGeocodeTool (4 test coordinates) **Running locally:** \`\`\`bash RUN_API_MONITORING=true npm test -- test/integration/live-api-monitoring.test.ts \`\`\` **Files:** - test/integration/live-api-monitoring.test.ts: Integration tests - .github/workflows/api-monitoring.yml: Daily scheduled workflow - test/integration/README.md: Complete documentation - .gitignore: Exclude test/failures/ directory This complements PR #73's non-fatal validation approach: - PR #73 makes validation failures non-fatal (resilience) - This PR detects schema drift early (observability) Together they provide resilient production behavior and early warning when schemas change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Addresses GitHub Advanced Security suggestion to follow principle of least privilege by limiting GITHUB_TOKEN permissions. Permissions added: - contents: read - for checking out code - issues: write - for creating/updating monitoring issues - actions: read - for accessing workflow run information 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Removed: - --reporter=verbose flag to minimize detailed output - tee test-output.log to avoid writing sensitive data to disk GitHub's automatic secret redaction will still protect the token, but this reduces the attack surface by minimizing logged details. Security improvements: - Less verbose output = less potential for token leakage - No file writing = no temporary sensitive data on disk - Standard test reporter provides sufficient monitoring output 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Contributor
Author
|
Closing this PR - moving API monitoring to private repo (mapbox/mcp-server-private#105) where we can use verbose logging and store detailed failure artifacts without security concerns. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements daily scheduled integration tests that make real API calls to detect schema drift and breaking changes from upstream Mapbox APIs.
This addresses the issue where Mapbox APIs don't follow strict semantic versioning - responses can change without notice, breaking our output schemas.
What This Adds
Monitoring System
Test Coverage
SearchAndGeocodeTool: 6 representative queries (Starbucks, Tokyo, addresses, generic terms)CategorySearchTool: 5 common categories (restaurant, coffee, hotel, gas_station, parking)ReverseGeocodeTool: 4 major cities (San Francisco, Tokyo, London, Paris)Files
test/integration/live-api-monitoring.test.ts: Integration test suite.github/workflows/api-monitoring.yml: Daily scheduled workflowtest/integration/README.md: Complete documentation.gitignore: Excludestest/failures/directoryHow It Works
RUN_API_MONITORING=trueto run)test/failures/with timestampapi-monitoring,schema-validation,needs-triageSecurity
Secret Protection (Public Repo)
The workflow uses
MAPBOX_ACCESS_TOKENsecret safely through multiple protection layers:GitHub's Built-in Protection (Primary):
***Defense in Depth Measures:
permissionsblock (least privilege:contents: read,issues: write,actions: read)Note: Mapbox APIs require tokens as query parameters (not headers), so tokens appear in URLs. GitHub's automatic redaction masks these in all workflow logs before display.
Running Locally
Relationship to PR #73
This complements PR #73's non-fatal validation approach:
Together they provide:
Testing
All existing tests pass. The 3 new monitoring tests are properly skipped by default:
🤖 Generated with Claude Code