Implement comprehensive cross-platform testing for native and WASM targets

[avrabe]

Summary

Establish comprehensive testing infrastructure that validates functionality across all supported platforms (Windows, macOS, Linux) and targets (native, WASM32-WASIP2, component model) with automated CI/CD integration.

Background

With WASM support (#26), multiple storage backends (#29), component model integration (#30), and various host runtimes (#32), we need robust testing to ensure:

• Feature parity across native and WASM targets
• Compatibility across different operating systems
• Performance characteristics are maintained
• Integration between components works correctly
• Regression detection for complex multi-target scenarios

Implementation Tasks

Test Infrastructure Setup

Multi-Target Test Matrix

# CI Matrix Configuration
targets:
  - native-linux
  - native-macos  
  - native-windows
  - wasm32-wasip2
  - component-model

features:
  - default
  - full (all features enabled)
  - minimal (core features only)
  - no-std (embedded compatibility)

runtimes:
  - native
  - wasmtime
  - wrt
  - browser (Chrome, Firefox, Safari)

Test Categories

• Unit Tests - Core functionality per crate
• Integration Tests - Cross-crate functionality
• Component Tests - WASM component behavior
• System Tests - End-to-end scenarios
• Performance Tests - Benchmarking and regression detection
• Compatibility Tests - Cross-platform validation

Platform-Specific Testing

Native Platform Tests

• File system operations across OS variants
• Network transport behavior differences
• Authentication storage backend compatibility
• Process management and signal handling
• Platform-specific error handling

WASM Target Tests

• WASI interface functionality validation
• Component model interface compliance
• Host integration compatibility
• Resource management and cleanup
• Performance characteristics in WASM context

Browser Environment Tests

• Web API integration (IndexedDB, WebSockets)
• Service Worker deployment scenarios
• Security model compliance
• Memory management in browser context
• Concurrent execution with Web Workers

Storage Backend Testing

Cross-Backend Compatibility

#[cfg(test)]
mod storage_tests {
    use super::*;
    
    // Test all storage backends with same interface
    #[test_matrix(
        storage_backend = [FileStorage, MemoryStorage, WasiStorage, JsonLinesStorage],
        platform = [native, wasm32_wasip2]
    )]
    async fn test_storage_operations(storage: StorageBackend, platform: Platform) {
        let storage = create_storage_backend(storage, platform).await;
        
        // Common test suite
        test_basic_crud_operations(&storage).await;
        test_concurrent_access(&storage).await;
        test_error_handling(&storage).await;
        test_persistence_across_restarts(&storage).await;
    }
}

Data Migration Testing

• Migration between storage backend types
• Data integrity during migration
• Rollback capability validation
• Performance impact measurement

Transport Layer Testing

Multi-Transport Testing

• stdio transport across platforms
• HTTP transport with different networking stacks
• Component model transport validation
• Transport fallback and negotiation
• Error handling and recovery

Network Condition Simulation

• Latency and packet loss simulation
• Connection interruption testing
• Bandwidth limitation testing
• Firewall and proxy testing

Component Model Testing

Component Isolation Testing

• Resource access boundary validation
• Memory isolation between components
• Capability-based security testing
• Component lifecycle management
• Inter-component communication validation

Composition Testing

• Multi-component application deployment
• Component dependency resolution
• Version compatibility testing
• Hot-swapping and updates
• Performance of composed applications

Performance Testing Infrastructure

Benchmark Suites

use criterion::{criterion_group, criterion_main, Criterion};

fn benchmark_mcp_operations(c: &mut Criterion) {
    let mut group = c.benchmark_group("mcp_operations");
    
    // Native benchmarks
    group.bench_function("native/list_resources", |b| {
        b.iter(|| native_list_resources())
    });
    
    // WASM benchmarks
    group.bench_function("wasm/list_resources", |b| {
        b.iter(|| wasm_list_resources())
    });
    
    // Component benchmarks
    group.bench_function("component/list_resources", |b| {
        b.iter(|| component_list_resources())
    });
}

Performance Regression Detection

• Automated performance baseline tracking
• Performance alerts for significant regressions
• Memory usage monitoring
• Startup time benchmarking
• Throughput and latency measurement

CI/CD Integration

GitHub Actions Workflows

name: Cross-Platform Tests

on: [push, pull_request]

jobs:
  test-native:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        rust: [stable, beta, nightly]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v3
      - name: Setup Rust
        uses: actions-rs/toolchain@v1
        with:
          toolchain: ${{ matrix.rust }}
      - name: Run tests
        run: cargo test --all-features

  test-wasm:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup WASM target
        run: rustup target add wasm32-wasip2
      - name: Install wasmtime
        run: curl https://wasmtime.dev/install.sh -sSf | bash
      - name: Build WASM
        run: cargo build --target wasm32-wasip2
      - name: Test WASM
        run: cargo test --target wasm32-wasip2

  test-components:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup component tools
        run: |
          cargo install wasm-tools
          cargo install wit-bindgen-cli
      - name: Test components
        run: |
          cargo component build
          cargo component test

Performance Monitoring

• Continuous benchmarking with historical tracking
• Performance dashboard integration
• Automated performance reports
• Regression detection and alerting

Testing Tools and Utilities

Test Harness Development

• Unified test runner for all targets
• Component test framework
• Mock host implementations for testing
• Test data generation and management
• Test result aggregation and reporting

Development Testing Tools

# Unified test runner
mcp-test run --target=all --features=full
mcp-test run --target=wasm32-wasip2 --runtime=wasmtime
mcp-test run --component --composition=examples/multi-app

# Performance testing
mcp-test bench --compare-targets
mcp-test profile --output=flamegraph --target=wasm32-wasip2

# Compatibility testing
mcp-test compat --storage-backends=all
mcp-test compat --transports=all --platforms=all

Test Data and Scenarios

Realistic Test Scenarios

• Large file operations and streaming
• High-concurrency stress testing
• Long-running stability testing
• Memory pressure testing
• Network partition and recovery

Test Data Management

• Reproducible test data sets
• Environment-specific test configuration
• Test data cleanup and isolation
• Sensitive data handling in tests

Quality Assurance

Code Coverage

• Coverage tracking across all targets
• Platform-specific coverage reporting
• Integration test coverage measurement
• Coverage regression prevention

Static Analysis

• Cross-platform linting with clippy
• Security vulnerability scanning
• Dependency audit across targets
• Documentation completeness checking

Integration Points

Development Workflow Integration

• Pre-commit hooks for multi-target testing
• IDE integration for cross-platform development
• Automated test selection based on changes
• Fast feedback for common development scenarios

Release Process Integration

• Automated release testing across all platforms
• Performance validation before releases
• Compatibility testing with previous versions
• Documentation validation and generation

Acceptance Criteria

• All tests pass on native and WASM targets
• Performance benchmarks within acceptable thresholds
• Cross-platform compatibility validated
• Component model integration thoroughly tested
• CI/CD pipeline provides comprehensive coverage
• Testing tools are developer-friendly and fast

Related Issues

• WASM32-WASIP2 Target Support (Add WASM32-WASIP2 target support for WebAssembly deployment #26)
• Storage Backend Abstraction (Create trait-based storage backend abstraction with WASM compatibility #29)
• WASI-MCP Component Interfaces (Implement WASI-MCP component model interfaces and reference implementation #30)
• WRT Integration Layer (Integrate WRT WebAssembly interpreter as MCP component host #32)

References

• Rust Testing Guide
• WASM Testing Best Practices
• Component Model Testing

[avrabe]

🧪 Testing Foundation Enhanced - Simplified Examples Enable Better Testing

PR #47 significantly improves our testing foundation through simplified development patterns!

🎯 Testing Benefits from Tool Discovery System:

1. Simplified Test Creation:

• 8-line MCP servers make test scenarios much easier to create
• Automatic schema generation provides reliable test validation
• Zero boilerplate reduces test maintenance overhead

2. Cross-Platform Testing Ready:

• Ultra-simple examples compile quickly for multiple targets
• Minimal dependencies reduce platform-specific issues
• Type-safe generation provides consistent behavior across platforms

3. WASM Testing Foundation:

• Simplified server structure perfect for WASM compilation testing
• Automatic tool registration reduces WASM-specific code paths
• Compile-time generation avoids runtime compatibility issues

🔧 Enhanced Testing Capabilities:

Native Target Testing:

// Perfect test subject - minimal and complete
#[mcp_server(name = "Test Server")]
#[derive(Default, Clone)]  
pub struct TestServer;

#[mcp_tools]
impl TestServer {
    pub fn test_function(&self, input: String) -> String {
        format!("Test: {}", input)
    }
}

WASM Target Testing:

• Same simple structure works for wasm32-wasip2 compilation tests
• Schema generation testing across platforms
• Integration testing with minimal overhead

🚀 Next Steps for Comprehensive Testing:

1. Cross-platform CI using ultra-simple examples
2. WASM compilation testing with simplified server structure
3. Schema generation validation across all supported types
4. Performance benchmarking with minimal test overhead

The enhanced developer experience makes comprehensive testing much more feasible and maintainable!

Ready to implement comprehensive cross-platform testing strategy.

[avrabe]

Testing Foundation Enhanced: Tool Discovery Enables Better Test Coverage 🧪

The recently activated tool discovery system (PR #47) provides significant improvements for cross-platform testing:

Enhanced Testing Capabilities

Automated Tool Testing

• Test suite can now automatically discover all tools in a server and validate their schemas
• Cross-platform testing can verify tool schema consistency between native and WASM builds
• Automated parameter validation testing using generated schemas

Schema Validation Testing

• Our test at mcp-macros/tests/tool_discovery_test.rs:85-131 demonstrates comprehensive schema testing
• Tests verify both required and optional parameter handling
• Type detection testing ensures consistent behavior across platforms

WASM-Specific Testing Benefits

• Tool discovery can validate WASM compatibility of tool signatures
• Schema generation helps identify platform-specific type issues
• Cross-compilation testing can verify tool interface consistency

Current Implementation

Already working:

• ✅ Enhanced Schema Tests: Comprehensive test validating required/optional parameters
• ✅ Type Detection Tests: Verifying String, integer, boolean schema generation
• ✅ Cross-Platform Foundation: Ultra-simple example works on all platforms

Testing Infrastructure Ready

With tool discovery active, we can now implement:

1. Automated Tool Coverage Testing: Discover and test all tools automatically
2. Cross-Platform Schema Validation: Ensure identical schemas across targets
3. WASM Compatibility Testing: Validate tool signatures for WASM deployment
4. Integration Test Automation: Schema-driven testing of tool behavior

The enhanced tool discovery system provides the introspection capabilities needed for comprehensive cross-platform testing. We now have the foundation to build the automated testing infrastructure described in this issue.