Implement comprehensive HTTP streaming support for large/long-running operations

[Copilot]

This PR implements comprehensive streaming capabilities for the Raindrop MCP server to efficiently handle large datasets and long-running operations as requested in the issue.

🌊 What's Implemented

Streaming Infrastructure

• Enhanced HTTP Transport: Leverages existing StreamableHTTPServerTransport with proper session management and SSE support
• STDIO Streaming: Custom streaming message format with chunked responses and progress notifications
• Smart Detection: Automatically enables streaming for datasets >50 items or operations marked as streaming-capable
• Memory Efficiency: Constant memory usage regardless of dataset size through chunked processing

New Streaming Tools

• streamSearchBookmarks: Search with chunked results for large bookmark collections
• streamHighlights: Stream highlights with pagination support (handles thousands of highlights)
• streamExportBookmarks: Export operations with real-time progress updates
• streamImportStatus: Monitor import progress with streaming status updates
• getStreamingCapabilities: Comprehensive streaming information and capabilities

New Streaming Resources

• highlights://stream/all: All highlights with chunked loading (25 items per chunk)
• search://stream/{query}: Search results with streaming support
• highlights://stream/collection/{id}: Collection highlights with streaming
• highlights://stream/raindrop/{id}: Raindrop highlights with streaming
• collections://stream/all: Collections with streaming awareness

📊 Technical Details

Chunked Transfer Implementation

// Automatic chunking for large responses
await this.streamingService.streamSearchResults(
  { search: 'javascript', chunkSize: 25 },
  (chunk) => {
    // Process each chunk of 25 bookmarks
    console.log(`Chunk ${chunk.page}: ${chunk.items.length} items`);
  },
  (progress) => {
    // Real-time progress updates
    console.log(`Progress: ${progress.percentage}%`);
  }
);

Transport Support

• HTTP: Full SSE streaming via existing StreamableHTTPServerTransport
• STDIO: Enhanced transport with streaming message format and chunked responses
• Progress Updates: Real-time notifications for long-running operations (exports/imports)
• Error Handling: Graceful fallback to standard responses if streaming fails

Performance Characteristics

• Chunk Size: Configurable (default: 25, max: 50 items per chunk)
• Memory Usage: Constant regardless of dataset size
• Network Efficiency: Automatic compression and optimal chunk timing
• Backward Compatibility: All existing tools continue to work unchanged

📚 Documentation & Testing

Comprehensive Documentation

• STREAMING.md: 294-line comprehensive guide with examples and best practices
• README.md: Updated with streaming features and usage examples
• Code Documentation: Detailed JSDoc comments throughout

Usage Examples

# Test streaming capabilities
npm run test:streaming

# Get streaming information
npm run streaming:capabilities

# Test streaming search via HTTP
curl -X POST http://localhost:3002/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"streamSearchBookmarks","arguments":{"search":"javascript","streaming":true,"chunkSize":25}},"id":1}'

🔧 When Streaming is Used

Automatically Enabled For:

• Search operations returning >50 bookmarks
• All highlights fetching (potentially thousands)
• Export/import operations (long-running)
• Large collection operations

Client Control:

• Clients can disable streaming with streaming: false parameter
• Configurable chunk sizes (1-50 items)
• Progress callback support for real-time updates

🎯 Benefits

1. Scalability: Handles datasets of any size efficiently
2. User Experience: Real-time progress for long operations
3. Memory Efficiency: Constant memory usage vs. dataset size
4. Network Optimization: Chunked transfer prevents timeouts
5. Backward Compatibility: Existing clients continue working unchanged

The implementation adds 1,600+ lines of code across 11 files, providing state-of-the-art streaming capabilities while maintaining full compatibility with existing MCP clients.

Fixes #7.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.