Trial AskAI markdown indexing

[thibaudcolas]

Summary

Trial Algolia's AskAI feature with dedicated Markdown indexing to improve AI-powered search answers for Draftail documentation.

Motivation

AskAI provides AI-generated answers based on documentation content. Using a dedicated Markdown index (separate from the keyword search index) can significantly improve answer quality by:

• Providing cleanly structured, text-only content to the LLM
• Removing navigation and layout elements that dilute content
• Optimizing chunk sizes for better context
• Enabling better filtering with facets (language, version, etc.)

Implementation

Set up a separate Markdown index using Algolia's crawler with the helpers.markdown() and helpers.splitTextIntoRecords() utilities. This would complement our existing DocSearch index.

Key Steps

1. Update crawler configuration to create a dedicated Markdown index
2. Configure the index with appropriate settings for AI consumption
3. Integrate the Markdown index with DocSearch's AskAI feature
4. Test answer quality and adjust maxRecordBytes as needed

Resources

• Algolia Markdown indexing documentation
• Current Algolia configuration: algolia/docsearch-configs:draftail.json

Benefits

• Better AI-generated answers for documentation queries
• Separation of search and AI answer concerns
• Ability to optimize content structure specifically for LLMs
• Potential cost optimization through better chunking

Acceptance Criteria

• Markdown index created and configured
• Crawler updated to populate the Markdown index
• AskAI integrated with both keyword and Markdown indexes
• Answer quality tested and validated
• Documentation updated with configuration details

[github-actions]

❌ Agentic Agentic Triage failed and wasn't able to produce a result.