Manic Miners Level Indexer

A comprehensive indexing system for Manic Miners community levels, automatically collecting and cataloging custom content from multiple sources including Archive.org, Discord channels, and GitHub repositories.

🎮 Overview

The Manic Miners Level Indexer is a TypeScript-based tool that creates a searchable, organized database of community-created levels for the Manic Miners game. It provides automated discovery, validation, and cataloging of custom levels with rich metadata, making it easy for players to find and enjoy community content.

🚀 Key Features

• Multi-Source Indexing: Collects levels from Internet Archive, Discord channels (Community and Archive), and the Hognose GitHub repository
• Automated Metadata Extraction: Captures title, author, description, tags, and game requirements
• Format Version Detection: Automatically identifies level format versions (below-v1, v1, v2)
• Quality Validation: Validates level files and metadata completeness
• Organized Storage: Maintains a structured directory with individual level folders
• Rich Analytics: Generates statistics on authors, tags, file sizes, and data quality
• Progress Tracking: Real-time progress updates during indexing operations
• Discord Authentication: Supports both automated and manual Discord authentication
• Export Capabilities: Export catalogs to JSON or CSV formats
• Skip Existing: Smart state management to avoid re-processing already indexed content
• Retry Logic: Automatic retry with exponential backoff for network failures
• Checksum Verification: Validate downloaded file integrity (Hognose and Internet Archive)
• Secure File Handling: All downloaded filenames are sanitized to prevent path traversal attacks
• Original Filename Preservation: Images and files maintain their original names while ensuring filesystem safety

📊 Latest Indexing Results

The indexer has successfully cataloged 1,241 community levels from various sources:

Source	Levels	Percentage
Discord Archive	460	37%
Discord Community	334	27%
Hognose	252	20%
Internet Archive	195	16%

Top Contributors:

• Hognose: 252 levels
• fn4.1150: 126 levels
• crystal2780: 123 levels
• batman1138: 96 levels
• exkajer: 43 levels

Last updated: January 2025

🚧 Upcoming Features

We're actively developing exciting new features to create the perfect index for the Manic Miners community:

🎨 Map Renderer (In Design)

• Visual Thumbnails: Generate preview images for every level
• Blueprint Generation: Create detailed map layouts showing tiles and objects
• Multiple Formats: Support for PNG, JPEG, WebP, and SVG outputs
• Batch Processing: Efficient rendering of thousands of levels
• Read the design document

📊 Level Profiler (In Design)

• Difficulty Analysis: Objective complexity scoring (1-10 scale)
• Resource Counting: Complete inventory of tiles, objects, and requirements
• Path Analysis: Route complexity and navigation challenges
• Gameplay Predictions: Estimated completion time and strategy hints
• Integration: Uses user's specialized parsing package
• Read the design document

🌐 Future Enhancements

• Web Interface: Browse and search the catalog online
• GraphQL API: Advanced querying capabilities
• Real-time Indexing: Automatic updates as new levels are posted
• Community Features: Ratings, reviews, and recommendations

📦 Quick Start

Prerequisites

• Node.js 18.0.0 or higher
• npm or yarn package manager
• (Optional) Discord user token for Discord indexing

Installation

# Clone the repository
git clone https://github.com/Wal33D/manic-miners-level-indexer.git
cd manic-miners-level-indexer

# Install dependencies
npm install

# Create a .env file (optional, for Discord indexing)
cp .env.example .env
# Edit .env and add your Discord token: DISCORD_USER_TOKEN=your_token_here

# Build the project
npm run build

Basic Usage

Index all sources with default configuration:

npm run index

Index individual sources:

# Index Internet Archive levels
npm run index:internet-archive

# Index Discord Community levels (requires authentication)
npm run index:discord:community

# Index Discord Archive levels (requires authentication)
npm run index:discord:archive

# Index Hognose repository levels
npm run index:hognose

Quick Test

Run a limited test to verify everything works:

npm run test:quick

This will index a small number of levels from each source for testing.

⚙️ Configuration

Create a config.json file to customize indexing behavior:

{
  "outputDir": "./output",
  "sources": {
    "internet_archive": {
      "enabled": true,
      "baseUrl": "https://archive.org/advancedsearch.php",
      "searchQueries": ["manic miners level"],
      "maxConcurrentDownloads": 5
    },
    "discord_community": {
      "enabled": true,
      "channels": [
        "1139908458968252457"
      ],
      "retryAttempts": 3,
      "downloadTimeout": 60000,
      "skipExisting": true
    },
    "discord_archive": {
      "enabled": true,
      "channels": [
        "683985075704299520"
      ],
      "retryAttempts": 3,
      "downloadTimeout": 60000,
      "skipExisting": true
    },
    "hognose": {
      "enabled": true,
      "githubRepo": "charredUtensil/groundhog",
      "retryAttempts": 3,
      "downloadTimeout": 60000,
      "verifyChecksums": true,
      "skipExisting": true
    }
  }
}

📜 Available Scripts

🎯 Indexing Scripts

Command	Description
npm run index	Index all enabled sources
npm run index:internet-archive	Index Internet Archive levels
npm run index:discord:community	Index Discord Community levels
npm run index:discord:archive	Index Discord Archive levels
npm run index:hognose	Index Hognose levels

🧪 Testing Scripts

Command	Description
npm test	Run unit tests
npm run test:quick	Quick integration test (limited data)
npm run test:all	Full integration test
npm run test:analysis	Test with detailed analysis report

🔧 Utility Scripts

Command	Description
npm run validate:catalog	Validate existing catalog
npm run rebuild:catalog	Rebuild catalog from existing levels
npm run clean:test	Clean test output directories

👨‍💻 Development Scripts

Command	Description
npm run dev	Run in development mode
npm run build	Build TypeScript to JavaScript
npm run lint	Run ESLint
npm run format	Format code with Prettier
npm run type-check	Check TypeScript types

📁 Output Structure

output/
├── catalog_index.json          # Master catalog of all levels
├── master_index.json          # Enhanced index with statistics
├── levels-internet-archive/   # Internet Archive levels
│   ├── catalog_index.json    # Source-specific catalog
│   └── [uuid]/               # Individual level directory
│       ├── catalog.json      # Level metadata
│       └── level.dat         # Game data file
├── levels-discord-community/ # Discord Community levels
├── levels-discord-archive/   # Discord Archive levels
└── levels-hognose/          # Hognose repository levels

💻 Programmatic Usage

import { MasterIndexer, IndexerConfig } from 'manic-miners-level-indexer';

const config: IndexerConfig = {
  outputDir: './my-levels',
  sources: {
    internet_archive: { enabled: true, retryAttempts: 3, downloadTimeout: 60000, skipExisting: true },
    discord_community: { enabled: true, channels: ['1139908458968252457'], retryAttempts: 3, downloadTimeout: 60000, skipExisting: true },
    discord_archive: { enabled: true, channels: ['683985075704299520'], retryAttempts: 3, downloadTimeout: 60000, skipExisting: true },
    hognose: { enabled: true, retryAttempts: 3, downloadTimeout: 60000, verifyChecksums: true, skipExisting: true }
  }
};

const indexer = new MasterIndexer(config);

// Index all sources
await indexer.indexAll();

// Get catalog statistics
const stats = await indexer.getCatalogStats();
console.log(`Total levels indexed: ${stats.totalLevels}`);

// Export catalog
const exportPath = await indexer.exportCatalog('json');
console.log(`Catalog exported to: ${exportPath}`);

📚 Documentation

Core Documentation

• Getting Started Guide - Detailed setup and first run
• Configuration Reference - All configuration options
• API Reference - Complete API documentation
• Troubleshooting - Common issues and solutions

Architecture & Technical

• Architecture Overview - System design and components
• Indexers Guide - Detailed indexer documentation
• Output Structure - File formats and schemas
• Discord Authentication - Discord setup guide

Development & Operations

• Development Guide - Contributing and development
• Branch Protection - Git workflow and PR process
• Deployment Guide - Production deployment instructions
• Improvement Plan - Incremental improvement roadmap

Roadmap

• Future Features & Roadmap - Planned features including Map Renderer and Level Profiler

🤝 Contributing

We welcome contributions! Please see our Development Guide for details on:

• Setting up a development environment
• Code style guidelines
• Testing requirements
• Submitting pull requests

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• The Manic Miners community for creating amazing custom content
• Archive.org for preserving gaming history
• The Hognose project for procedural level generation
• All level authors who share their creative work

💬 Support

For issues, questions, or suggestions:

• Open an issue on GitHub
• Check the Troubleshooting Guide
• Review existing discussions