VortexForge 🌀

VortexForge is a high-performance, distributed task orchestrator built in Rust. It is designed for low-latency task scheduling across heterogeneous clusters, utilizing a priority-based state machine and a write-ahead log (WAL) for fault tolerance.

🚀 Key Features

• Priority-Aware Scheduling: Uses a min-max heap to ensure high-priority tasks are dispatched first.
• Asynchronous Runtime: Powered by Tokio for handling thousands of concurrent connections and tasks.
• Persistent State: Implements a Write-Ahead Log (WAL) to ensure zero data loss during node failures.
• Thread-Safe Concurrency: Leverages DashMap and atomic primitives for lock-free read access across worker threads.
• Zero-Copy Protocol: Prepared for Protobuf integration for efficient inter-node communication.

🏗️ Architecture

VortexForge operates on a Leader-Worker topology:

1. The Scheduler: Maintains the global task queue and manages state transitions.
2. Workers: Pull tasks via a long-polling mechanism and execute them in isolated green threads.
3. The WAL: Every state change is serialized and flushed to disk before the task is acknowledged.

🛠️ Getting Started

Prerequisites

• Rust (1.70+)
• Cargo (included with Rust)

Installation

git clone [https://github.com/vortex-labs/vortexforge.git](https://github.com/vortex-labs/vortexforge.git)
cd vortexforge
cargo build --release

Running the Node

# Start the orchestrator and local workers
cargo run

📂 Project Structure

• src/core/: The "brain" — contains the scheduler, state machine, and worker logic.
• src/network/: Networking primitives and custom binary protocol handling.
• src/storage/: Persistence layer (WAL implementation).
• proto/: Protocol Buffer definitions for cross-language compatibility.

📜 License

Distributed under the MIT License. See LICENSE for more information.