Root Mean Square Layer Normalization

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Efficient kernel for RMS normalization with fused operations, includes both forward and backward passes, compatibility with PyTorch.

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

A modern minimal LLM implementation made to be easily modified by non-professionals and trained on consumer hardware.

Simple character level Transformer

Generative models nano version for fun. No STOA here, nano first.

A from-scratch PyTorch LLM implementing Sparse Mixture-of-Experts (MoE) with Top-2 gating. Integrates modern Llama-3 components (RMSNorm, SwiGLU, RoPE, GQA) and a custom-coded Byte-Level BPE tokenizer. Pre-trained on a curated corpus of existential & dark philosophical literature.

Optimized Fused RMSNorm implementation with CUDA. Features vectorized memory access (float4), warp-level reductions, and efficient backward pass for LLM training

🚀 Build your own LLM easily with OpenLabLM, a lightweight, hackable codebase tailored for hobbyists using a single consumer GPU.

《Root Mean Square Layer Normalization》论文阅读笔记