Learning-Resources

A compilation of resources for keeping up with the latest trends in NLP.

Note: This resource list is a work in progress. More papers and topics will be added regularly. Contributions and suggestions are welcome!

Some Fundamental Transformers

1. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
2. GPT1
3. GPT2
4. T5
5. XLNet
6. RoBERTa
7. ALBERT
8. LongFormer

Papers for understanding fundamentals

1. Attention is all you need
2. Memory Is All You Need
3. Language Models are Few-Shot Learners

Reinforcement Learning for LLMs

Basics of RL - OpenAI

Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism

InstructGPT

DPO:

1. DPO paper
2. Blog - Math behind DPO

PPO:

1. Proximal Policy Optimization Algorithms
2. PPO Docs OpenAI
3. Understanding PPO from First Principles Blog

GRPO:

1. DeepSeekMath
2. Blog - GRPO Explained
3. DeepSeek-R1

Mechanistic Interpretability

1. Basic Mech Interp Essay
2. Toy Neural Nets with low dimensional inputs
3. Mechanistic Interpretability for AI Safety Review
4. A Mathematical Framework for Transformer Circuits
5. Circuit Tracing: Revealing Computational Graphs in Language Models

Scaling Laws

1. Scaling Laws for Neural Language Models
2. Scaling Laws for Autoregressive Generative Modeling
3. Sacling Laws of Synthetic Data for Lnguage Models

MLSys

1. Mixed Precision Training
2. Matrix multiplication - Nvidia Blog
3. Understanding GPU Performance - Nvidia Blog