Train a 26M ChatBot from zero.
2 hours. ¥3. One 3090.
That's it.
26M
Parameters
2h
Training
¥3
Cost
1/7000
vs GPT-3
✨ What You Get
💰 Ultra-Cheap
One 3090. Zero to ChatBot in 2 hours, costing just ¥3.
📦 Full Stack
Complete pipeline: Tokenizer → Pretrain → SFT → LoRA → PPO/GRPO/SPO
🔬 Latest RL (2025)
PPO, GRPO, SPO + YaRN length extrapolation. Native PyTorch implementation.
📖 Learn by Reading
Pure PyTorch. No black boxes. Understand every line of code.
🔌 Plug & Play
Compatible with vLLM, ollama, llama.cpp, transformers.
⚡ OpenAI API
Drop-in replacement for FastGPT, Open-WebUI, Dify.
📦 Models
| Model | Parameters | Hidden Dim | Layers | Memory |
|---|---|---|---|---|
| MiniMind2-Small | 26M | 512 | 8 | ~0.5 GB |
| MiniMind2 | 104M | 768 | 16 | ~1.0 GB |
| MiniMind2-MoE | 145M | 640 | 8 | ~1.0 GB |
📰 What's New
🔥 2025-10-24 (Latest)
- 🔥 RLAIF algorithms: PPO, GRPO, SPO (native PyTorch)
- Checkpoint resume training: auto-save & cross-GPU recovery
- RLAIF dataset: rlaif-mini.jsonl (10K samples); Simplified DPO dataset with Chinese data
- YaRN algorithm for RoPE length extrapolation
- Adaptive Thinking in reasoning models
- Tool Calling & Reasoning tags support
- Complete RLAIF chapter with training curves
- SwanLab integration (WandB alternative for China)
- Code standardization & bug fixes
⚙️ 2025-04-26 (Major Refactor)
- Model parameter renaming (align with Transformers)
- Generate method refactor (GenerationMixin)
- ✅ llama.cpp, vllm, ollama support
- Vocab update: <s></s> → <|im_start|><|im_end|>
- Code structure standardization
🎉 2025-02-09 (MiniMind2 Release)
- Complete codebase rewrite
- MiniMind2 series: 26M, 104M, 145M models
- JSONL data format (no preprocessing needed)
- LoRA from scratch (no peft dependency)
- DPO native PyTorch implementation
- White-box model distillation
- DeepSeek-R1 distillation models
- HQ pretraining data (2h on single 3090)
🎬 2024-10-05 (Vision Multimodal)
- Vision multimodal support
- MiniMind-V project launch
🔧 2024-09-27 (Data Preprocessing)
- Pretrain dataset preprocessing update
- Text integrity preservation
- Code cleanup
📝 2024-09-17 (MoE & Tokenizer)
- MiniMind-V1-MoE model release
- Standardized minimind_tokenizer
- Removed mistral_tokenizer variants
🚀 2024-09-01 (V1 Release)
- MiniMind-V1 (108M) release
- 3 epochs pretrain + 10 epochs SFT
- ModelScope online demo
⭐ 2024-08-27 (First Release)
- 🚀 First open-source release
- MiniMind project launched
🎮 Inside MiniMind
🎯 Try It Online
📦 Get the Code
💡 Why MiniMind?
- Train from zero, not just finetune
- Pure PyTorch—no magic, no black boxes
- Understand by building, not by reading docs
- Works on your laptop—no cloud GPUs needed
- 2025 RLAIF algorithms: PPO, GRPO, SPO
- OpenAI API compatible—plug into any UI
- Vision support via MiniMind-V
- Code you can actually read and modify
💭 "Building a Lego plane beats flying first class."