armenian-gpt

Summary

ArmGPT is a student-friendly implementation of a GPT language model specifically trained on Armenian text. It's built from scratch with a transformer architecture featuring modern components like RMSNorm, SwiGLU, and RoPE. The project includes a full pipeline: training a BPE tokenizer on a 63GB Armenian corpus, pre-training the model, and fine-tuning it for Q&A chat. It provides a live demo, detailed setup instructions for various platforms (including Windows/WSL2), and is inspired by nanoGPT.