HakobSargsyan/armenian-token-cost
A Python tool that measures and compares the tokenization cost of Armenian text versus other languages (English, Russian, Georgian) using OpenAI's tiktoken library. It provides reproducible analysis showing that Armenian script was severely disadvantaged under older tokenizers (cl100k_base) and remains at a cost disadvantage under newer ones (o200k_base), impacting API cost, context window usage, and model performance.
A student project implementing multiple tokenization methods (BPE, WordPiece, SentencePiece, tiktoken) for Armenian l...
A demonstration repository for a pre-trained Armenian text embedding model, showcasing applications in text classific...
AzgIntel is an AI-powered NLP pipeline for Armenian text classification using a BERT-based multilingual model. It pro...
ArmGPT is a student-friendly implementation of a GPT language model specifically trained on Armenian text. It's built...
A comprehensive evaluation harness for an Armenian language learning pipeline that converts TikTok videos into Russia...
hy-text is an open-source reference tool for evaluating and ensuring text quality in modern Eastern Armenian for AI a...