tokenization

Browse 2 Armenian open source projects tagged with tokenization.

Low-resource-Armenian-NLP
levongevorgian/Low-resource-Armenian-NLP

A research project investigating and improving tokenization efficiency for the low-resource Armenian language. It inv...

Stars: 0
Armenian-tokenizer
nairabarseghyan/Armenian-tokenizer

A student project implementing multiple tokenization methods (BPE, WordPiece, SentencePiece, tiktoken) for Armenian l...

Jupyter Notebook Stars: 0