tokenization

Browse 2 Armenian open source projects tagged with tokenization.

Low-resource-Armenian-NLP
levongevorgian/Low-resource-Armenian-NLP

A research project focused on improving NLP for the low-resource Armenian language by analyzing tokenizer inefficienc...

Stars: 0
Armenian-tokenizer
nairabarseghyan/Armenian-tokenizer

A student project implementing multiple tokenization methods (BPE, WordPiece, SentencePiece, tiktoken) for Armenian l...

Jupyter Notebook Stars: 0