openai-tiktoken

Browse 1 Armenian open source projects tagged with openai-tiktoken.

armenian-token-cost
HakobSargsyan/armenian-token-cost

A Python tool that measures and compares the tokenization cost of Armenian text versus other languages (English, Russ...

Python Stars: 0