tiktoken

Tiktoken is a fast tokenizer for use with OpenAI's language models. Tokenization is the process of converting text into numbers before feeding it to an AI model — language models do not process words or characters directly, but instead work with chunks called tokens. A token typically corresponds to about four characters of English text, though the exact mapping depends on the encoding.

Tiktoken implements Byte Pair Encoding (BPE), an algorithm that learns to split text into common subword chunks based on frequency in training data. This approach is both reversible (tokens can be decoded back to the original text) and lossless, and it handles arbitrary text including content the tokenizer has never seen before. Because common subwords like "ing" appear as single tokens, models can generalize better about language patterns.

The library is between three and six times faster than comparable tokenizers, making it practical for applications that need to count tokens or process large amounts of text quickly. It can be installed via pip and used in Python. Tiktoken includes functions to get the tokenizer for a specific OpenAI model, to encode and decode text, and to extend the tokenizer with custom special tokens or entirely new encodings via a plugin system. An educational submodule is also included for learning how BPE works step by step.