ai-tokenizer
TypeScript
★ 58
updated 8mo ago
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
No plain-English explanation yet — one is being written right now. Check back in a minute.