ai-tokenizer
★ 0
updated 8mo ago
⑂ fork
A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
No plain-English explanation yet — one is being written right now. Check back in a minute.