gitmyhub

compare-tokenizers

TypeScript ★ 31 updated 2y ago

A test suite comparing Node.js BPE tokenizers for use with AI models.

No plain-English explanation yet — one is being written right now. Check back in a minute.