rust-tokenizers
★ 0
updated 6y ago
⑂ fork
Rust-tokenizer is a drop-in replacement for the tokenization methods from the Transformers library
No plain-English explanation yet — one is being written right now. Check back in a minute.