uax29

Go ★ 0 updated 4y ago ⑂ fork ▣ archived

A tokenizer based on Unicode text segmentation (UAX 29), for Go

No plain-English explanation yet — one is being written right now. Check back in a minute.