compute-optimal-tokenization
Python
★ 4
updated 25d ago
The repository contains raw data results and code for scaling laws fitting and visualization used in "Compute Optimal Tokenization" paper.
No plain-English explanation yet — one is being written right now. Check back in a minute.