DeepMoE
Python
★ 8
updated 1mo ago
Deep MoE is an all-new mixture-of-experts architecture a transformer layer that does its thinking in a compressed thought space rather than on the wide residual stream.
No plain-English explanation yet — one is being written right now. Check back in a minute.