MInference
★ 1
updated 1y ago
⑂ fork
[NeurIPS'24 Spotlight, ICLR'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
No plain-English explanation yet — one is being written right now. Check back in a minute.