Foundation Model Inference ORG

Running large language models on a single GPU for throughput-oriented scenarios.

A Python tool that runs massive AI language models on a single consumer GPU by spreading the model across GPU memory, RAM, and disk, designed for overnight batch jobs like classifying thousands of documents, not real-time chat.

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

No description.