kvcache.ai ORG

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

Research toolkit for running very large AI language models on modest hardware by offloading expert layers to CPU, letting you run models like DeepSeek-V3 without a fleet of expensive high-end GPUs.

No description.

A high-throughput and memory-efficient inference and serving engine for LLMs

No description.

SGLang is a fast serving framework for large language models and vision language models.

FlashInfer: Kernel Library for LLM Serving

DeepEP: an efficient expert-parallel communication library that supports fault tolerance

SGLang is a fast serving framework for large language models and vision language models.

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

A unified library of SOTA model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

GPU cluster manager for optimized AI model deployment

SGLang is a fast serving framework for large language models and vision language models.