gitmyhub

TLLM_QMM

C++ ★ 16 updated 1y ago

TLLM_QMM strips the implementation of quantized kernels of Nvidia's TensorRT-LLM, removing NVInfer dependency and exposes ease of use Pytorch module. We modified the dequantation and weight preprocessing to align with popular quantization alogirthms such as AWQ and GPTQ, and combine them with new FP8 quantization.

No plain-English explanation yet — one is being written right now. Check back in a minute.