turboderp

An OAI compatible exllamav2 API that's both lightweight and fast

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

No description.

A Gradio web UI for Large Language Models. Supports transformers, GPTQ, llama.cpp (GGUF), Llama models.

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

Formatron empowers everyone to control the format of language models' output with minimal overhead.

Enforce the output format (JSON Schema, Regex etc) of a language model