kyutai ORG

Moshi is a speech-text foundation model and full-duplex spoken dialogue framework. It uses Mimi, a state-of-the-art streaming neural audio codec.

An open AI model for real-time full-duplex voice conversation, both sides can speak at once like a phone call, with implementations for research, Apple Silicon devices, and production deployments.

A TTS that fits in your CPU (and pocket)

Kyutai's Speech-To-Text and Text-To-Speech models based on the Delayed Streams Modeling framework.

Hibiki is a model for streaming speech translation (also known as simultaneous translation). Unlike offline translation—where one waits for the end of the source utterance to start translating--- Hibiki adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk.

Make text LLMs listen and speak

No description.

A real-time and multilingual speech translation model

Kyutai with an "eye"

Code for the blog "Neural audio codecs: how to get audio into LLMs"

No description.

MoshiRAG is a compact full-duplex speech language model augmented with asynchronous knowledge retrieval to improve factuality without sacrificing real-time interactivity.

To bring back voice to those who lost it

python bindings for symphonia/opus - read various audio formats from python and write opus files

Official implementation and models for OVIE (One View Is Enough! Monocular Training for In-the-Wild Novel View Generation)

No description.

A small rust-based data loader

A vision-language model with an improved cross-attention mechanism for scalable streaming inference

No description.

No description.

Rust crate for some audio utilities

JAX bindings for the FlashAttention 3 kernels

JAX bindings for the flash-attention3 kernels

Proof of concept for running moshi/hibiki using webrtc

JAX bindings for the flash-attention2 kernels

Ogg-vorbis reader with fast random access

No description.

Framework for writing deep learning training loops. Lightweight, and retaining full freedom to design as you see fits. It handles checkpointing, logging, distributed, compatibility with Dora, and more!

Dora is an experiment management framework. It expresses grid searches as pure python files as part of your repo. It identifies experiments with a unique hash signature. Scale up to hundreds of experiments without losing your sanity.

Animations for the blog "Neural audio codecs: how to get audio into LLMs"