openlily

OpenLily is a voice assistant you run on your own computer, similar in concept to Alexa or Google Home but fully under your control. You speak into your microphone, the system processes your words, sends them to a language model, and speaks the answer back through your speakers. There is no cloud service in the loop unless you choose one, and no subscription or account is required.

The project is built to be configurable at every step. The speech-to-text, language model, and text-to-speech pieces can each be swapped independently. You can use OpenAI's API for all three, use a local model running on your own hardware through Ollama, or mix providers. There is also a fully offline option using MLX Whisper for speech recognition, Ollama for the language model, and Kokoro for voice output, which currently requires Apple Silicon hardware.

A wake word feature is built in using an on-device model called openWakeWord, so the assistant sits idle and only starts listening when it hears a chosen phrase. No internet connection is needed for wake word detection. Turn-taking (deciding when you have finished speaking) is handled by two on-device models: Silero VAD and Smart Turn v3.

Tools are available as optional add-ons: web search, browser automation using Playwright, and email. Each tool is opt-in, so you enable only what you need.

Setup requires running from the server directory, installing dependencies with uv, and installing PortAudio on your operating system for microphone access. A configuration file handles API key selection and brain choice. The first run is slower because on-device models download once.

OpenLily is aimed at developers and technically inclined users who want a private, customizable voice assistant that they can run on a desktop, a Raspberry Pi, or a small dedicated machine.