context-1-data-gen
The data generation pipeline behind Chroma's Context-1 search agent, producing synthetic multi-step search training examples across four domains.
This repository contains the data generation pipeline used to train Chroma's Context-1 model, a search agent. The pipeline creates synthetic training examples by simulating multi-step search tasks — scenarios where finding an answer requires chaining together several searches rather than a single lookup. It covers four domains: open web searches, SEC financial filings, patent prior-art research, and email archives.
Each domain follows the same three-stage process: explore (find relevant starting material), verify (confirm the information is correct), and extend (expand to related tasks). The pipeline uses the Anthropic Claude API as its core AI backbone, along with OpenAI embeddings and ChromaDB for indexing. Running it requires API keys for several services. The resulting training data was used to build the Context-1 model weights, which are available separately on Hugging Face.
Where it fits
- Generate synthetic multi-step search training data across web, financial filings, patents, or email archives.
- Study or reproduce the explore, verify, extend pipeline used to train the Context-1 model.
- Adapt the pipeline to build training data for a custom search agent in a different domain.