sentence-transformers
State-of-the-Art Embeddings, Retrieval, and Reranking
Sentence Transformers is a Python library that converts text into numbers capturing meaning, so you can find semantically similar documents, build semantic search, or detect duplicates even with different wording.
Sentence Transformers is a Python framework for converting text into numerical representations called embeddings — fixed-size lists of numbers that capture the meaning of the text. Two pieces of text with similar meanings end up with similar numbers, which makes it possible to measure how semantically related they are, even if they use completely different words.
The library provides three main types of models. Sentence Transformer models (also called embedding models) convert text into dense embeddings useful for tasks like semantic search, finding duplicate content, and grouping similar documents. Cross-Encoder models (also called reranker models) take two pieces of text together and score how well they match — useful for re-ranking a list of search results to put the most relevant ones first. Sparse Encoder models produce a different kind of representation where most values are zero, which can be more efficient for certain retrieval scenarios.
The framework includes over 15,000 pre-trained models that can be downloaded and used immediately, as well as tools for fine-tuning your own models on custom data. It is installed via pip and works with Python 3.10 and above. The full documentation is at sbert.net.
Where it fits
- Build a semantic search engine that finds relevant documents based on meaning, not just keyword matches.
- Group similar customer support tickets together to find patterns without reading each one manually.
- Re-rank a list of search results to put the most relevant ones at the top using a Cross-Encoder model.
- Detect near-duplicate content in a large document collection by comparing their embeddings.