📚Documentation |
🛠️ Installation |
🌎 Home Page |
🐿🐴🐁🐘🐆 Model Zoo |
🚨 News |
🪲 Reporting Issues

🫶 Getting Assistance |
∞ DeepLabCut Online Course |
📝 Publications |
👩🏾‍💻👨‍💻 DeepLabCut AI Residency

!Version


Welcome! 👋

DeepLabCut™️ is a toolbox for state-of-the-art markerless pose estimation of animals performing various behaviors. As long as you can see (label) what you want to track, you can use this toolbox, as it is animal and object agnostic. Read a short development and application summary below.

Installation

Please click the link above for all the information you need to get started! Please note that currently we support only Python 3.10+ (see conda files for guidance).

Quick start

Developers Stable Release: very quick start (Python 3.10+ required) to install
DeepLabCut with the PyTorch engine

• [1] Install PyTorch (install and then select the desired

CUDA version if you want to use a GPU): pip install torch torchvision. Or as an example for GPU support (please check pytorch docs to get the perfect version for your CUDA):

bash
conda install pytorch cudatoolkit=11.3 -c pytorch

• [2] Then, install DeepLabCut (with all functions + the GUI):

bash
pip install --pre  "deeplabcut[gui]"

or pip install --pre "deeplabcut" (headless version with PyTorch)!

To use the TensorFlow (TF) engine: you'll need to run pip install "deeplabcut[gui,tf]" or pip install "deeplabcut[tf]" (headless version with TF). Alternatively, we also offer more targeted optional TensorFlow installs for specific CUDA setups, e.g. deeplabcut[tf-cu11] or deeplabcut[tf-cu12]. Please refer to our installation instructions for more detailed information on Python version, CUDA compatibility, etc.
We aim to deprecate the tensorflow backend in version 3.2 (release date TBD).

<!-- Link below just returns to the top of the README -->

Documentation: The DeepLabCut Process

Our docs walk you through using DeepLabCut, and key API points. For an overview of the toolbox and workflow for project management, see our step-by-step at Nature Protocols paper.

<!-- Course is outdated as of first 3.0rc release -->
<!-- For a deeper understanding and more resources for you to get started with Python and DeepLabCut, please check out our free online course! https://deeplabcut.github.io/DeepLabCut/docs/course.html -->

[Code demo](examples/README.md)

🐭 Pose tracking of single animals demo 

See more demos here.
We provide data and several Jupyter Notebooks, walking you through a demo dataset to test your installation, and another to run DeepLabCut from the start on your own data.
We also show how to use the code in Docker, and on Google Colab.

Why use DeepLabCut?

DeepLabCut continues to be actively maintained and we strive to provide a user-friendly GUI and API for computer vision researchers and life scientists alike. This means we integrate state-of-the-art models and frameworks, while providing our "best-guess" defaults for life scientists.
We highly encourage you to read our papers to get a better understanding of what to use and how to modify the models for your setting.

Performance 🔥

In general, we provide all the tooling for you to train and use custom models with various high-performance backbones.

Pretrained Models

We also provide two foundation pretrained animal models: SuperAnimal-Quadruped & SuperAnimal-TopViewMouse.
To gauge their *out-of-distribution* performance, we provide the following tables.

These models are trained on the SuperAnimal-Quadruped dataset with *AP-10K* held out for out-of-domain testing and the SuperAnimal-TopViewMouse dataset with *DLC-openfield* held out for out-of-distribution testing (see Ye et al. 2024).
We provide models that include AP-10K in the API (and GUI).
Note, there are many different models to select from in DeepLabCut 3.0. We strongly recommend you check this Guide for more details.
This table, and those below, give you a sense of performance in real-world complex in-the-wild and lab mouse data, respectively.
This link provides the model weights to reproduce the numbers; but please note, our full models are in our DLClibrary and released in the API.

DLC 3.0 Pose Estimation (Top Down Models)

| Model Name | Type | mAP SA-Q on AP-10K | mAP SA-TVM on DLC-OpenField |
|------------------------------|------------|---------------------|-----------------------------|
| top_down_resnet_50 | Top-Down | 54.9 | 93.5 |
| top_down_resnet_101 | Top-Down | 55.9 | 94.1 |
| top_down_hrnet_w32 | Top-Down | 52.5 | 92.4 |
| top_down_hrnet_w48 | Top-Down | 55.3 | 93.8 |
| rtmpose_s | Top-Down | 52.9 | 92.9 |
| rtmpose_m | Top-Down | 55.4 | 94.8 |
| rtmpose_x | Top-Down | 57.6 | 94.5 |

The History

Development and Applications

In 2018, we demonstrated the capabilities for trail tracking, reaching in mice and various Drosophila behaviors during egg-laying (see Mathis et al. for details). There is, however, nothing specific that makes the toolbox only applicable to these tasks and/or species.
The toolbox has already been successfully applied (by us and others) to rats, humans, various fish species, bacteria, leeches, various robots, cheetahs, mouse whiskers and race horses.
DeepLabCut utilized the feature detectors (ResNets + readout layers) of one of the state-of-the-art algorithms for human pose estimation by Insafutdinov et al., called DeeperCut, which inspired the name for our toolbox (see references below). Since this time, the package has changed substantially.
The code has been re-tooled and re-factored since 2.1+: We have added faster and higher performance variants with MobileNetV2s, EfficientNets, and our own DLCRNet backbones (see Pretraining boosts out-of-domain robustness for pose estimation and Lauer et al 2022). Additionally, we have improved the inference speed and provided both additional and novel augmentation methods, added real-time, and multi-animal support.
In v3.0+ we have updated the backend to support PyTorch. This brings not only an easier installation process for users, but performance gains, developer flexibility, and a lot of new tools! Importantly, the high-level API stays the same, so it will be a seamless transition for users 💜!
We currently provide state-of-the-art performance for animal pose estimation and the labs (M. Mathis Lab and A. Mathis Group) have both top journal and computer vision conference papers.

Left: Due to transfer learning it requires little training data for multiple, challenging behaviors (see Mathis et al. 2018 for details). Mid Left: The feature detectors are robust to video compression (see Mathis/Warren for details). Mid Right: It allows 3D pose estimation with a single network and camera (see Mathis/Warren). Right: It allows 3D pose estimation with a single network trained on data from multiple cameras together with standard triangulation methods (see Nath* and Mathis* et al. 2019).

Ecosystem

DeepLabCut is part of a larger open-source eco-system, providing behavioral tracking for neuroscience, ecology, medical, and technical applications.
Moreover, many new tools are being actively developed. See DLC-Utils for some helper code.

Code contributors

DeepLabCut was originally developed by Alexander Mathis & Mackenzie Mathis, and was extended in 2.0 with the core dev team consisting of Tanmay Nath (2.0-2.1), Jessy Lauer (2.1-2.4), and Niels Poulsen (2.3-3.0).
DeepLabCut is an open-source tool and has benefited from suggestions and edits by many individuals including early contributors: Mert Yuksekgonul, Tom Biasi, Richard Warren, Ronny Eichler, Hao Wu, Federico Claudi, Gary Kane and Jonny Saunders as well as the 100+ contributors.
Please see AUTHORS for more details!

🤩 This is an actively developed package and we welcome community development and involvement:


Get Assistance & be part of the DLC Community✨

| 🚉 Platform | 🎯 Goal | ⏱️ Estimated Response Time | 📢 Support Squad |
|------------------------------------------------------------|--------------------

…

Collapse