DefTruth

🛠 A lite C++ toolkit: contains 100+ Awesome AI models, support MNN, NCNN, TNN, ONNXRuntime and TensorRT. 🎉🎉

⚡️An Easy-to-use and Fast Deep Learning Model Deployment Toolkit

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

No description.

📚200+ Tensor/CUDA Cores Kernels, ⚡️flash-attn-mma, ⚡️hgemm with WMMA, MMA and CuTe (98%~100% TFLOPS of cuBLAS/FA2 🎉🎉).

📖A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, FlashAttention, PagedAttention, MLA, Parallelism, etc. 🎉🎉

Demos for how to use the shared libs of Lite.AI.ToolKit🚀🚀🌟. (https://github.com/DefTruth/lite.ai.toolkit)

自己的学习历程，重点包括各种好玩的图像处理算法、运动捕捉、机器学习

Flash Attention in ~100 lines of CUDA (forward pass only)

CMake Cookbook recipes.

⚡Super fast and lightweight anchor-free object detection model. 🔥Only 980 KB(int8) / 1.8MB (fp16) and run 97FPS on cellphone🔥

You Only Look Once for Panopitic Driving Perception.（https://arxiv.org/abs/2108.11250）

No description.

⚡️Write HGEMM from scratch using Tensor Cores with WMMA, MMA and CuTe API, Achieve Peak⚡️ Performance.

paper and its code for AI System

Awesome multilingual OCR toolkits based on PaddlePaddle (practical ultra lightweight OCR system, support 80+ languages recognition, provide data annotation and synthesis tools, support training and deployment among server, mobile, embedded and IoT devices)

Development repository for the Triton language and compiler

This repository includes the official project of Mask Guided (MG) Matting, presented in our paper: Mask Guided Matting via Progressive Refinement Network

Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!

YOLOv6: a single-stage object detection framework dedicated to industrial applications.

The Compute Library is a set of computer vision and machine learning functions optimised for both Arm CPUs and GPUs using SIMD technologies.

PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT

🎉持续更新：CUDA 12.2 PTX-ISA-8.2学习笔记，部分中文翻译 + 个人理解 + 内联汇编示例，讲解CUDA 12.2 PTX-ISA-8.2 汇编指令；进行中.....

pybind11中文文档（个人翻译）

Multi-platform high performance deep learning inference engine (飞桨多端多平台高性能深度学习推理引擎）

No description.

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

A high-throughput and memory-efficient inference and serving engine for LLMs

TensorRT Model Optimizer is a unified library of state-of-the-art model optimization techniques such as quantization and sparsity. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM or TensorRT to optimize inference speed on NVIDIA GPUs.

Cross-platform, customizable ML solutions for live and streaming media.

:zap: Based on Yolo's low-power, ultra-lightweight universal target detection algorithm, the parameter is only 250k, and the speed of the smart phone mobile terminal can reach ~300fps+

No description.

Deep learning model converter for PaddlePaddle. (『飞桨』深度学习模型转换工具)

Show PDF preview in VSCode.

High-efficiency floating-point neural network inference operators for mobile, server, and Web

程序语言与编译技术相关资料（持续更新中）

Implementations of SIMD instruction sets for systems which don't natively support them.

No description.

📚FFPA(Split-D): Yet another Faster Flash Prefill Attention with O(1) GPU SRAM complexity for headdim > 256, ~2x↑🎉vs SDPA EA.

My Personal GitHub Pages generated by Github Copilot.

No description.

PTX ISA 9.1 documentation converted to searchable markdown. Includes Claude Code skill for CUDA development.

ChatGLM2-6B: An Open Bilingual Chat LLM | 开源双语对话语言模型

ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型

ByteTrack: Multi-Object Tracking by Associating Every Detection Box

[NeurIPS'24 Spotlight, ICLR'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

FlashMLA: Efficient MLA Decoding Kernel for Hopper GPUs

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs

study of cutlass

[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型

CUDA Templates for Linear Algebra Subroutines

Fast and memory-efficient exact attention

Material for cuda-mode lectures

本项目旨在分享大模型相关技术原理以及实战经验。

For releasing code related to compression methods for transformers, accompanying our publications

CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

Person Image Synthesis via Denoising Diffusion Model (CVPR 2023)

Sample codes for my CUDA programming book

This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.

stb single-file public domain libraries for C/C++

Android library (AAR). Highly configurable, easily extendable deep zoom view for displaying huge images without loss of detail. Perfect for photo galleries, maps, building plans etc.

[IJCV 2022] Bridging Composite and Real: Towards End-to-end Deep Image Matting

DECA: Detailed Expression Capture and Animation (SIGGRAPH 2021)

Arm NN ML Software. The code here is a read-only mirror of https://review.mlplatform.org/admin/repos/ml/armnn

FB (Facebook) + GEMM (General Matrix-Matrix Multiplication) - https://code.fb.com/ml-applications/fbgemm/

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）

OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version.

👑 Easy-to-use and powerful NLP library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis and 🖼 Diffusion AIGC system etc.

CV-CUDA™ is an open-source, graphics processing unit (GPU)-accelerated library for cloud-scale image processing and computer vision.

A translator from Intel SSE intrinsics to Arm/Aarch64 NEON implementation

ONNX Model Exporter for PaddlePaddle

The minimal opencv for Android, iOS, ARM Linux, Windows, Linux, MacOS, WebAssembly

在Android使用深度学习模型实现图像识别，本项目提供了多种使用方式，使用到的框架如下：Tensorflow Lite、Paddle Lite、MNN、TNN

State-of-the-art 2D and 3D Face Analysis Project

🔥 A cross-platform build utility based on Lua

YOLO5Face: Why Reinventing a Face Detector (https://arxiv.org/abs/2105.12931)

http://www.facegood.cc

Android face detect and segmentation,facemesh by ncnn

Transformer related optimization, including BERT, GPT

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.

中文翻译 + 部分个人理解: ARMv9 SVE/SVE2 内联函数

No description.

整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。

👹 mattematte: A C++ ToolKit for matting, segmentation, SR and colorization with MNN, ONNXRuntime and Vulkan.

A Trimap-Free Solution for Portrait Matting in Real Time

A toolkit for making real world machine learning and data analysis applications in C++

CPU implementation of Soft-NMS for rotated rectangles

《Python 3 源码剖析》

A lightweight class for saving the best Tensorflow checkpoints.

An implement of Soft NMS algorithm in Python

An independent Python feature port of Claude Code, entirely rewritting from scratch using oh-my-codex. Educational Purpose only.

Simplify your onnx model

HunyuanImage-3.0: A Powerful Native Multimodal Model for Image Generation

A curated list of research papers, resources, and advancements on Diffusion Cache and related efficient diffusion model acceleration techniques.

A curated list of recent papers on efficient video attention for video diffusion models, including sparsification, quantization, and caching, etc.

Pytorch Implementation (unofficial) of the paper "Mean Flows for One-step Generative Modeling" by Geng et al.

Efficient facial landmark detector

text and image to video generation: CogVideoX (2024) and CogVideo (ICLR 2023)

SpargeAttention: A training-free sparse attention that can accelerate any model inference.

Code and data for the Chain-of-Draft (CoD) paper

xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism

DeepSeek 系列工作解读、扩展和复现。

Context parallel attention that accelerates DiT model inference with dynamic caching

Fast implementations of causal linear attention for autogressive language modeling (Pytorch)

LMDeploy is a toolkit for compressing, deploying, and serving LLM

Real-time face swap for PC streaming or video calls

人像卡通化探索项目 (photo-to-cartoon translation project)

Several optimization methods of half-precision general matrix multiplication (HGEMM) using tensor core with WMMA API and MMA PTX instruction.

No description.

CainCamera is an Android Project to learn about development of beauty camera, image and short video

DeepFaceLab is the leading software for creating deepfakes.

No description.

The Triton TensorRT-LLM Backend

No description.

Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface.

No description.

Official PyTorch Implementation of "Scalable Diffusion Models with Transformers"

NVIDIA® TensorRT™, an SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applications.

No description.

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

DeepCache: Accelerating Diffusion Models for Free

Best inference performance optimization framework for HuggingFace Diffusers on NVIDIA GPUs.

BEVFormer inference on TensorRT, including INT8 Quantization and Custom TensorRT Plugins (float/half/half2/int8).

Hackable and optimized Transformers building blocks, supporting a composable construction.

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

Run Mixtral-8x7B models in Colab or consumer desktops

A curated list of awesome distributed systems books, papers, resources and shiny things.

Awesome-LLM: a curated list of Large Language Model

Large Language Model Systems Paper List

No description.

An awesome & curated list of best LLMOps tools for developers

A standalone GEMM kernel for fp16 activation and quantized weight, extracted from FasterTransformer

Build Production-Grade AI Applications

A curated list of awesome Python frameworks, libraries, software and resources

No description.

A treasure chest for visual classification and recognition powered by PaddlePaddle

row-major matmul optimization

Infrastructure to enable deployment of ML models to low-power resource-constrained embedded targets (including microcontrollers and digital signal processors).

Easy-to-use image segmentation library with awesome pre-trained model zoo, supporting wide-range of practical tasks in Semantic Segmentation, Interactive Segmentation, Panoptic Segmentation, Image Matting, 3D Segmentation, etc.

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

An extension library of WMMA API (Tensor Core API)

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

106点人脸关键点检测的PFLD算法实现

No description.

Examples from Programming in Parallel with CUDA

Samples for CUDA Developers which demonstrates features in CUDA Toolkit

Some prebuilt libs of onnxruntime(1.7.0~1.10.0) for Android.

Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.

Common source, scripts and utilities for creating Triton backends.

No description.

Actively maintained ONNX Optimizer

Visualizer for neural network, deep learning, and machine learning models

Node.js to Python bridge :sparkles::snake::rocket::sparkles:

No description.

Productive & portable high-performance programming in Python.

Lightning fast C++/CUDA neural network framework

MNN is a blazing fast, lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba

Low-precision matrix multiplication

No description.

opencv

The platform independent header allowing to compile any C/C++ code containing ARM NEON intrinsic functions for x86 target systems using SIMD up to SSE4 intrinsic functions

JNI Bind is a set of advanced syntactic sugar for writing efficient correct JNI Code in C++17 (and up).

lib, demo, model, data

Arm neon optimization practice

The pre- and post- processing library for ONNX Runtime

Retinaface get 80.99% in widerface hard val using mobilenet0.25.

YOLOv3、YOLOv4、YOLOv5、YOLOv5-Lite、YOLOv6、YOLOv7、YOLOX、YOLOX-Lite、TensorRT、NCNN、Tengine、OpenVINO

A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.

:zap: A newly designed ultra lightweight anchor free target detection algorithm， weight only 250K parameters， reduces the time consumption by 30% compared with yolo-fastest, and the post-processing is simpler

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data

YOLOv5 in PyTorch > ONNX > CoreML > TFLite

3DV 2021: Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

No description.

PyTorch implementation of "Towards Accurate Facial Motion Retargeting with Identity-Consistent and Expression-Exclusive Constraints" (AAAI2022)

A simple, fully convolutional model for real-time instance segmentation.

Background Matting: The World is Your Green Screen

package google mediapipe hand and holistic tracking into a dynamic link library

Official repository for the paper F, B, Alpha Matting

[IJCAI'21] Deep Automatic Natural Image Matting

[ACM MM 2021] Privacy-Preserving Portrait Matting

Official repository of Semantic Image Matting

Four landmark detection algorithms, implemented in PyTorch.

Become a virtual character with just your webcam!

Face and Body Tracking for VRM 3D models on the web.

UE4 MediaPipe plugin

Blendshape and kinematics calculator for Mediapipe/Tensorflow.js Face, Eyes, Pose, and Finger tracking models.

No description.

chineseocr lite android onnx ，超轻量级中文ocr android demo，支持竖排文字识别, 支持onnx推理(psenet+anglenet+crnn)

No description.

Implementation of the deformation transfer paper and its application in generating all the ARkit facial blend shapes for any 3D face

Use ARkit application to drive face-blendshapes in TouchDesigner.

C++ inplementation of CNN(Cnnvolutional Neural Network) for image classification

A cross platform OCR Library based on PaddleOCR & OnnxRuntime

植物识别 (Plant Recognition)

Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set (CVPRW 2019). A PyTorch implementation.

昆虫识别 (Insect Recognition)

Public repository for the CVPR 2020 paper AvatarMe and the TPAMI 2021 AvatarMe++

yolort is a runtime stack for yolov5 on specialized accelerators such as libtorch, onnxruntime, tvm, tensorrt and ncnn.