Phil Wang

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

PyTorch implementation of Vision Transformer (ViT) for image classification, treating image patches as tokens and processing them through a Transformer encoder.

Implementation of Alphafold 3 from Google Deepmind in Pytorch

Implementation of Imagen, Google's Text-to-Image Neural Network, in Pytorch

A concise but complete full-attention transformer with a set of promising experimental features from various papers

Vector (and Scalar) Quantization, in Pytorch

Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI

Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch

Implementation of Denoising Diffusion Probabilistic Model in Pytorch

Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by https://twitter.com/advadnoun

Simplest working implementation of Stylegan2, state of the art generative adversarial network, in Pytorch. Enabling everyone to experience disentanglement

Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN. Technique was originally created by https://twitter.com/advadnoun

Reformer, the efficient Transformer, in Pytorch

🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI

Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch

Unofficial implementation of Titans, SOTA memory for transformers, in Pytorch

Implementation of GigaGAN, new SOTA GAN out of Adobe. Culmination of nearly a decade of research into GANs

Usable Implementation of "Bootstrap Your Own Latent" self-supervised learning, from Deepmind, in Pytorch

Implementation of 'lightweight' GAN, proposed in ICLR 2021, in Pytorch. High resolution image generations that can be trained within a day or two

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

Implementation of SoundStorm, Efficient Parallel Audio Generation from Google Deepmind, in Pytorch

Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute

Implementation of the training framework proposed in Self-Rewarding Language Model, from MetaAI

Implementation of Video Diffusion Models, Jonathan Ho's new paper extending DDPMs to Video Generation - in Pytorch

Implementation of Natural Speech 2, Zero-shot Speech and Singing Synthesizer, in Pytorch

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

Implementation of TabTransformer, attention network for tabular data, in Pytorch

An All-MLP solution for Vision, from Google AI

Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch

Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch

Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs

Transformer based on a variant of attention that is linear complexity in respect to sequence length

Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch

Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper

Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

Implementation of TimeSformer from Facebook AI, a pure attention-based solution for video classification

A concise but complete implementation of CLIP with various experimental improvements from recent papers

Implementation of Voicebox, new SOTA Text-to-speech network from MetaAI, in Pytorch

Implementation of Bottleneck Transformer in Pytorch

Implementation of MagViT2 Tokenizer in Pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

A simple way to keep track of an Exponential Moving Average (EMA) version of your Pytorch model

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

Implementation of the Point Transformer layer, in Pytorch

Implementation of π₀, the robotic foundation model architecture proposed by Physical Intelligence

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

Implementation of Classifier Free Guidance in Pytorch, with emphasis on text conditioning, and flexibility to include multiple text embedding models

Implementation of a single layer of the MMDiT, proposed in Stable Diffusion 3, in Pytorch

Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

Implementation of E(n)-Equivariant Graph Neural Networks, in Pytorch

Implementation of E2-TTS, "Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS", in Pytorch

Pytorch implementation of SIREN - Implicit Neural Representations with Periodic Activation Function

An implementation of local windowed attention for language modeling

Implementation of Slot Attention from GoogleAI

Implementation of rectified flow and some of its followup research / improvements in Pytorch

Implementation of RT1 (Robotic Transformer) in Pytorch

Implementation of the convolutional module from the Conformer paper, for use in Transformers

Implementation of Autoregressive Diffusion in Pytorch

Implementation of gMLP, an all-MLP replacement for Transformers, in Pytorch

Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

Implementation of Axial attention - attending to multi-dimensional data efficiently

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch

Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"

Implementation of Bit Diffusion, Hinton's group's attempt at discrete denoising diffusion, in Pytorch

Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.

Implementation of the proposed minGRU in Pytorch

Explorations into training LLMs to use clinical calculators from patient history, using open sourced models. Will start with Wells' Criteria

Explorations into some recent techniques surrounding speculative decoding

Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch

Implementation of Linformer for Pytorch

Fully featured implementation of Routing Transformer

Quick implementation of nGPT, learning entirely on the hypersphere, from NvidiaAI

Implementation of a U-net complete with efficient attention as well as the latest research findings

Implementation of the Equiformer, SE3/E3 equivariant attention network that reaches new SOTA, and adopted for use by EquiFold for protein folding

Implementation of Lumiere, SOTA text-to-video generation from Google Deepmind, in Pytorch

Implementation of a Transformer, but completely in Triton

Implementation of Spear-TTS - multi-speaker text-to-speech attention network, in Pytorch

Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention

Use Jax functions in Pytorch

Implementation of MetNet-3, SOTA neural weather model out of Google Deepmind, in Pytorch

Implementation of Graph Transformer in Pytorch, for potential use in replicating Alphafold2

A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

Implementation of Flash Attention in Jax

Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT

Implementation of Block Recurrent Transformer - Pytorch

Implementation of Recurrent Interface Network (RIN), for highly efficient generation of images and video without cascading networks, in Pytorch

A simple cross attention that updates both the source and target in one step

Implementation of Danijar's latest iteration for his Dreamer line of work

Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways - in Jax (Equinox framework)

Attempt to make multiple residual streams from Bytedance's Hyper-Connections paper accessible to the public

Unofficial implementation of Tiny Recursive Model (TRM), improvement to HRM from Sapient AI, by Alexia Jolicoeur-Martineau

Implementation of TiTok, proposed by Bytedance in "An Image is Worth 32 Tokens for Reconstruction and Generation"

Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch

Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning

Implementation of ProteinBERT in Pytorch

Pytorch implementation of Compressive Transformers, from Deepmind

Implementation of Chroma, generative models of protein using DDPM and GNNs, in Pytorch

Implementation of a framework for Genie2 in Pytorch

Implementation of the new SOTA for model based RL, from the paper "Improving Transformer World Models for Data-Efficient RL", in Pytorch

Implementation of ETSformer, state of the art time-series Transformer, in Pytorch

A simple to use pytorch wrapper for contrastive self-supervised learning on any neural network

Implementation of Nyström Self-attention, from the paper Nyströmformer

Implementation of the proposed Adam-atan2 from Google Deepmind in Pytorch

Implementation of Diffusion Policy, Toyota Research's supposed breakthrough in leveraging DDPMs for learning policies for real-world Robotics

Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind

Pytorch implementation of MIMO, Controllable Character Video Synthesis with Spatial Decomposed Modeling, from Alibaba Intelligence Group

Implementation of STAM (Space Time Attention Model), a pure and simple attention model that reaches SOTA for video classification

A practical implementation of GradNorm, Gradient Normalization for Adaptive Loss Balancing, in Pytorch

Implementation of RQ Transformer, proposed in the paper "Autoregressive Image Generation using Residual Quantization"

An implementation of PPO in Pytorch

Implementation of the Large Behavioral Model architecture for dexterous manipulation from Toyota Research Institute

LocoFormer - Generalist Locomotion via Long-Context Adaptation

Implementation of Mimic-Video, Video-Action Models for SOTA Generalizable Robot Control Beyond VLAs

Pytorch implementation of Evolutionary Policy Optimization, from Wang et al. of the Robotics Institute at Carnegie Mellon University

Implementation of a multimodal diffusion transformer in Pytorch

Implementation of HS-TasNet, "Real-time Low-latency Music Source Separation using Hybrid Spectrogram-TasNet"

Implementation of the MetaController proposed in "Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning" from the Paradigms of Intelligence team at Google

Implementation of the proposed DeepCrossAttention by Heddes et al at Google research, in Pytorch

Explorations into the proposal from the paper "Grokfast, Accelerated Grokking by Amplifying Slow Gradients"

A Transformer made of Rotation-equivariant Attention using Vector Neurons

Implementation of Gated State Spaces, from the paper "Long Range Language Modeling via Gated State Spaces", in Pytorch

Implementation of AlphaGenome, Deepmind's updated genomic attention model

Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch

Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch

Implementation of the transformer proposed in "Building Blocks for a Complex-Valued Transformer Architecture"

Thispersondoesnotexist went down, so this time, while building it back up, I am going to open source all of it.

Implementation of the Belief State Encoder / Decoder in the new breakthrough robotics paper from ETH Zürich

Implementation of a Transformer that Ponders, using the scheme from the PonderNet paper

Implementation of N-Grammer, augmenting Transformers with latent n-grams, in Pytorch

Implementation of the dynamic chunking mechanism in H-net by Hwang et al. of Carnegie Mellon

Implementation of Dreamcraft3D, 3D content generation in Pytorch

The Gaussian Histogram Loss (HL-Gauss) proposed by Imani et al. with a few convenient wrappers for regression, in Pytorch

Exploration into the proposed architecture from Sapient Intelligence of Singapore 🇸🇬

Implementation of the general framework for AMIE, from the paper "Towards Conversational Diagnostic AI", out of Google Deepmind

Implementation of D4RT, Efficiently Reconstructing Dynamic Scenes, from Deepmind

Explorations into whether a transformer with RL can direct a genetic algorithm to converge faster

Implementation of Memory-Compressed Attention, from the paper "Generating Wikipedia By Summarizing Long Sequences"

An implementation of (Induced) Set Attention Block, from the Set Transformers paper

Implementation of Soft Actor Critic and some of its improvements in Pytorch

Efficient implementation (and explorations) into polar coordinate positional embedding (PoPE) - from Gopalakrishnan et al. under Schmidhuber

Contrastive Reinforcement Learning

Implementation of RL-100, Performant Robotic Manipulation with Real-World Reinforcement Learning

Implementation of the world model architecture for self driving out of Wayve

Toy genetic algorithm in Pytorch

Exploring an idea where one forgets about efficiency and carries out attention across each edge of the nodes (tokens)

Implementation of the proposed LVMAE, from the paper, Extending Video Masked Autoencoders to 128 frames, in Pytorch

Unofficial implementation of Hippoformer, Integrating Hippocampus-inspired Spatial Memory with Transformers

Implementation of various evolutionary algorithms, starting with evolutionary strategies

Some utility functions to help myself (and perhaps others) go faster with ML/AI work

Implementation of a Light Recurrent Unit in Pytorch

Just a repository that will house some MLPs and their variants, so to avoid having to reimplement them again and again for different projects (especially RL)

Implementation of 2-simplicial attention proposed by Clift et al. (2019) and the recent attempt to make practical in Fast and Simplex, Roy et al. (2025)

Implementation of Humanoid Standing Up, from the paper "Learning Humanoid Standing-up Control across Diverse Postures" out of Shanghai, in Pytorch

Implementation of Kalmanformer, modeling the Kalman gain with a transformer

Explorations into NEAT and some of its derivative research

Exploration into Discrete Distribution Network, by Lei Yang out of Beijing

Implementation of the Hybrid Perception Block and Dual-Pruned Self-Attention block from the ITTR paper for Image to Image Translation using Transformers

Explorations into the proposed Streaming Deep Reinforcement Learning, from University of Alberta

Implementation of Poly-attention, a higher-order self-attention proposed by Chakrabarti et al. of Columbia

Implementation of Fast Weight Attention

Explorations into the proposed SDFT, Self-Distillation Enables Continual Learning, from Shenfeld et al. of MIT

Implementation of the model architecture for SRT-H

Causal Attention with Lookahead Keys

Implementation of Recurrent Independent Mechanisms in Pytorch

Implementation of RigidFormer, Learning Rigid Dynamics using Transformers

Implementation of WorldParticle, Unified World Simulation of Lagrangian Particle Dynamics via Transformer

Implementation and explorations into DiscoRL, Discovering state-of-the-art reinforcement learning algorithms, David Silver's last work at Deepmind

Embedding and readout for simple multi-categorical and gaussian continuous

Implementation of the Hierarchical Latent Action Model, proposed by Hanjung Kim et al. of Yonsei University

Exploration into some new research surrounding value networks

Implementation and explorations into PopuLoRA, Co-Evolving LLM Populations for Reasoning Self-Play

Implementation of Multiscreen proposed by Ken Nakanishi for "Screening is Enough"

No description.

Implementation of Attention Schema-based Attention Control (ASAC), A Cognitive-Inspired Approach for Attention Management in Transformers

Implementation and explorations into MPO / DMPO

Implementation of the pseudo projector proposed by Vitaly Bulgakov at Mass General Brigham

A simple numpy memmap replay buffer for RL and personal use-cases

Flash Attention Triton kernel with support for second-order derivatives

Wrappers around environments that will take care of providing representations from self supervised learning automagically

Explorations into the proposal from the paper, Visual-tactile pretraining and online multitask learning for humanlike manipulation dexterity

Implementation of RISE

Development repository for the Triton language and compiler

DeepSeek 4 Flash local inference engine for Metal and CUDA

Toward High-Accuracy Open-Source Biomolecular Structure Prediction.