🎮 EgoCS-400K: An Egocentric Gameplay Dataset for World Models

A large-scale ego-centric CS dataset with synchronized video, replay telemetry, action chains, segments, and captions.

Quick Navigation:
Overview
 · 
Dataset Composition
 · 
Data Construction

📖 Overview

EgoCS-400K aligns first-person gameplay video with demo-derived tick-level telemetry,
keyboard/mouse inputs, atomic actions, protected action chains, DP-based temporal segments,
and multi-grained video-language captions.

🎥 Player-view video   +  
🕹️ Per-tick game state   +  
⚔️ Action/segment annotations   +  
📝 Multi-grained captions

> 📢 Release status. EgoCS-400K is publicly released. The dataset, viewer,
> annotations, code, technical report, and project page will continue to be
> expanded and updated.

⏱️ 10,000+ hrsgameplay video
🔁 40,000+ roundsround-level trajectories
🗺️ 13 mapsCSGO / CS2 scenes
👥 10 playersper-round perspectives
🧩 6 annotationsfrom ticks to captions

🚀 Ongoing Updates

• [x] Release the EgoCS-400K dataset on Hugging Face.
• [x] Release the interactive dataset viewer on Hugging Face Spaces.
• [ ] Release the technical report.
• [ ] Release the processing and annotation code.

✨ Why EgoCS? Key Features

> Beyond raw gameplay video: EgoCS connects every visible moment to the
> player's controls, game state, actions, and temporally coherent behaviors.

🎥 Replay-Grounded Video
First-person video aligned with per-tick camera, spatial, weapon, player, round, and event states.

⌨️ Control-Level Traces
Fine-grained keyboard and mouse inputs, including movement, fire, reload, inspect, and scope.

🧩 Multi-Level Annotations
One hierarchy spanning per-tick states, atomic actions, protected chains, segments, and full sequences.

🧮 Action-Aware Segmentation
Dynamic programming creates model-ready clips without breaking meaningful action chains.

🔍 Quality-Aware Filtering
Visual, spatial, and action signals remove invalid renders and low-information segments.

🤖 VLM-Ready & Interactive
Structured captioning context plus a viewer for inspecting videos, timelines, actions, and segments.

🧩 Dataset Composition

Multi-Grained Composition

EgoCS is designed for models that need more than a caption over a clip. Each
sample can be studied at multiple temporal and semantic levels:

| 🧩 Level | 🎮 What it represents | 🎯 Typical supervision |
| --- | --- | --- |
| Player-view sequence | A full first-person round/player trajectory | Long video understanding, player behavior modeling |
| Coherent segment | A model-ready clip with stable temporal boundaries | Segment captioning, retrieval, video-language alignment |
| Protected action chain | A group of actions that should not be cut apart | Action-aware segmentation, long action modeling |
| Atomic replay action | Fire, inspect, reload, switch, jump, grenade, scope, etc. | Fine-grained action recognition/localization |
| Per-tick state trace | 32 Hz movement, camera, input, weapon, and player state | Grounded control/action analysis and evaluation |

🔎 Data Examples

The videos below present two EgoCS-400K samples with synchronized first-person
video, keyboard/mouse states, replay-grounded actions, environment attributes,
and VLM-generated prompts.

<!-- TEMP-HIDDEN: Annotation Files — uncomment to restore

📁 Annotation Files

The Hugging Face-ready export organizes each player-round sample as follows:

text
matches//round-XX//
|-- sample_meta.json
|-- videos/
|   |-- original.mp4
|   `-- overlay.mp4                        # Optional
|-- parsed/
|   |-- action.csv                         # Per-tick player state and controls
|   |-- segment.csv                        # Atomic action spans
|   `-- summary.json
|-- annotations/
|   |-- action.json                        # Atomic action timeline
|   |-- protected_action.json              # Protected action chains
|   |-- dp_segment.json                    # DP-based coherent segments
|   |-- camera.json                        # Camera trajectory features
|   `-- movement.json                      # Movement/spatial features
`-- captions/
    |-- protected_caption.json
    `-- segment_caption.json

Global indexes are exported in both JSONL and Parquet:

text
metadata/
|-- matches.jsonl / matches.parquet
|-- rounds.jsonl / rounds.parquet
|-- player_rounds.jsonl / player_rounds.parquet
|-- segments.jsonl / segments.parquet
|-- action_events.jsonl / action_events.parquet
`-- captions.jsonl / captions.parquet

Raw .dem files are used for local processing but are not included in the
published dataset export.
-->

<!-- TEMP-HIDDEN: Released Samples — uncomment to restore

🧪 Released Samples

The current release includes two parsed player-round examples:

| Match | Round | Sequence | Player | Duration | ⚔️ Atomic actions | 🎞️ Final segments |
| --- | ---: | ---: | --- | ---: | ---: | ---: |
| 9ine-vs-sparta-m1-dust2 | 8 | 9 | SoLb | 78.9 s | 37 | 14 |
| 9ine-vs-sparta-m1-dust2 | 10 | 11 | bnox | 109.4 s | 90 | 18 |

Release statistics:

| Item | Value |
| --- | ---: |
| Matches | 1 |
| Maps | 1 (dust2) |
| Player-round samples | 2 |
| Total video duration | 188.3 seconds |
| Atomic actions | 127 |
| Protected action chains | 40 |
| Final video segments | 32 |
| Demo tick rate | 64 Hz |
-->

🏗️ Data Construction

How EgoCS-400K is built from public replay files into synchronized videos,
structured annotations, filtered segments, and multi-grained captions — the full
step-by-step construction pipeline is documented in a dedicated guide.

Demo Collection · Video Rendering · Rendered Video Filter · Per-tick Annotation ·
DP Segmentation · Spatial / Action Filter · VLM Captioning

🗂️ Repository Layout

text
.
|-- README.md
|-- data/
|   |-- derived/parsed_rounds/              # Lightweight CSV/JSON examples
|   `-- viewer_assets/round_player_viewer/  # Viewer HTML and annotations
|-- docs/
|   |-- data-construction.md               # Data construction & pipeline guide
|   `-- images/                            # README figures
|       |-- egocs-teaser.png
|       `-- pipeline-overview.png
|-- pipeline/
|   |-- parse_dem/                         # Demo parsing
|   |-- overlay/                           # Video overlay rendering
|   |-- action_timeline/                   # Timeline and protected chains
|   |-- viewer_build/                      # Viewer asset generation
|-- requirements.txt
`-- .gitignore                             # Excludes demos, videos, caches

🚀 Getting Started

✅ Requirements

• Python 3.10 or newer
• ffmpeg and ffprobe on PATH for video processing
• An API key only if VLM caption generation is used

bash
python -m venv .venv

# Linux/macOS
source .venv/bin/activate

# Windows PowerShell
# .venv\Scripts\Activate.ps1

pip install -r requirements.txt

Install pyarrow as well when creating Parquet metadata:

bash
pip install pyarrow

🧾 Parse a Demo

bash
python pipeline/parse_dem/cs2_round_player_parser.py \
  data/raw/demos/9ine-vs-sparta-m1-dust2.dem \
  --out-dir data/derived/parsed_rounds/9ine-vs-sparta-m1-dust2 \
  --write-events

Use --steamid or --sequence to limit the export. Both options can be
repeated.

🖥️ Build Viewer Assets

Build the two samples included in this release:

bash
python pipeline/viewer_build/build_round_player_viewer.py \
  --game-root data/derived/videos_overlay/9ine-vs-sparta-m1-dust2 \
  --all-sequences-root data/derived/parsed_rounds/9ine-vs-sparta-m1-dust2 \
  --original-video-root data/raw/videos_original \
  --output-root data/viewer_assets/round_player_viewer \
  --round 8 \
  --round 10

Overlay video is optional:

bash
python pipeline/viewer_build/build_round_player_viewer.py \
  --all-sequences-root data/derived/parsed_rounds/9ine-vs-sparta-m1-dust2 \
  --original-video-root data/raw/videos_original \
  --output-root data/viewer_assets/round_player_viewer \
  --skip-overlay \
  --round 8 \
  --round 10

Use --overlay-optional when only some samples have overlay videos.

📝 Citation

If you use this dataset or pipeline, please cite:

bibtex
@misc{guo2026egocs400k,
  title={EgoCS-400K: An Egocentric Gameplay Dataset for World Models},
  author={Guo, Rongjin and Liang, Dong and Liu, Yuhao and Liu, Fang and Huang, Tianyu and Hancke, Gerhard P. and Lau, Rynson W. H.},
  year={2026},
  note={Project page: https://EgoCS-400K.github.io}
}

Collapse