perf-tools

perf-tools is a collection of shell scripts for measuring performance on Linux systems, created by Brendan Gregg, a well-known author in the systems performance field. The tools are built on top of two tracing systems that come built into the Linux kernel: ftrace and perf_events. These kernel features let you observe what the operating system is doing at a low level, such as which functions are being called, how long disk operations take, and which processes are starting or stopping.

The individual tools each do one specific job. For example, iosnoop shows disk I/O operations as they happen with timing details, iolatency produces a histogram showing how long disk reads and writes take, execsnoop reports every new process that starts along with its command-line arguments, and opensnoop tracks which files are being opened and by which processes. There are also tools for counting kernel function calls, tracing network TCP retransmissions, measuring cache hit and miss rates, and inspecting signals sent between processes.

The tools are designed to work on Linux 3.2 and newer kernels with minimal dependencies. Most require the kernel to have ftrace enabled, which many distributions include by default. Some also require the perf command, available as a package on most Linux distributions, and a mounted debugfs filesystem. A few tools mention using eBPF as a future direction for more capable versions.

The collection is described as in-development and unsupported. Many tools include workarounds for limitations in existing kernels and come with caveats documented in their manual pages. The README includes multiple example outputs showing what each tool prints when run on a real system.