Hermit

Tools for Reproducible Linux Execution

Hermit runs unmodified Linux programs in a controlled, repeatable environment. It controls inputs that can change a program's outcome: thread scheduling, time, and random numbers. Use it to reproduce failures, explore different thread schedules, and investigate concurrency bugs.

Get started Rust API

Read the original introduction:
Hermit: Deterministic Linux for Controlled Testing and Software Bug-finding

Pause animation

GuestYour Linux program

HermitControls execution

LinuxHost kernel

Thread scheduling
Clocks
Random bytes
Process IDs
File
metadata
CPU clock
and features

Hermit sits between your program and Linux, controlling system calls and processor events that can make executions differ.

Why Hermit?

A running program, such as a build job or a CI test, has more inputs than its arguments and files. Its result can also depend on which thread runs first, what the clock says, or which random bytes it receives. These implicit inputs can make a failure difficult to reproduce.

Repeat an execution

Run a supported program again with the same inputs and settings to investigate a failure.

Look for concurrency bugs

Change the scheduling seed to explore other thread interleavings. Keep the seed to reproduce an interesting run with the same program and Hermit build.

Narrow down a failure

Compare passing and failing schedules to find changes in event order that trigger the bug.

Reproducible runs need a fixed program, input files, environment, and Hermit configuration, including the starting clock time. Responses from external services outside the container must also be controlled: repeatable, blocked, or recorded.

Try Hermit

Install Hermit on x86-64 Linux, then try reading random bytes:

In your terminal

hermit run -- /bin/sh -c 'od -An -N8 -tx1 /dev/urandom'

Run the command twice. With the same Hermit build and random seed, it prints the same bytes. Hermit supplies repeatable data when the program asks Linux for randomness.

To run your own program, place hermit run -- before its command. The user guide explains host requirements, clock settings, and schedule exploration.

Beta

Record and replay

Record a program execution, then replay it to investigate a failure. Hermit offers a workflow similar to Mozilla's rr, with GDB debugging. Its experimental Debug Adapter Protocol (DAP) adapter is tested end to end with GDB 17.2, including reverse stepping through a recording; other GDB versions are not tested. See the instructions below for details.

Record/replay supports fewer workloads than deterministic execution.

Record/replay guide DAP status and instructions

Software components

Use Hermit to run programs, or use the libraries below to build your own tools.

Run programs

Hermit

A command-line tool for repeatable execution and concurrency testing of x86-64 Linux programs. Hermit uses Reverie to intercept system calls and processor events, then controls scheduling, time, and other inputs to the program.

Source Rust API

Build tools

Reverie

A Rust library for building tools that observe and change the behavior of Linux programs. Write handlers for system calls and processor events to inspect a program, change results, or perform additional operations. Hermit uses these handlers to implement deterministic execution.

Source and guide

Patch instructions

LiteInst

A library for patching x86-64 machine instructions while a program runs. LiteInst2 brings the original C++ project's patching core to Rust. Hermit's experimental LiteInst integration uses this technique to intercept program operations.

LiteInst2 Rust port Original LiteInst PLDI '16 — Living on the edge PLDI '17 — Instruction punning Hermit integration

Compatibility

Hermit supports x86-64 Linux. Which programs work depends on the system calls they use, the host machine, and how Hermit intercepts their execution.

These interception mechanisms are called execution backends. Start with the default, ptrace, which uses Linux's process-tracing interface and is the most thoroughly tested. Other backends are experimental or specialized.

Recorded passing workloads on the ptrace backend include:

Python scripts using hash collections, files, and subprocesses.
Node.js running a hot JavaScript loop and generating random numbers.
Rust programs using randomized hash collections, files, and clocks.
SQLite creating and reopening a database, then running queries that use random numbers and time.

These checks compare repeated executions. The scorecard records results separately for each workload, mode, and backend; try the workload you depend on.

Latest compatibility scorecard

Recent scorecard builds lists each kept build with its date, Hermit commit, and size.

Experiments

Linux, filesystems, and builds

Some of the larger workloads explored with Hermit.

Boot and inspect Linux under QEMU

A minimal Linux guest boots under QEMU with Hermit's controlled thread scheduling and QEMU's instruction-based clock. Repeated boots matched after log normalization. Separate experiments with the drgn kernel debugger reproduced kernel task lists at chosen stopping points.

QEMU boot guide

Reproduce a btrfs concurrency bug

With a historical bug restored in an instrumented btrfs-convert and its progress timer replaced by a pipe, Hermit's chaos scheduler triggered a use-after-free. Repeating the seed reproduced the same failure report; the fixed version had no such crashes in the same sweep.

Explore thread schedules

Reproduce build outputs

Hermit has rebuilt Debian packages such as hello, hostname, and tree with byte-identical package files across independent build environments. A Nix builder prototype also made an example build that reads the clock and random bytes reproducible.