Reproduce · openplan-bench

Running this yourself

Everything on the leaderboard comes from files in the repository. There is no hidden state and no database.

git clone https://github.com/openplan-labs/openplan-bench.git
cd openplan-bench

# The classical suites resolve corpus_root against the suite file, so the
# corpus belongs at corpus/pddl-examples inside this checkout.
git clone https://github.com/openplan-labs/pddl-examples corpus/pddl-examples

python -m venv .venv && source .venv/bin/activate
pip install -e '.[all,dev]'

python -m openplan_bench run suites/ci-weekly.yaml --out results/
python -m openplan_bench run suites/classical-coverage.yaml --out results/
python -m openplan_bench run suites/classical-smoke.yaml --out results/
python -m openplan_bench run suites/mapf-density.yaml --out results/
python -m openplan_bench run suites/mapf-scaling.yaml --out results/
python -m openplan_bench report results/
python -m openplan_bench site --results results/ --out docs/

What will differ on your machine

Wall times will differ, possibly a lot. A faster CPU moves every number; a shared or thermally throttled one moves them unevenly. Compare within one table.
Coverage may differ at the boundary. Instances that finished just inside the budget here may time out on a slower machine. That is a real result and it will be recorded as a timeout. The budget each suite was measured under is on the methodology page and in every table's caption; changing timeout_s in a suite file changes what the coverage column means, so say so if you do.
Node expansions, plan cost, plan length, makespan and validity will not differ. Those are deterministic given the instance and the seed. If they differ, something has genuinely changed — that is the comparison worth making after a code change.
The corpus moves. pddl-examples syncs new domains daily. Suites name their instances explicitly so a re-run measures the same problems; check out the corpus at the SHA in the results if you need byte-identical inputs.

Reading the results files

Results live in results/<suite>/<date>.csv, one row per measured run, never overwritten. latest.json beside them holds the same rows plus the suite header. Rows are wide on purpose: each carries the planner, instance, seed, outcome, metrics and the budget it ran under, the machine, the package versions and the harness SHA that produced it.

Two columns are easy to confuse. timeout_s is the budget the run was given; wall_time_s is what was measured. On a timeout row that measurement is the elapsed time at which the run was stopped, which sits a little past the budget — it is not the budget, and it is not an estimate of how long a solution would have taken.

import openplan_bench.records as records
rows = records.read_csv("results/ci-weekly/2026-08-20.csv")
print(sum(r.outcome == "timeout" for r in rows), "timeouts")

Adding your own planner

Write one adapter, add one line to the registry, and it appears here. The interface is three methods and is documented in the README; openplan_bench/adapters/fake.py is the smallest complete example.