OpenPlan Labs

openplan-bench

One harness, one problem set, one results schema for every planner in the org. Planners not included — they are extras.

What a number here means

A benchmark that reports only its successes is an advertisement. This one records every run it was asked to make, and the tables above show the failures next to the wins because that is the comparison a reader needs.

Outcomes

solved
The planner returned a plan and an independent validator accepted it. For classical planning the plan is replayed through the grounded task and the goal is re-checked; for MAPF the returned paths are re-scanned for vertex and edge conflicts. A planner's own claim of success is never sufficient — a plan the validator rejects is shown as invalid and does not count towards coverage.
unsolved
The planner terminated on its own and reported no solution. For an incomplete planner this is data, not a bug: it means the search strategy failed on this instance, not that the instance is unsolvable.
timeout
The wall-clock budget was exhausted. Two numbers are recorded and they are not the same one. timeout_s is the budget the run was given. wall_time_s is the elapsed time at which the run was actually stopped — a little past the budget, because a planner notices its own limit and unwinds, and so it is a measurement rather than a constant. Neither is an extrapolation of how long a solution would have taken; solvability is unknown. This is the single most important distinction on the page: a timeout is not a proof of unsolvability. Timing statistics in the tables are computed from solved runs only, so a timeout moves coverage and nothing else.
memory
The address-space limit fired, or the process was killed by the OOM killer. As with a timeout, solvability is unknown.
error
The planner raised, crashed, or produced output the adapter could not read. The message is kept in the row. Parse failures on a domain a planner does not support land here, which is deliberate: "cannot read this domain" is a real limitation and belongs in the table.
skipped
Deliberately not run. The CUDA arm on a machine with no working GPU is the common case, and the row says so rather than vanishing.
not-installed
The backend was not importable in the environment that produced these results. The runs were still enumerated, so a gap in the table is always explained.

Coverage

An instance counts as solved for a configuration only when every seed returned a valid solution. A configuration that solves an instance on two seeds out of three is reported as partial, not as solved: a coverage number that a re-run would not reproduce is not worth printing. Runs that never happened are excluded from the denominator rather than counted as failures.

Timing

Budgets

A coverage number is meaningless without the budget it was measured under: “solved 9 of 18” says nothing until you know whether the planner had twenty seconds or twenty minutes. Every table on this site carries its suite's budget in its caption; this is all of them in one place.

Read from the committed rows, not from the suite files — these are the budgets the measurements were actually taken under.
SuiteWall clockAddress spaceRows
Weekly regression suite15 s2,048 MiB68
Classical planning — coverage20 s3,072 MiB108
Classical planning — smoke30 s3,072 MiB234
MAPF — obstacle density20 s3,072 MiB150
MAPF — scaling with agent count20 s3,072 MiB400

The wall-clock budget is enforced twice — the planner is asked to stop itself at it, and the harness kills the child 15 s later if it has not. The address-space figure is a hard RLIMIT_AS set inside the child before the planner is imported, so exceeding it produces a memory row rather than a swapping machine and a meaningless sweep.

Hardware these numbers came from

Every row carries the CPU model, platform, Python version, package versions, seed and harness git SHA it was produced under, so results files from different machines stay interpretable after they are concatenated.

timestamp_utc is the exception and is worth stating exactly: in a schema 2 results file it is one stamp per suite run, taken when the run started and copied onto every row of that file — not a per-row clock reading, and not the moment any individual measurement was taken. Rows written from schema 3 onwards are stamped as each one is produced. The published tables above therefore show a run start, not a per-row time, and say so.

What this does not measure

Read this section before quoting a number.