What this is. Every planner in OpenPlan Labs, run over shared problem sets by one harness that records timeouts and errors as results rather than dropping them. 5 suite(s), 960 recorded runs. What “solved” means · How to reproduce this.
Weekly regression suiterunner-grade
A small cross-section of both families, run on a shared GitHub Actions runner every week. Its job is to notice that something stopped working, not to rank hardware. Compare coverage and node counts across runs; ignore the seconds.
Runner-grade timings. This suite was measured on a shared GitHub Actions runner. Absolute wall times there vary by a factor of two or more between runs and tell you nothing about the hardware a planner would see in practice. Coverage, plan cost, node counts and validity are still exact — those do not depend on how fast the machine was. Compare times within a single run of this table, never across runs or against your laptop.
Single-sample timings. 32 of the 44 cells in this suite are one measurement: one seed, one repetition. Median, Min and Max are then the same number three times — a sample, not an observed range. Node counts, cost and validity are unaffected, being deterministic; the seconds should not be quoted with a spread until the suite is re-run with more seeds.
These six count rows — one per (configuration, instance, seed, repetition). The Solved column in the leaderboard below counts instances, and only those a configuration solved on every seed, so the two numbers are different measurements of different things and will not agree.
Leaderboard
| Configuration | Family | Coverage | Solved | Instances | Timeouts | Errors | Unsolved | Median time solved only | Total time solved only | Expanded solved only |
|---|---|---|---|---|---|---|---|---|---|---|
| jupyddl:gbfs/hff | classical | 100% | 8 | 8 | 0 | 0 | 0 | 2.1 ms | 518.0 ms | 118 |
| jupyddl:bfs | classical | 75% | 6 | 8 | 2 | 0 | 0 | 1.7 ms | 11.8 ms | 156 |
| jupyddl:astar/hmax | classical | 75% | 6 | 8 | 2 | 0 | 0 | 1.9 ms | 16.2 ms | 153 |
| jupyddl:astar/lmcut | classical | 75% | 6 | 8 | 2 | 0 | 0 | 2.7 ms | 54.6 ms | 153 |
| pymapf:pibt | mapf | 100% | 3 | 3 | 0 | 0 | 0 | 4.4 ms | 15.9 ms | 155 |
| pymapf:lacam | mapf | 100% | 3 | 3 | 0 | 0 | 0 | 4.8 ms | 17.1 ms | 155 |
| pymapf:prioritized | mapf | 100% | 3 | 3 | 0 | 0 | 0 | 3.5 ms | 21.7 ms | 84 |
| pymapf:cbs | mapf | 67% | 2 | 3 | 1 | 0 | 0 | 1.7 ms | 3.3 ms | 987 |
Coverage is Solved ÷ Instances, and both are columns here so the denominator is never implied. Median time, Total time and Expanded are computed over the instances a configuration actually solved, so each row has a different denominator and none of those three are comparable across rows — including when you sort by them. A configuration that solves only the easy instances posts the fastest time and the smallest node count on this table; that is an artefact of the subset it solved, not a result. Read them against Coverage and Instances, never instead of them.
Figures
Every instance
| Instance | Configuration | Outcome | Median time | Min | Max | Cost | Length | Makespan | Expanded | Samples seeds x reps | Note |
|---|---|---|---|---|---|---|---|---|---|---|---|
| blocksworld/problem | jupyddl:astar/hmax | solved | 1.2 ms | 1.2 ms | 1.2 ms | 2 | 2 | — | 2 | 1/1 | |
| miconic/s1-0 | jupyddl:astar/hmax | solved | 1.8 ms | 1.8 ms | 1.8 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:astar/hmax | solved | 1.8 ms | 1.8 ms | 1.8 ms | 3 | 3 | — | 3 | 1/1 | |
| miconic/s10-0 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-0 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| pallet/problem | jupyddl:astar/hmax | solved | 5.9 ms | 5.9 ms | 5.9 ms | 12 | 12 | — | 127 | 1/1 | |
| switch/problem | jupyddl:astar/hmax | solved | 3.7 ms | 3.7 ms | 3.7 ms | 3 | 3 | — | 5 | 1/1 | |
| tsp/problem | jupyddl:astar/hmax | solved | 1.9 ms | 1.9 ms | 1.9 ms | 15 | 5 | — | 12 | 1/1 | |
| blocksworld/problem | jupyddl:astar/lmcut | solved | 1.5 ms | 1.5 ms | 1.5 ms | 2 | 2 | — | 2 | 1/1 | |
| miconic/s1-0 | jupyddl:astar/lmcut | solved | 2.4 ms | 2.4 ms | 2.4 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:astar/lmcut | solved | 1.9 ms | 1.9 ms | 1.9 ms | 3 | 3 | — | 3 | 1/1 | |
| miconic/s10-0 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-0 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| pallet/problem | jupyddl:astar/lmcut | solved | 40.9 ms | 40.9 ms | 40.9 ms | 12 | 12 | — | 127 | 1/1 | |
| switch/problem | jupyddl:astar/lmcut | solved | 4.8 ms | 4.8 ms | 4.8 ms | 3 | 3 | — | 5 | 1/1 | |
| tsp/problem | jupyddl:astar/lmcut | solved | 3.0 ms | 3.0 ms | 3.0 ms | 15 | 5 | — | 12 | 1/1 | |
| blocksworld/problem | jupyddl:bfs | solved | 1.1 ms | 1.1 ms | 1.1 ms | 2 | 2 | — | 2 | 1/1 | |
| miconic/s1-0 | jupyddl:bfs | solved | 1.7 ms | 1.7 ms | 1.7 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:bfs | solved | 1.7 ms | 1.7 ms | 1.7 ms | 3 | 3 | — | 4 | 1/1 | |
| miconic/s10-0 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-0 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| pallet/problem | jupyddl:bfs | solved | 2.5 ms | 2.5 ms | 2.5 ms | 12 | 12 | — | 124 | 1/1 | |
| switch/problem | jupyddl:bfs | solved | 3.1 ms | 3.1 ms | 3.1 ms | 3 | 3 | — | 4 | 1/1 | |
| tsp/problem | jupyddl:bfs | solved | 1.6 ms | 1.6 ms | 1.6 ms | 15 | 5 | — | 18 | 1/1 | |
| blocksworld/problem | jupyddl:gbfs/hff | solved | 1.4 ms | 1.4 ms | 1.4 ms | 2 | 2 | — | 2 | 1/1 | |
| miconic/s1-0 | jupyddl:gbfs/hff | solved | 1.8 ms | 1.8 ms | 1.8 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:gbfs/hff | solved | 1.8 ms | 1.8 ms | 1.8 ms | 3 | 3 | — | 3 | 1/1 | |
| miconic/s10-0 | jupyddl:gbfs/hff | solved | 217.7 ms | 217.7 ms | 217.7 ms | 35 | 35 | — | 43 | 1/1 | |
| miconic/s11-0 | jupyddl:gbfs/hff | solved | 287.6 ms | 287.6 ms | 287.6 ms | 39 | 39 | — | 46 | 1/1 | |
| pallet/problem | jupyddl:gbfs/hff | solved | 2.4 ms | 2.4 ms | 2.4 ms | 12 | 12 | — | 12 | 1/1 | |
| switch/problem | jupyddl:gbfs/hff | solved | 3.6 ms | 3.6 ms | 3.6 ms | 3 | 3 | — | 3 | 1/1 | |
| tsp/problem | jupyddl:gbfs/hff | solved | 1.7 ms | 1.7 ms | 1.7 ms | 15 | 5 | — | 5 | 1/1 | |
| random_obstacles/16x16/n16/d0.15 | pymapf:cbs | partial (timeout) | 1.43 s | 133.6 ms | 2.72 s | 173.5 | — | 21.5 | 488 | 2/3 | time limit (15s) reached after 4474 nodes |
| random_obstacles/16x16/n4/d0.15 | pymapf:cbs | solved | 995 µs | 945 µs | 1.9 ms | 42 | — | 15 | 1 | 3/3 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:cbs | solved | 2.3 ms | 1.2 ms | 6.9 ms | 64 | — | 14 | 2 | 3/3 | |
| random_obstacles/16x16/n16/d0.15 | pymapf:lacam | solved | 9.6 ms | 9.5 ms | 10.7 ms | 191 | — | 19 | 19 | 3/3 | |
| random_obstacles/16x16/n4/d0.15 | pymapf:lacam | solved | 2.7 ms | 2.6 ms | 3.2 ms | 44 | — | 16 | 16 | 3/3 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:lacam | solved | 4.8 ms | 4.5 ms | 5.3 ms | 64 | — | 14 | 14 | 3/3 | |
| random_obstacles/16x16/n16/d0.15 | pymapf:pibt | solved | 9.1 ms | 8.7 ms | 9.9 ms | 191 | — | 19 | 19 | 3/3 | |
| random_obstacles/16x16/n4/d0.15 | pymapf:pibt | solved | 2.4 ms | 2.3 ms | 2.7 ms | 44 | — | 16 | 16 | 3/3 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:pibt | solved | 4.4 ms | 4.1 ms | 4.6 ms | 64 | — | 14 | 14 | 3/3 | |
| random_obstacles/16x16/n16/d0.15 | pymapf:prioritized | solved | 16.9 ms | 16.5 ms | 19.2 ms | 177 | — | 19 | 16 | 3/3 | |
| random_obstacles/16x16/n4/d0.15 | pymapf:prioritized | solved | 1.4 ms | 1.3 ms | 2.2 ms | 42 | — | 15 | 4 | 3/3 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:prioritized | solved | 3.5 ms | 3.2 ms | 4.9 ms | 64 | — | 14 | 8 | 3/3 |
Median, Min and Max are over the solved repetitions of that cell only, so a partially-solved cell times the repetitions that finished and a cell that never solved shows —. A timeout row is therefore absent from every time column here: what it recorded is wall_time_s in the results file, the elapsed time at which the run was stopped — slightly past the budget, because a planner notices its own limit and unwinds — and never an estimate of how long a solution would have taken. The budget itself is the separate timeout_s column. Sorting this table reorders rows whose Samples denominators differ; the Samples column is shown so that is visible.
Classical planning — coverage
Eighteen instances spanning trivial to genuinely hard for a pure-Python planner, run over six jupyddl configurations with a 20-second per-instance budget. Read the cactus plot rather than the mean runtime: the configurations differ in how many instances they finish at all, not in how fast they finish the easy ones.
Single-sample timings. Every cell in this suite is one measurement: one seed, one repetition. Median, Min and Max are then the same number three times — a sample, not an observed range. Node counts, cost and validity are unaffected, being deterministic; the seconds should not be quoted with a spread until the suite is re-run with more seeds.
These six count rows — one per (configuration, instance, seed, repetition). The Solved column in the leaderboard below counts instances, and only those a configuration solved on every seed, so the two numbers are different measurements of different things and will not agree.
Leaderboard
| Configuration | Family | Coverage | Solved | Instances | Timeouts | Errors | Unsolved | Median time solved only | Total time solved only | Expanded solved only |
|---|---|---|---|---|---|---|---|---|---|---|
| jupyddl:gbfs/hff | classical | 89% | 16 | 18 | 0 | 1 | 1 | 2.4 ms | 1.71 s | 368 |
| jupyddl:gbfs/hadd | classical | 89% | 16 | 18 | 0 | 1 | 1 | 2.4 ms | 1.74 s | 383 |
| jupyddl:bfs | classical | 50% | 9 | 18 | 7 | 1 | 1 | 995 µs | 11.4 ms | 170 |
| jupyddl:dijkstra | classical | 50% | 9 | 18 | 7 | 1 | 1 | 1.0 ms | 12.0 ms | 201 |
| jupyddl:astar/hmax | classical | 50% | 9 | 18 | 7 | 1 | 1 | 1.2 ms | 16.7 ms | 180 |
| jupyddl:astar/lmcut | classical | 50% | 9 | 18 | 7 | 1 | 1 | 2.0 ms | 52.2 ms | 164 |
Coverage is Solved ÷ Instances, and both are columns here so the denominator is never implied. Median time, Total time and Expanded are computed over the instances a configuration actually solved, so each row has a different denominator and none of those three are comparable across rows — including when you sort by them. A configuration that solves only the easy instances posts the fastest time and the smallest node count on this table; that is an artefact of the subset it solved, not a result. Read them against Coverage and Instances, never instead of them.
Figures
Every instance
| Instance | Configuration | Outcome | Median time | Min | Max | Cost | Length | Makespan | Expanded | Samples seeds x reps | Note |
|---|---|---|---|---|---|---|---|---|---|---|---|
| blocksworld/problem | jupyddl:astar/hmax | solved | 709 µs | 709 µs | 709 µs | 2 | 2 | — | 2 | 1/1 | |
| dinner/problem | jupyddl:astar/hmax | solved | 2.1 ms | 2.1 ms | 2.1 ms | 1 | 1 | — | 1 | 1/1 | |
| flip/problem | jupyddl:astar/hmax | solved | 2.3 ms | 2.3 ms | 2.3 ms | 3 | 3 | — | 22 | 1/1 | |
| grid/problem | jupyddl:astar/hmax | error | — | — | — | — | — | — | — | 0/1 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:astar/hmax | solved | 1.0 ms | 1.0 ms | 1.0 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:astar/hmax | solved | 1.1 ms | 1.1 ms | 1.1 ms | 3 | 3 | — | 3 | 1/1 | |
| miconic/s1-2 | jupyddl:astar/hmax | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s10-0 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s10-1 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s10-2 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-0 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-1 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s12-0 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s12-1 | jupyddl:astar/hmax | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| pallet/problem | jupyddl:astar/hmax | solved | 4.4 ms | 4.4 ms | 4.4 ms | 12 | 12 | — | 127 | 1/1 | |
| switch/problem | jupyddl:astar/hmax | solved | 2.8 ms | 2.8 ms | 2.8 ms | 3 | 3 | — | 5 | 1/1 | |
| tsp/problem | jupyddl:astar/hmax | solved | 1.2 ms | 1.2 ms | 1.2 ms | 15 | 5 | — | 12 | 1/1 | |
| vehicle/problem | jupyddl:astar/hmax | unsolved | — | — | — | — | — | — | — | 0/1 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:astar/lmcut | solved | 904 µs | 904 µs | 904 µs | 2 | 2 | — | 2 | 1/1 | |
| dinner/problem | jupyddl:astar/lmcut | solved | 2.5 ms | 2.5 ms | 2.5 ms | 1 | 1 | — | 1 | 1/1 | |
| flip/problem | jupyddl:astar/lmcut | solved | 7.7 ms | 7.7 ms | 7.7 ms | 3 | 3 | — | 6 | 1/1 | |
| grid/problem | jupyddl:astar/lmcut | error | — | — | — | — | — | — | — | 0/1 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:astar/lmcut | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:astar/lmcut | solved | 1.2 ms | 1.2 ms | 1.2 ms | 3 | 3 | — | 3 | 1/1 | |
| miconic/s1-2 | jupyddl:astar/lmcut | solved | 1.2 ms | 1.2 ms | 1.2 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s10-0 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s10-1 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s10-2 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-0 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-1 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s12-0 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s12-1 | jupyddl:astar/lmcut | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| pallet/problem | jupyddl:astar/lmcut | solved | 32.2 ms | 32.2 ms | 32.2 ms | 12 | 12 | — | 127 | 1/1 | |
| switch/problem | jupyddl:astar/lmcut | solved | 3.5 ms | 3.5 ms | 3.5 ms | 3 | 3 | — | 5 | 1/1 | |
| tsp/problem | jupyddl:astar/lmcut | solved | 2.0 ms | 2.0 ms | 2.0 ms | 15 | 5 | — | 12 | 1/1 | |
| vehicle/problem | jupyddl:astar/lmcut | unsolved | — | — | — | — | — | — | — | 0/1 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:bfs | solved | 591 µs | 591 µs | 591 µs | 2 | 2 | — | 2 | 1/1 | |
| dinner/problem | jupyddl:bfs | solved | 1.9 ms | 1.9 ms | 1.9 ms | 1 | 1 | — | 1 | 1/1 | |
| flip/problem | jupyddl:bfs | solved | 898 µs | 898 µs | 898 µs | 3 | 3 | — | 9 | 1/1 | |
| grid/problem | jupyddl:bfs | error | — | — | — | — | — | — | — | 0/1 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:bfs | solved | 1.0 ms | 1.0 ms | 1.0 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:bfs | solved | 991 µs | 991 µs | 991 µs | 3 | 3 | — | 4 | 1/1 | |
| miconic/s1-2 | jupyddl:bfs | solved | 995 µs | 995 µs | 995 µs | 4 | 4 | — | 4 | 1/1 | |
| miconic/s10-0 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s10-1 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s10-2 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-0 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-1 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s12-0 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s12-1 | jupyddl:bfs | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| pallet/problem | jupyddl:bfs | solved | 1.8 ms | 1.8 ms | 1.8 ms | 12 | 12 | — | 124 | 1/1 | |
| switch/problem | jupyddl:bfs | solved | 2.3 ms | 2.3 ms | 2.3 ms | 3 | 3 | — | 4 | 1/1 | |
| tsp/problem | jupyddl:bfs | solved | 959 µs | 959 µs | 959 µs | 15 | 5 | — | 18 | 1/1 | |
| vehicle/problem | jupyddl:bfs | unsolved | — | — | — | — | — | — | — | 0/1 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:dijkstra | solved | 617 µs | 617 µs | 617 µs | 2 | 2 | — | 4 | 1/1 | |
| dinner/problem | jupyddl:dijkstra | solved | 1.9 ms | 1.9 ms | 1.9 ms | 1 | 1 | — | 2 | 1/1 | |
| flip/problem | jupyddl:dijkstra | solved | 1.1 ms | 1.1 ms | 1.1 ms | 3 | 3 | — | 27 | 1/1 | |
| grid/problem | jupyddl:dijkstra | error | — | — | — | — | — | — | — | 0/1 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:dijkstra | solved | 1.0 ms | 1.0 ms | 1.0 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:dijkstra | solved | 1.0 ms | 1.0 ms | 1.0 ms | 3 | 3 | — | 4 | 1/1 | |
| miconic/s1-2 | jupyddl:dijkstra | solved | 1.0 ms | 1.0 ms | 1.0 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s10-0 | jupyddl:dijkstra | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s10-1 | jupyddl:dijkstra | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s10-2 | jupyddl:dijkstra | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-0 | jupyddl:dijkstra | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s11-1 | jupyddl:dijkstra | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s12-0 | jupyddl:dijkstra | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| miconic/s12-1 | jupyddl:dijkstra | timeout | — | — | — | — | — | — | — | 0/1 | planner stopped on its internal time limit |
| pallet/problem | jupyddl:dijkstra | solved | 2.0 ms | 2.0 ms | 2.0 ms | 12 | 12 | — | 127 | 1/1 | |
| switch/problem | jupyddl:dijkstra | solved | 2.4 ms | 2.4 ms | 2.4 ms | 3 | 3 | — | 5 | 1/1 | |
| tsp/problem | jupyddl:dijkstra | solved | 1.0 ms | 1.0 ms | 1.0 ms | 15 | 5 | — | 24 | 1/1 | |
| vehicle/problem | jupyddl:dijkstra | unsolved | — | — | — | — | — | — | — | 0/1 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:gbfs/hadd | solved | 687 µs | 687 µs | 687 µs | 2 | 2 | — | 2 | 1/1 | |
| dinner/problem | jupyddl:gbfs/hadd | solved | 2.1 ms | 2.1 ms | 2.1 ms | 1 | 1 | — | 1 | 1/1 | |
| flip/problem | jupyddl:gbfs/hadd | solved | 1.3 ms | 1.3 ms | 1.3 ms | 3 | 3 | — | 3 | 1/1 | |
| grid/problem | jupyddl:gbfs/hadd | error | — | — | — | — | — | — | — | 0/1 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:gbfs/hadd | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:gbfs/hadd | solved | 1.1 ms | 1.1 ms | 1.1 ms | 3 | 3 | — | 3 | 1/1 | |
| miconic/s1-2 | jupyddl:gbfs/hadd | solved | 1.0 ms | 1.0 ms | 1.0 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s10-0 | jupyddl:gbfs/hadd | solved | 216.7 ms | 216.7 ms | 216.7 ms | 39 | 39 | — | 61 | 1/1 | |
| miconic/s10-1 | jupyddl:gbfs/hadd | solved | 175.9 ms | 175.9 ms | 175.9 ms | 36 | 36 | — | 42 | 1/1 | |
| miconic/s10-2 | jupyddl:gbfs/hadd | solved | 169.9 ms | 169.9 ms | 169.9 ms | 37 | 37 | — | 38 | 1/1 | |
| miconic/s11-0 | jupyddl:gbfs/hadd | solved | 232.7 ms | 232.7 ms | 232.7 ms | 42 | 42 | — | 46 | 1/1 | |
| miconic/s11-1 | jupyddl:gbfs/hadd | solved | 260.5 ms | 260.5 ms | 260.5 ms | 42 | 42 | — | 44 | 1/1 | |
| miconic/s12-0 | jupyddl:gbfs/hadd | solved | 322.4 ms | 322.4 ms | 322.4 ms | 46 | 46 | — | 59 | 1/1 | |
| miconic/s12-1 | jupyddl:gbfs/hadd | solved | 352.1 ms | 352.1 ms | 352.1 ms | 45 | 45 | — | 52 | 1/1 | |
| pallet/problem | jupyddl:gbfs/hadd | solved | 1.5 ms | 1.5 ms | 1.5 ms | 12 | 12 | — | 12 | 1/1 | |
| switch/problem | jupyddl:gbfs/hadd | solved | 2.6 ms | 2.6 ms | 2.6 ms | 3 | 3 | — | 3 | 1/1 | |
| tsp/problem | jupyddl:gbfs/hadd | solved | 1.1 ms | 1.1 ms | 1.1 ms | 15 | 5 | — | 9 | 1/1 | |
| vehicle/problem | jupyddl:gbfs/hadd | unsolved | — | — | — | — | — | — | — | 0/1 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:gbfs/hff | solved | 676 µs | 676 µs | 676 µs | 2 | 2 | — | 2 | 1/1 | |
| dinner/problem | jupyddl:gbfs/hff | solved | 2.2 ms | 2.2 ms | 2.2 ms | 1 | 1 | — | 1 | 1/1 | |
| flip/problem | jupyddl:gbfs/hff | solved | 1.4 ms | 1.4 ms | 1.4 ms | 3 | 3 | — | 3 | 1/1 | |
| grid/problem | jupyddl:gbfs/hff | error | — | — | — | — | — | — | — | 0/1 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:gbfs/hff | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s1-1 | jupyddl:gbfs/hff | solved | 1.1 ms | 1.1 ms | 1.1 ms | 3 | 3 | — | 3 | 1/1 | |
| miconic/s1-2 | jupyddl:gbfs/hff | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 1/1 | |
| miconic/s10-0 | jupyddl:gbfs/hff | solved | 175.0 ms | 175.0 ms | 175.0 ms | 35 | 35 | — | 43 | 1/1 | |
| miconic/s10-1 | jupyddl:gbfs/hff | solved | 174.5 ms | 174.5 ms | 174.5 ms | 34 | 34 | — | 37 | 1/1 | |
| miconic/s10-2 | jupyddl:gbfs/hff | solved | 173.8 ms | 173.8 ms | 173.8 ms | 34 | 34 | — | 40 | 1/1 | |
| miconic/s11-0 | jupyddl:gbfs/hff | solved | 240.2 ms | 240.2 ms | 240.2 ms | 39 | 39 | — | 46 | 1/1 | |
| miconic/s11-1 | jupyddl:gbfs/hff | solved | 259.4 ms | 259.4 ms | 259.4 ms | 37 | 37 | — | 41 | 1/1 | |
| miconic/s12-0 | jupyddl:gbfs/hff | solved | 324.3 ms | 324.3 ms | 324.3 ms | 42 | 42 | — | 63 | 1/1 | |
| miconic/s12-1 | jupyddl:gbfs/hff | solved | 348.3 ms | 348.3 ms | 348.3 ms | 44 | 44 | — | 61 | 1/1 | |
| pallet/problem | jupyddl:gbfs/hff | solved | 1.7 ms | 1.7 ms | 1.7 ms | 12 | 12 | — | 12 | 1/1 | |
| switch/problem | jupyddl:gbfs/hff | solved | 2.6 ms | 2.6 ms | 2.6 ms | 3 | 3 | — | 3 | 1/1 | |
| tsp/problem | jupyddl:gbfs/hff | solved | 1.1 ms | 1.1 ms | 1.1 ms | 15 | 5 | — | 5 | 1/1 | |
| vehicle/problem | jupyddl:gbfs/hff | unsolved | — | — | — | — | — | — | — | 0/1 | search space exhausted without finding a plan |
Median, Min and Max are over the solved repetitions of that cell only, so a partially-solved cell times the repetitions that finished and a cell that never solved shows —. A timeout row is therefore absent from every time column here: what it recorded is wall_time_s in the results file, the elapsed time at which the run was stopped — slightly past the budget, because a planner notices its own limit and unwinds — and never an estimate of how long a solution would have taken. The budget itself is the separate timeout_s column. Sorting this table reorders rows whose Samples denominators differ; the Samples column is shown so that is visible.
Classical planning — smoke
Eight hand-written instances plus the five one-passenger miconic problems, across six jupyddl configurations. Classical planning here is deterministic, so the seed does nothing and the three repetitions exist purely to give the timing a median rather than a single sample.
These six count rows — one per (configuration, instance, seed, repetition). The Solved column in the leaderboard below counts instances, and only those a configuration solved on every seed, so the two numbers are different measurements of different things and will not agree.
Leaderboard
| Configuration | Family | Coverage | Solved | Instances | Timeouts | Errors | Unsolved | Median time solved only | Total time solved only | Expanded solved only |
|---|---|---|---|---|---|---|---|---|---|---|
| jupyddl:bfs | classical | 85% | 11 | 13 | 0 | 1 | 1 | 1.0 ms | 13.6 ms | 534 |
| jupyddl:dijkstra | classical | 85% | 11 | 13 | 0 | 1 | 1 | 1.0 ms | 13.9 ms | 627 |
| jupyddl:gbfs/hadd | classical | 85% | 11 | 13 | 0 | 1 | 1 | 1.1 ms | 14.9 ms | 147 |
| jupyddl:gbfs/hff | classical | 85% | 11 | 13 | 0 | 1 | 1 | 1.1 ms | 15.0 ms | 135 |
| jupyddl:astar/hmax | classical | 85% | 11 | 13 | 0 | 1 | 1 | 1.1 ms | 18.8 ms | 564 |
| jupyddl:astar/lmcut | classical | 85% | 11 | 13 | 0 | 1 | 1 | 1.2 ms | 56.8 ms | 514 |
Coverage is Solved ÷ Instances, and both are columns here so the denominator is never implied. Median time, Total time and Expanded are computed over the instances a configuration actually solved, so each row has a different denominator and none of those three are comparable across rows — including when you sort by them. A configuration that solves only the easy instances posts the fastest time and the smallest node count on this table; that is an artefact of the subset it solved, not a result. Read them against Coverage and Instances, never instead of them.
Figures
Every instance
| Instance | Configuration | Outcome | Median time | Min | Max | Cost | Length | Makespan | Expanded | Samples seeds x reps | Note |
|---|---|---|---|---|---|---|---|---|---|---|---|
| blocksworld/problem | jupyddl:astar/hmax | solved | 682 µs | 662 µs | 685 µs | 2 | 2 | — | 2 | 3/3 | |
| dinner/problem | jupyddl:astar/hmax | solved | 2.1 ms | 2.1 ms | 2.1 ms | 1 | 1 | — | 1 | 3/3 | |
| flip/problem | jupyddl:astar/hmax | solved | 2.2 ms | 2.1 ms | 2.3 ms | 3 | 3 | — | 22 | 3/3 | |
| grid/problem | jupyddl:astar/hmax | error | — | — | — | — | — | — | — | 0/3 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:astar/hmax | solved | 1.1 ms | 1.0 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-1 | jupyddl:astar/hmax | solved | 1.1 ms | 1.1 ms | 1.1 ms | 3 | 3 | — | 3 | 3/3 | |
| miconic/s1-2 | jupyddl:astar/hmax | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-3 | jupyddl:astar/hmax | solved | 1.1 ms | 1.0 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-4 | jupyddl:astar/hmax | solved | 1.0 ms | 1.0 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| pallet/problem | jupyddl:astar/hmax | solved | 4.3 ms | 4.3 ms | 4.4 ms | 12 | 12 | — | 127 | 3/3 | |
| switch/problem | jupyddl:astar/hmax | solved | 3.0 ms | 2.7 ms | 3.3 ms | 3 | 3 | — | 5 | 3/3 | |
| tsp/problem | jupyddl:astar/hmax | solved | 1.2 ms | 1.1 ms | 1.3 ms | 15 | 5 | — | 12 | 3/3 | |
| vehicle/problem | jupyddl:astar/hmax | unsolved | — | — | — | — | — | — | — | 0/3 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:astar/lmcut | solved | 904 µs | 894 µs | 928 µs | 2 | 2 | — | 2 | 3/3 | |
| dinner/problem | jupyddl:astar/lmcut | solved | 2.4 ms | 2.3 ms | 2.4 ms | 1 | 1 | — | 1 | 3/3 | |
| flip/problem | jupyddl:astar/lmcut | solved | 7.7 ms | 7.6 ms | 7.8 ms | 3 | 3 | — | 6 | 3/3 | |
| grid/problem | jupyddl:astar/lmcut | error | — | — | — | — | — | — | — | 0/3 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:astar/lmcut | solved | 1.2 ms | 1.2 ms | 1.2 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-1 | jupyddl:astar/lmcut | solved | 1.2 ms | 1.2 ms | 1.3 ms | 3 | 3 | — | 3 | 3/3 | |
| miconic/s1-2 | jupyddl:astar/lmcut | solved | 1.2 ms | 1.1 ms | 1.2 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-3 | jupyddl:astar/lmcut | solved | 1.2 ms | 1.2 ms | 1.3 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-4 | jupyddl:astar/lmcut | solved | 1.2 ms | 1.2 ms | 1.2 ms | 4 | 4 | — | 4 | 3/3 | |
| pallet/problem | jupyddl:astar/lmcut | solved | 33.8 ms | 33.0 ms | 34.3 ms | 12 | 12 | — | 126 | 3/3 | |
| switch/problem | jupyddl:astar/lmcut | solved | 3.5 ms | 3.4 ms | 3.6 ms | 3 | 3 | — | 5 | 3/3 | |
| tsp/problem | jupyddl:astar/lmcut | solved | 2.5 ms | 2.1 ms | 2.8 ms | 15 | 5 | — | 12 | 3/3 | |
| vehicle/problem | jupyddl:astar/lmcut | unsolved | — | — | — | — | — | — | — | 0/3 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:bfs | solved | 571 µs | 565 µs | 583 µs | 2 | 2 | — | 2 | 3/3 | |
| dinner/problem | jupyddl:bfs | solved | 1.9 ms | 1.9 ms | 2.0 ms | 1 | 1 | — | 1 | 3/3 | |
| flip/problem | jupyddl:bfs | solved | 964 µs | 882 µs | 999 µs | 3 | 3 | — | 9 | 3/3 | |
| grid/problem | jupyddl:bfs | error | — | — | — | — | — | — | — | 0/3 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:bfs | solved | 997 µs | 994 µs | 1.0 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-1 | jupyddl:bfs | solved | 1.0 ms | 990 µs | 1.0 ms | 3 | 3 | — | 4 | 3/3 | |
| miconic/s1-2 | jupyddl:bfs | solved | 1.0 ms | 1.0 ms | 1.0 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-3 | jupyddl:bfs | solved | 1.1 ms | 1.0 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-4 | jupyddl:bfs | solved | 1.0 ms | 985 µs | 1.0 ms | 4 | 4 | — | 4 | 3/3 | |
| pallet/problem | jupyddl:bfs | solved | 1.8 ms | 1.8 ms | 1.9 ms | 12 | 12 | — | 124 | 3/3 | |
| switch/problem | jupyddl:bfs | solved | 2.3 ms | 2.2 ms | 2.3 ms | 3 | 3 | — | 4 | 3/3 | |
| tsp/problem | jupyddl:bfs | solved | 973 µs | 936 µs | 976 µs | 15 | 5 | — | 18 | 3/3 | |
| vehicle/problem | jupyddl:bfs | unsolved | — | — | — | — | — | — | — | 0/3 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:dijkstra | solved | 590 µs | 580 µs | 594 µs | 2 | 2 | — | 4 | 3/3 | |
| dinner/problem | jupyddl:dijkstra | solved | 1.9 ms | 1.9 ms | 1.9 ms | 1 | 1 | — | 2 | 3/3 | |
| flip/problem | jupyddl:dijkstra | solved | 1.1 ms | 1.1 ms | 1.2 ms | 3 | 3 | — | 27 | 3/3 | |
| grid/problem | jupyddl:dijkstra | error | — | — | — | — | — | — | — | 0/3 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:dijkstra | solved | 1.0 ms | 996 µs | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-1 | jupyddl:dijkstra | solved | 995 µs | 966 µs | 1.0 ms | 3 | 3 | — | 4 | 3/3 | |
| miconic/s1-2 | jupyddl:dijkstra | solved | 1.0 ms | 996 µs | 1.0 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-3 | jupyddl:dijkstra | solved | 1.0 ms | 986 µs | 1.0 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-4 | jupyddl:dijkstra | solved | 1.0 ms | 997 µs | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| pallet/problem | jupyddl:dijkstra | solved | 1.9 ms | 1.9 ms | 1.9 ms | 12 | 12 | — | 127 | 3/3 | |
| switch/problem | jupyddl:dijkstra | solved | 2.3 ms | 2.2 ms | 2.3 ms | 3 | 3 | — | 5 | 3/3 | |
| tsp/problem | jupyddl:dijkstra | solved | 1.0 ms | 985 µs | 1.3 ms | 15 | 5 | — | 24 | 3/3 | |
| vehicle/problem | jupyddl:dijkstra | unsolved | — | — | — | — | — | — | — | 0/3 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:gbfs/hadd | solved | 670 µs | 670 µs | 675 µs | 2 | 2 | — | 2 | 3/3 | |
| dinner/problem | jupyddl:gbfs/hadd | solved | 2.1 ms | 2.1 ms | 2.1 ms | 1 | 1 | — | 1 | 3/3 | |
| flip/problem | jupyddl:gbfs/hadd | solved | 1.4 ms | 1.3 ms | 1.4 ms | 3 | 3 | — | 3 | 3/3 | |
| grid/problem | jupyddl:gbfs/hadd | error | — | — | — | — | — | — | — | 0/3 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:gbfs/hadd | solved | 1.1 ms | 1.0 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-1 | jupyddl:gbfs/hadd | solved | 1.1 ms | 1.0 ms | 1.1 ms | 3 | 3 | — | 3 | 3/3 | |
| miconic/s1-2 | jupyddl:gbfs/hadd | solved | 1.1 ms | 1.1 ms | 1.2 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-3 | jupyddl:gbfs/hadd | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-4 | jupyddl:gbfs/hadd | solved | 1.0 ms | 1.0 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| pallet/problem | jupyddl:gbfs/hadd | solved | 1.5 ms | 1.4 ms | 1.5 ms | 12 | 12 | — | 12 | 3/3 | |
| switch/problem | jupyddl:gbfs/hadd | solved | 2.6 ms | 2.5 ms | 2.6 ms | 3 | 3 | — | 3 | 3/3 | |
| tsp/problem | jupyddl:gbfs/hadd | solved | 1.3 ms | 1.2 ms | 1.8 ms | 15 | 5 | — | 9 | 3/3 | |
| vehicle/problem | jupyddl:gbfs/hadd | unsolved | — | — | — | — | — | — | — | 0/3 | search space exhausted without finding a plan |
| blocksworld/problem | jupyddl:gbfs/hff | solved | 683 µs | 679 µs | 706 µs | 2 | 2 | — | 2 | 3/3 | |
| dinner/problem | jupyddl:gbfs/hff | solved | 2.1 ms | 2.1 ms | 2.2 ms | 1 | 1 | — | 1 | 3/3 | |
| flip/problem | jupyddl:gbfs/hff | solved | 1.4 ms | 1.4 ms | 1.4 ms | 3 | 3 | — | 3 | 3/3 | |
| grid/problem | jupyddl:gbfs/hff | error | — | — | — | — | — | — | — | 0/3 | jupyddl.parser.ast.UnsupportedFeatureError: nested/numeric term in atom is not supported: ['wall', 'xpos', ['+', 'ypos', '1']] |
| miconic/s1-0 | jupyddl:gbfs/hff | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-1 | jupyddl:gbfs/hff | solved | 1.1 ms | 1.1 ms | 1.1 ms | 3 | 3 | — | 3 | 3/3 | |
| miconic/s1-2 | jupyddl:gbfs/hff | solved | 1.1 ms | 1.0 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-3 | jupyddl:gbfs/hff | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| miconic/s1-4 | jupyddl:gbfs/hff | solved | 1.1 ms | 1.1 ms | 1.1 ms | 4 | 4 | — | 4 | 3/3 | |
| pallet/problem | jupyddl:gbfs/hff | solved | 1.7 ms | 1.6 ms | 1.7 ms | 12 | 12 | — | 12 | 3/3 | |
| switch/problem | jupyddl:gbfs/hff | solved | 2.7 ms | 2.6 ms | 2.7 ms | 3 | 3 | — | 3 | 3/3 | |
| tsp/problem | jupyddl:gbfs/hff | solved | 1.0 ms | 1.0 ms | 1.1 ms | 15 | 5 | — | 5 | 3/3 | |
| vehicle/problem | jupyddl:gbfs/hff | unsolved | — | — | — | — | — | — | — | 0/3 | search space exhausted without finding a plan |
Median, Min and Max are over the solved repetitions of that cell only, so a partially-solved cell times the repetitions that finished and a cell that never solved shows —. A timeout row is therefore absent from every time column here: what it recorded is wall_time_s in the results file, the elapsed time at which the run was stopped — slightly past the budget, because a planner notices its own limit and unwinds — and never an estimate of how long a solution would have taken. The budget itself is the separate timeout_s column. Sorting this table reorders rows whose Samples denominators differ; the Samples column is shown so that is visible.
MAPF — obstacle density
Sixteen agents on 20x20 grids, obstacle density from 5% to 30%, five seeds. Same solvers as the scaling suite. The generator guarantees every start and goal lies in one connected region, so a failure here is a solver failure and never an unreachable goal.
These six count rows — one per (configuration, instance, seed, repetition). The Solved column in the leaderboard below counts instances, and only those a configuration solved on every seed, so the two numbers are different measurements of different things and will not agree.
Leaderboard
| Configuration | Family | Coverage | Solved | Instances | Timeouts | Errors | Unsolved | Median time solved only | Total time solved only | Expanded solved only |
|---|---|---|---|---|---|---|---|---|---|---|
| pymapf:pibt | mapf | 100% | 6 | 6 | 0 | 0 | 0 | 13.0 ms | 78.2 ms | 874 |
| pymapf:lacam | mapf | 100% | 6 | 6 | 0 | 0 | 0 | 13.9 ms | 84.7 ms | 875 |
| pymapf:wcbs | mapf | 83% | 5 | 6 | 1 | 0 | 0 | 57.3 ms | 351.4 ms | 863 |
| pymapf:prioritized | mapf | 67% | 4 | 6 | 0 | 0 | 2 | 24.4 ms | 96.2 ms | 416 |
| pymapf:cbs | mapf | 17% | 1 | 6 | 5 | 0 | 0 | 74.7 ms | 74.7 ms | 7,616 |
Coverage is Solved ÷ Instances, and both are columns here so the denominator is never implied. Median time, Total time and Expanded are computed over the instances a configuration actually solved, so each row has a different denominator and none of those three are comparable across rows — including when you sort by them. A configuration that solves only the easy instances posts the fastest time and the smallest node count on this table; that is an artefact of the subset it solved, not a result. Read them against Coverage and Instances, never instead of them.
Figures
Every instance
| Instance | Configuration | Outcome | Median time | Min | Max | Cost | Length | Makespan | Expanded | Samples seeds x reps | Note |
|---|---|---|---|---|---|---|---|---|---|---|---|
| random_obstacles/20x20/n16/d0.05 | pymapf:cbs | partial (timeout) | 67.8 ms | 16.0 ms | 8.51 s | 182 | — | 23 | 14 | 3/5 | time limit (20s) reached after 3403 nodes |
| random_obstacles/20x20/n16/d0.1 | pymapf:cbs | solved | 74.7 ms | 30.2 ms | 118.4 ms | 216 | — | 28 | 15 | 5/5 | |
| random_obstacles/20x20/n16/d0.15 | pymapf:cbs | partial (timeout) | 159.8 ms | 77.0 ms | 16.77 s | 212 | — | 22 | 46 | 3/5 | time limit (20s) reached after 6289 nodes |
| random_obstacles/20x20/n16/d0.2 | pymapf:cbs | partial (timeout) | 82.3 ms | 56.2 ms | 108.4 ms | 230.5 | — | 22.5 | 26 | 2/5 | time limit (20s) reached after 4607 nodes |
| random_obstacles/20x20/n16/d0.25 | pymapf:cbs | partial (timeout) | 3.56 s | 3.56 s | 3.56 s | 214 | — | 26 | 712 | 1/5 | time limit (20s) reached after 4858 nodes |
| random_obstacles/20x20/n16/d0.3 | pymapf:cbs | partial (timeout) | 99.5 ms | 99.5 ms | 99.5 ms | 193 | — | 23 | 30 | 1/5 | time limit (20s) reached after 4822 nodes |
| random_obstacles/20x20/n16/d0.05 | pymapf:lacam | solved | 14.0 ms | 13.8 ms | 15.0 ms | 240 | — | 24 | 24 | 5/5 | |
| random_obstacles/20x20/n16/d0.1 | pymapf:lacam | solved | 14.5 ms | 12.9 ms | 15.1 ms | 246 | — | 28 | 28 | 5/5 | |
| random_obstacles/20x20/n16/d0.15 | pymapf:lacam | solved | 13.7 ms | 12.9 ms | 14.4 ms | 242 | — | 26 | 26 | 5/5 | |
| random_obstacles/20x20/n16/d0.2 | pymapf:lacam | solved | 16.6 ms | 15.6 ms | 17.9 ms | 237 | — | 29 | 29 | 5/5 | |
| random_obstacles/20x20/n16/d0.25 | pymapf:lacam | solved | 13.3 ms | 12.8 ms | 15.3 ms | 283 | — | 32 | 32 | 5/5 | |
| random_obstacles/20x20/n16/d0.3 | pymapf:lacam | solved | 12.5 ms | 10.5 ms | 13.9 ms | 327 | — | 33 | 33 | 5/5 | |
| random_obstacles/20x20/n16/d0.05 | pymapf:pibt | solved | 13.4 ms | 12.9 ms | 14.3 ms | 240 | — | 24 | 24 | 5/5 | |
| random_obstacles/20x20/n16/d0.1 | pymapf:pibt | solved | 13.6 ms | 13.0 ms | 13.9 ms | 246 | — | 28 | 28 | 5/5 | |
| random_obstacles/20x20/n16/d0.15 | pymapf:pibt | solved | 12.7 ms | 12.3 ms | 13.1 ms | 242 | — | 26 | 26 | 5/5 | |
| random_obstacles/20x20/n16/d0.2 | pymapf:pibt | solved | 14.7 ms | 13.6 ms | 18.6 ms | 237 | — | 29 | 29 | 5/5 | |
| random_obstacles/20x20/n16/d0.25 | pymapf:pibt | solved | 12.1 ms | 11.5 ms | 13.2 ms | 283 | — | 32 | 32 | 5/5 | |
| random_obstacles/20x20/n16/d0.3 | pymapf:pibt | solved | 11.8 ms | 10.3 ms | 12.6 ms | 327 | — | 33 | 33 | 5/5 | |
| random_obstacles/20x20/n16/d0.05 | pymapf:prioritized | solved | 23.6 ms | 18.6 ms | 28.9 ms | 214 | — | 24 | 16 | 5/5 | |
| random_obstacles/20x20/n16/d0.1 | pymapf:prioritized | solved | 25.4 ms | 19.9 ms | 39.8 ms | 231 | — | 28 | 16 | 5/5 | |
| random_obstacles/20x20/n16/d0.15 | pymapf:prioritized | solved | 22.2 ms | 21.3 ms | 23.6 ms | 220 | — | 25 | 16 | 5/5 | |
| random_obstacles/20x20/n16/d0.2 | pymapf:prioritized | solved | 25.1 ms | 22.2 ms | 31.2 ms | 212 | — | 25 | 16 | 5/5 | |
| random_obstacles/20x20/n16/d0.25 | pymapf:prioritized | partial (unsolved) | 28.4 ms | 28.0 ms | 67.3 ms | 241 | — | 26 | 16 | 3/5 | agent 'a14' has no path under the reservations of the 14 higher-priority agents |
| random_obstacles/20x20/n16/d0.3 | pymapf:prioritized | partial (unsolved) | 32.9 ms | 22.0 ms | 47.8 ms | 242 | — | 24 | 16 | 3/5 | agent 'a9' has no path under the reservations of the 9 higher-priority agents |
| random_obstacles/20x20/n16/d0.05 | pymapf:wcbs | solved | 43.1 ms | 16.3 ms | 120.3 ms | 220 | — | 24 | 10 | 5/5 | |
| random_obstacles/20x20/n16/d0.1 | pymapf:wcbs | solved | 54.0 ms | 29.5 ms | 70.9 ms | 216 | — | 28 | 9 | 5/5 | |
| random_obstacles/20x20/n16/d0.15 | pymapf:wcbs | solved | 61.2 ms | 50.5 ms | 152.0 ms | 215 | — | 24 | 16 | 5/5 | |
| random_obstacles/20x20/n16/d0.2 | pymapf:wcbs | solved | 57.3 ms | 33.8 ms | 195.8 ms | 226 | — | 25 | 13 | 5/5 | |
| random_obstacles/20x20/n16/d0.25 | pymapf:wcbs | solved | 135.7 ms | 65.2 ms | 377.0 ms | 242 | — | 29 | 26 | 5/5 | |
| random_obstacles/20x20/n16/d0.3 | pymapf:wcbs | partial (timeout) | 615.9 ms | 34.2 ms | 5.00 s | 258 | — | 29 | 65.5 | 4/5 | time limit (20s) reached after 1286 nodes |
Median, Min and Max are over the solved repetitions of that cell only, so a partially-solved cell times the repetitions that finished and a cell that never solved shows —. A timeout row is therefore absent from every time column here: what it recorded is wall_time_s in the results file, the elapsed time at which the run was stopped — slightly past the budget, because a planner notices its own limit and unwinds — and never an estimate of how long a solution would have taken. The budget itself is the separate timeout_s column. Sorting this table reorders rows whose Samples denominators differ; the Samples column is shown so that is visible.
MAPF — scaling with agent count
Random-obstacle grids at 15% density, from 4 to 32 agents, over five seeds. Five pymapf solvers and both cuplan CPU solvers on the same generated instances. CBS and weighted CBS are optimal and bounded-suboptimal respectively, on sum of costs. LaCAM (Okumura, 2023) is complete but not optimal; PIBT (Okumura et al., 2022) is neither; prioritized planning is neither. Completeness is a statement about eventual termination, not about finishing inside this suite's 20-second budget. Expect the optimal solvers to fall off first — that is the result, not a failure of the run.
These six count rows — one per (configuration, instance, seed, repetition). The Solved column in the leaderboard below counts instances, and only those a configuration solved on every seed, so the two numbers are different measurements of different things and will not agree.
Leaderboard
| Configuration | Family | Coverage | Solved | Instances | Timeouts | Errors | Unsolved | Median time solved only | Total time solved only | Expanded solved only |
|---|---|---|---|---|---|---|---|---|---|---|
| pymapf:lacam | mapf | 100% | 10 | 10 | 0 | 0 | 0 | 11.9 ms | 149.3 ms | 1,251 |
| pymapf:wcbs | mapf | 100% | 10 | 10 | 0 | 0 | 0 | 50.0 ms | 1.86 s | 2,052 |
| pymapf:pibt | mapf | 90% | 9 | 10 | 0 | 0 | 1 | 9.0 ms | 121.0 ms | 1,199 |
| cuplan:prioritized@cpu | mapf | 80% | 8 | 10 | 0 | 0 | 2 | 14.4 ms | 160.6 ms | — |
| pymapf:prioritized | mapf | 80% | 8 | 10 | 0 | 0 | 2 | 10.0 ms | 266.5 ms | 720 |
| cuplan:pibt@cpu | mapf | 70% | 7 | 10 | 0 | 0 | 3 | 11.5 ms | 83.4 ms | — |
| pymapf:cbs | mapf | 50% | 5 | 10 | 5 | 0 | 0 | 3.9 ms | 90.4 ms | 4,012 |
| cuplan:pibt@cuda | mapf | not run | — | — | — | — | — | — | — | — |
Coverage is Solved ÷ Instances, and both are columns here so the denominator is never implied. Median time, Total time and Expanded are computed over the instances a configuration actually solved, so each row has a different denominator and none of those three are comparable across rows — including when you sort by them. A configuration that solves only the easy instances posts the fastest time and the smallest node count on this table; that is an artefact of the subset it solved, not a result. Read them against Coverage and Instances, never instead of them.
Figures
Every instance
| Instance | Configuration | Outcome | Median time | Min | Max | Cost | Length | Makespan | Expanded | Samples seeds x reps | Note |
|---|---|---|---|---|---|---|---|---|---|---|---|
| random_obstacles/16x16/n16/d0.15 | cuplan:pibt@cpu | partial (unsolved) | 11.3 ms | 10.6 ms | 11.6 ms | 207.5 | — | 22 | — | 4/5 | backend=cpu |
| random_obstacles/16x16/n24/d0.15 | cuplan:pibt@cpu | solved | 14.4 ms | 11.9 ms | 18.2 ms | 332 | — | 26 | — | 5/5 | backend=cpu |
| random_obstacles/16x16/n32/d0.15 | cuplan:pibt@cpu | partial (unsolved) | 14.3 ms | 12.3 ms | 17.9 ms | 460 | — | 23.5 | — | 4/5 | backend=cpu |
| random_obstacles/16x16/n4/d0.15 | cuplan:pibt@cpu | solved | 9.0 ms | 8.5 ms | 9.4 ms | 44 | — | 16 | — | 5/5 | backend=cpu |
| random_obstacles/16x16/n8/d0.15 | cuplan:pibt@cpu | solved | 9.4 ms | 9.0 ms | 10.7 ms | 70 | — | 16 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n16/d0.15 | cuplan:pibt@cpu | solved | 13.0 ms | 12.5 ms | 19.5 ms | 318 | — | 30 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n24/d0.15 | cuplan:pibt@cpu | solved | 14.8 ms | 14.1 ms | 23.8 ms | 420 | — | 31 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n32/d0.15 | cuplan:pibt@cpu | partial (unsolved) | 17.8 ms | 16.5 ms | 25.8 ms | 589.5 | — | 30.5 | — | 4/5 | backend=cpu |
| random_obstacles/24x24/n4/d0.15 | cuplan:pibt@cpu | solved | 11.3 ms | 9.8 ms | 16.8 ms | 81 | — | 24 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n8/d0.15 | cuplan:pibt@cpu | solved | 11.5 ms | 11.2 ms | 11.9 ms | 128 | — | 25 | — | 5/5 | backend=cpu |
| random_obstacles/16x16/n16/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/16x16/n24/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/16x16/n32/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/16x16/n4/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/16x16/n8/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/24x24/n16/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/24x24/n24/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/24x24/n32/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/24x24/n4/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/24x24/n8/d0.15 | cuplan:pibt@cuda | skipped | — | — | — | — | — | — | — | 0/5 | no working CUDA device (cuplan.cuda_available() returned False) |
| random_obstacles/16x16/n16/d0.15 | cuplan:prioritized@cpu | solved | 15.7 ms | 13.8 ms | 16.7 ms | 180 | — | 19 | — | 5/5 | backend=cpu |
| random_obstacles/16x16/n24/d0.15 | cuplan:prioritized@cpu | partial (unsolved) | 23.7 ms | 21.6 ms | 28.5 ms | 274.5 | — | 20.5 | — | 4/5 | backend=cpu |
| random_obstacles/16x16/n32/d0.15 | cuplan:prioritized@cpu | partial (unsolved) | 30.8 ms | 28.6 ms | 35.0 ms | 360 | — | 21 | — | 3/5 | backend=cpu |
| random_obstacles/16x16/n4/d0.15 | cuplan:prioritized@cpu | solved | 5.7 ms | 4.2 ms | 9.0 ms | 42 | — | 15 | — | 5/5 | backend=cpu |
| random_obstacles/16x16/n8/d0.15 | cuplan:prioritized@cpu | solved | 6.3 ms | 6.1 ms | 12.6 ms | 64 | — | 16 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n16/d0.15 | cuplan:prioritized@cpu | solved | 27.9 ms | 24.5 ms | 45.2 ms | 283 | — | 30 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n24/d0.15 | cuplan:prioritized@cpu | solved | 34.9 ms | 33.7 ms | 40.0 ms | 377 | — | 31 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n32/d0.15 | cuplan:prioritized@cpu | solved | 48.9 ms | 48.0 ms | 65.0 ms | 534 | — | 29 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n4/d0.15 | cuplan:prioritized@cpu | solved | 8.2 ms | 5.0 ms | 10.4 ms | 77 | — | 26 | — | 5/5 | backend=cpu |
| random_obstacles/24x24/n8/d0.15 | cuplan:prioritized@cpu | solved | 13.1 ms | 11.0 ms | 15.6 ms | 129 | — | 26 | — | 5/5 | backend=cpu |
| random_obstacles/16x16/n16/d0.15 | pymapf:cbs | partial (timeout) | 376.8 ms | 127.2 ms | 2.50 s | 171.5 | — | 20 | 136.5 | 4/5 | time limit (20s) reached after 6296 nodes |
| random_obstacles/16x16/n24/d0.15 | pymapf:cbs | partial (timeout) | 9.59 s | 9.59 s | 9.59 s | 261 | — | 22 | 1,697 | 1/5 | time limit (20s) reached after 4914 nodes |
| random_obstacles/16x16/n32/d0.15 | pymapf:cbs | timeout | — | — | — | — | — | — | — | 0/5 | time limit (20s) reached after 1884 nodes |
| random_obstacles/16x16/n4/d0.15 | pymapf:cbs | solved | 1.4 ms | 799 µs | 2.1 ms | 42 | — | 15 | 1 | 5/5 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:cbs | solved | 3.9 ms | 1.1 ms | 5.9 ms | 64 | — | 16 | 5 | 5/5 | |
| random_obstacles/24x24/n16/d0.15 | pymapf:cbs | solved | 76.6 ms | 36.3 ms | 1.39 s | 275 | — | 30 | 24 | 5/5 | |
| random_obstacles/24x24/n24/d0.15 | pymapf:cbs | partial (timeout) | 1.78 s | 826.7 ms | 2.73 s | 351 | — | 28 | 134 | 2/5 | time limit (20s) reached after 2508 nodes |
| random_obstacles/24x24/n32/d0.15 | pymapf:cbs | partial (timeout) | 5.68 s | 5.68 s | 5.68 s | 490 | — | 29 | 265 | 1/5 | time limit (20s) reached after 1657 nodes |
| random_obstacles/24x24/n4/d0.15 | pymapf:cbs | solved | 3.0 ms | 702 µs | 18.9 ms | 77 | — | 26 | 1 | 5/5 | |
| random_obstacles/24x24/n8/d0.15 | pymapf:cbs | solved | 5.5 ms | 3.1 ms | 366.6 ms | 128 | — | 25 | 3 | 5/5 | |
| random_obstacles/16x16/n16/d0.15 | pymapf:lacam | solved | 8.7 ms | 7.7 ms | 9.4 ms | 190 | — | 19 | 19 | 5/5 | |
| random_obstacles/16x16/n24/d0.15 | pymapf:lacam | solved | 14.2 ms | 12.8 ms | 14.9 ms | 335 | — | 23 | 23 | 5/5 | |
| random_obstacles/16x16/n32/d0.15 | pymapf:lacam | solved | 18.5 ms | 16.7 ms | 21.8 ms | 520 | — | 29 | 29 | 5/5 | |
| random_obstacles/16x16/n4/d0.15 | pymapf:lacam | solved | 2.2 ms | 2.1 ms | 2.6 ms | 44 | — | 16 | 16 | 5/5 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:lacam | solved | 4.1 ms | 3.7 ms | 4.4 ms | 64 | — | 16 | 16 | 5/5 | |
| random_obstacles/24x24/n16/d0.15 | pymapf:lacam | solved | 18.9 ms | 18.0 ms | 19.7 ms | 286 | — | 32 | 32 | 5/5 | |
| random_obstacles/24x24/n24/d0.15 | pymapf:lacam | solved | 27.7 ms | 25.7 ms | 29.3 ms | 405 | — | 31 | 31 | 5/5 | |
| random_obstacles/24x24/n32/d0.15 | pymapf:lacam | solved | 40.7 ms | 39.5 ms | 42.3 ms | 590 | — | 31 | 31 | 5/5 | |
| random_obstacles/24x24/n4/d0.15 | pymapf:lacam | solved | 4.8 ms | 4.5 ms | 5.3 ms | 86 | — | 24 | 24 | 5/5 | |
| random_obstacles/24x24/n8/d0.15 | pymapf:lacam | solved | 9.5 ms | 9.1 ms | 9.9 ms | 128 | — | 27 | 27 | 5/5 | |
| random_obstacles/16x16/n16/d0.15 | pymapf:pibt | solved | 7.8 ms | 7.4 ms | 8.5 ms | 190 | — | 19 | 19 | 5/5 | |
| random_obstacles/16x16/n24/d0.15 | pymapf:pibt | solved | 12.9 ms | 12.2 ms | 14.0 ms | 335 | — | 23 | 23 | 5/5 | |
| random_obstacles/16x16/n32/d0.15 | pymapf:pibt | partial (unsolved) | 16.8 ms | 15.9 ms | 17.7 ms | 492.5 | — | 27.5 | 27.5 | 4/5 | livelock: 301 timesteps without reaching the goals |
| random_obstacles/16x16/n4/d0.15 | pymapf:pibt | solved | 1.9 ms | 1.9 ms | 2.3 ms | 44 | — | 16 | 16 | 5/5 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:pibt | solved | 3.7 ms | 3.4 ms | 4.1 ms | 64 | — | 16 | 16 | 5/5 | |
| random_obstacles/24x24/n16/d0.15 | pymapf:pibt | solved | 17.9 ms | 17.7 ms | 18.5 ms | 286 | — | 32 | 32 | 5/5 | |
| random_obstacles/24x24/n24/d0.15 | pymapf:pibt | solved | 27.1 ms | 25.5 ms | 27.4 ms | 405 | — | 31 | 31 | 5/5 | |
| random_obstacles/24x24/n32/d0.15 | pymapf:pibt | solved | 36.1 ms | 34.4 ms | 38.2 ms | 590 | — | 31 | 31 | 5/5 | |
| random_obstacles/24x24/n4/d0.15 | pymapf:pibt | solved | 4.5 ms | 4.3 ms | 5.0 ms | 86 | — | 24 | 24 | 5/5 | |
| random_obstacles/24x24/n8/d0.15 | pymapf:pibt | solved | 9.0 ms | 8.7 ms | 9.2 ms | 128 | — | 27 | 27 | 5/5 | |
| random_obstacles/16x16/n16/d0.15 | pymapf:prioritized | solved | 12.4 ms | 10.6 ms | 14.2 ms | 170 | — | 19 | 16 | 5/5 | |
| random_obstacles/16x16/n24/d0.15 | pymapf:prioritized | partial (unsolved) | 37.9 ms | 35.0 ms | 42.0 ms | 273 | — | 20.5 | 24 | 4/5 | agent 'a9' has no path under the reservations of the 9 higher-priority agents |
| random_obstacles/16x16/n32/d0.15 | pymapf:prioritized | partial (unsolved) | 70.9 ms | 68.1 ms | 73.7 ms | 369 | — | 21 | 32 | 2/5 | agent 'a28' has no path under the reservations of the 28 higher-priority agents |
| random_obstacles/16x16/n4/d0.15 | pymapf:prioritized | solved | 1.3 ms | 998 µs | 1.9 ms | 42 | — | 15 | 4 | 5/5 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:prioritized | solved | 2.6 ms | 2.4 ms | 3.8 ms | 70 | — | 16 | 8 | 5/5 | |
| random_obstacles/24x24/n16/d0.15 | pymapf:prioritized | solved | 30.3 ms | 24.3 ms | 35.9 ms | 275 | — | 30 | 16 | 5/5 | |
| random_obstacles/24x24/n24/d0.15 | pymapf:prioritized | solved | 69.1 ms | 64.4 ms | 97.8 ms | 379 | — | 31 | 24 | 5/5 | |
| random_obstacles/24x24/n32/d0.15 | pymapf:prioritized | solved | 139.4 ms | 128.5 ms | 147.9 ms | 511 | — | 29 | 32 | 5/5 | |
| random_obstacles/24x24/n4/d0.15 | pymapf:prioritized | solved | 3.7 ms | 1.3 ms | 4.5 ms | 77 | — | 26 | 4 | 5/5 | |
| random_obstacles/24x24/n8/d0.15 | pymapf:prioritized | solved | 7.7 ms | 6.7 ms | 9.8 ms | 128 | — | 26 | 8 | 5/5 | |
| random_obstacles/16x16/n16/d0.15 | pymapf:wcbs | solved | 47.0 ms | 27.8 ms | 70.3 ms | 170 | — | 19 | 15 | 5/5 | |
| random_obstacles/16x16/n24/d0.15 | pymapf:wcbs | solved | 199.4 ms | 139.8 ms | 438.6 ms | 275 | — | 21 | 42 | 5/5 | |
| random_obstacles/16x16/n32/d0.15 | pymapf:wcbs | solved | 831.1 ms | 729.7 ms | 10.46 s | 372 | — | 23 | 103 | 5/5 | |
| random_obstacles/16x16/n4/d0.15 | pymapf:wcbs | solved | 1.3 ms | 783 µs | 3.6 ms | 42 | — | 15 | 1 | 5/5 | |
| random_obstacles/16x16/n8/d0.15 | pymapf:wcbs | solved | 2.2 ms | 1.1 ms | 6.0 ms | 64 | — | 16 | 2 | 5/5 | |
| random_obstacles/24x24/n16/d0.15 | pymapf:wcbs | solved | 52.9 ms | 36.5 ms | 91.4 ms | 275 | — | 30 | 9 | 5/5 | |
| random_obstacles/24x24/n24/d0.15 | pymapf:wcbs | solved | 176.6 ms | 122.8 ms | 341.6 ms | 361 | — | 31 | 21 | 5/5 | |
| random_obstacles/24x24/n32/d0.15 | pymapf:wcbs | solved | 545.3 ms | 452.2 ms | 992.2 ms | 498 | — | 29 | 38 | 5/5 | |
| random_obstacles/24x24/n4/d0.15 | pymapf:wcbs | solved | 3.1 ms | 729 µs | 18.8 ms | 77 | — | 26 | 1 | 5/5 | |
| random_obstacles/24x24/n8/d0.15 | pymapf:wcbs | solved | 5.4 ms | 3.2 ms | 18.3 ms | 128 | — | 26 | 3 | 5/5 |
Median, Min and Max are over the solved repetitions of that cell only, so a partially-solved cell times the repetitions that finished and a cell that never solved shows —. A timeout row is therefore absent from every time column here: what it recorded is wall_time_s in the results file, the elapsed time at which the run was stopped — slightly past the budget, because a planner notices its own limit and unwinds — and never an estimate of how long a solution would have taken. The budget itself is the separate timeout_s column. Sorting this table reorders rows whose Samples denominators differ; the Samples column is shown so that is visible.