Research note
Test the plan, not the confidence of the answer
Turn an impressive reasoning demonstration into a task with explicit states, constraints, perturbations and a checkable result.
THE SHORT ANSWER
A valid plan must satisfy the task’s constraints, not merely explain itself convincingly.
Why it matters
An explanation can sound coherent while containing a step that cannot be executed. To evaluate planning, make success independent of the model’s own description. Define the starting state, permitted actions and desired final state before looking at a result.
Read the evidence
The September 2024 PlanBench study examines planning performance and raises questions about accuracy, efficiency and guarantees. It is a study of particular models and tasks at that time, not evidence for a current universal limit on AI reasoning. Its useful contribution here is the insistence on checking the plan against a specified problem.
How to use this resource
Proposed exercise: ask an assistant to schedule three fictional jobs sharing one machine. Define durations and dependencies yourself, and use a small independent checker or a manually verified table.
- Write the correct final conditions before generating an answer.
- Check every proposed action against the current state and permitted transitions.
- Repeat with renamed jobs and a different order of the same facts.
- Include an impossible case and require the system to say that no valid schedule exists.
- Keep model version, settings, time allowance and all failures with the result.
Limits and evidence
The scheduling exercise is an editorial proposal and has not been executed. A small task cannot establish general intelligence, and success on one wording may not transfer. This note does not rank contemporary models or report a new benchmark.
Put it to work.
- Specify a starting state and a goal.
- Use a check independent of the answer.
- Include a controlled variation.
- Retain failures and the task budget.
Go to the source.
Original material, with its own scope and publication date. This note is an explanation, not a CooScope performance test.
- Valmeekam et al. — Preliminary evaluation of o1 on PlanBench (20 September 2024)Link / source review: 2026-10-04