Agentic Reasoning Traces

If you want to evaluate whether an agent reasons well, you need examples of reasoning labelled at the level of individual steps — not just final answers marked right or wrong. An agent can reach the correct answer through a chain that is mostly luck, and a step-level view is the only thing that shows it.

So I built the dataset by hand: 15 traces, 142 individual steps, each classified under a nine-type taxonomy covering things like planning, retrieval, verification and revision. Five traces are autobiographical, recording how I actually worked through a problem. The other ten are synthetic.

Three of the traces carry deliberate errors. That was the part I cared most about: a dataset where every trace is clean tells you nothing about whether your auditing process can catch a bad step. Seeding known failures gives you a control — if the evaluation misses those, it is not measuring what you think it is measuring.

The habit it taught me — plant known failures so you can tell whether your instrument detects them — is the same instinct that later caught twenty-one bugs in the world models benchmark.