Workflow
Turn an expert’s model anecdote into a test pack
Preserve the question behind an early-access impression, then replace the implied ranking with a small, repeatable task record.
THE SHORT ANSWER
A useful anecdote gives you candidate tasks; your own acceptance rules determine what the answers show.
Read the author’s caveat
Karpathy’s archived Grok 3 post describes a brief early-access exploration and explicitly leaves room for later evaluation. It mixes coding, lookup and puzzle observations. That makes it a source of candidate questions, not a current leaderboard. The original Arena paper offers a different evidence type: anonymous pairwise human preference, rather than one expert’s informal exploration.
Keep the interesting failure
Select one task that resembles work you actually do. Write the expected behavior before running anything: for a generated grid, check tile count, numbering and slider behavior; for lookup, check whether the cited passage supports the answer. Keep the prompt, model identifier, settings and unedited output together. Use new variants so a familiar demonstration does not become the whole test.
Report a bounded finding
Repeat the chosen task within a fixed budget and keep failed attempts. Separate correctness, usability and preference in the record. A pleasing interface may still contain the wrong tile count. If you later share a result, state the task set and conditions rather than borrowing the author’s broader impression. This workflow has not been run here; its value is making a future comparison inspectable.
Put it to work.
- Choose one relevant task.
- Write expected behavior first.
- Retain every attempt.
- Separate preference from correctness.
Go to the source.
Original material, with its own scope and publication date. This note is an explanation, not a CooScope performance test.
Source published · Editorial review
Archived English post text; primary sources reviewed separately. Generated annotations, media and missing replies are outside the evidence scope.
- @karpathy — archived X post (2025-02-18)
- Chiang et al. — Chatbot Arena, original paperLink / source review: 2026-10-15