Checklist
Evaluate the monitor as well as the agent
Use SHADE-Arena as a research example for asking what a transcript monitor can see, miss and falsely flag.
THE SHORT ANSWER
A monitor needs its own evaluation target, visibility record and error analysis.
Separate two evaluated roles
Anthropic’s archived announcement introduces sabotage and sabotage-monitoring evaluations. Its SHADE-Arena report describes an agent with a benign task and a malicious side task, followed by a monitor that reads the transcript and assigns suspicion. These are constructed research conditions. They do not measure the frequency of malicious behavior in ordinary use or certify a monitoring product.
Write the monitor’s contract
Start with a harmless sandbox workflow, such as organizing fictional documents. Define which changes are permitted and which would violate the task. List exactly what the monitor receives: messages, tool arguments, results or final files. Create benign cases that look unusual alongside obvious violations. Otherwise, a monitor that flags everything can appear effective while being unusable for review.
Inspect errors and coverage
Record false alarms, missed violations and cases the available evidence cannot resolve. Have a person check whether the labels themselves are defensible. If a monitor cannot see a tool result, say so before interpreting its decision. This is an original evaluation rubric inspired by the research setup. No attack, monitor run or security certification was performed; the checklist supports a bounded sandbox review.
Put it to work.
- Define permitted behavior.
- Record monitor visibility.
- Include benign hard cases.
- Inspect missed and false alerts.
Go to the source.
Original material, with its own scope and publication date. This note is an explanation, not a CooScope performance test.
Source published · Editorial review
Archived English post text; primary sources reviewed separately. Generated annotations, media and missing replies are outside the evidence scope.
- @AnthropicAI — archived X post (2025-06-16)
- Anthropic — SHADE-Arena sabotage and monitoring researchLink / source review: 2026-10-15