We give an AI assistant a work conversation and ZeroWidth's tools, then check what it records. A good assistant records each decision that was made, once and with the right details, and records nothing when nothing was decided.
- Store manager
I have a three-step plan to get more visitors who buy, and I'll watch the conversion rate. Step one is reshoot every product on a white background. Step two is rewrite the product descriptions. Step three is add customer videos.
I'm only agreeing to step one: reshoot every product on a white background.
Steps two and three are not decided, so leave them as ideas for now.
Right
Recorded
Reshoot every product on a white background
- Goal
- More visitors who buy
- Metric
- Conversion rate, up
Only the step that was agreed, recorded once.
Wrong
Recorded
Reshoot every product on a white background, rewrite the product descriptions and add customer videos
Records two steps nobody agreed to.
Each model runs every conversation in two setups:
Tools only The assistant has ZeroWidth's tools for recording and looking up decisions, and nothing else from us.
Tools and instructions The assistant also reads our short instructions on when to record a decision and what to include.
The models: Claude Sonnet 5.5, Claude Haiku 5.5, GPT-6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash.
An answer is right when the assistant records the decision once, or records nothing when nothing was decided. Recording something that wasn't decided is a false positive. There are eight kinds of conversation:
A decision made The person decides something. Record the decision.
A decision reversed The person undoes an earlier decision. Find the earlier one, then record the change.
Someone else decided The person passes on a colleague's decision. Record the decision, with the colleague as the one who decided.
A plan partly agreed Only the first step is agreed. Record that step only.
A choice left for later "We'll decide after the board meeting." Record nothing. A note that the choice was left for later is allowed.
A choice handed to someone else "Sam will pick the vendor." Record nothing about the choice.
Not a decision An idea talked through and never agreed. Record nothing.
Off the record The person asks to keep a decision out. Record nothing.
In the pilot, every model recorded every decision it should have. Where the models differed was the other half of the job: recording things that weren't decisions. Our instructions cut those sharply.
Right answers
83%[71–90]with tools only
96%[89–99]with tools and instructions
Both setups ran the same conversations (p = 0.02)
If a note that a choice was left for later counts as wrong: 71% → 89%
We settled some scoring rules for these runs after seeing the results. From the first full run, the conversations and rules are fixed before any model runs.
Results by model
| Model | Tools only | Tools and instructions | Change |
|---|---|---|---|
| Claude Haiku 5.5 | 81% [57–93]13 of 16 | 100% [81–100]16 of 16 | ↑ +19 points |
| Gemini 3.8 Flash | 80% [49–94]8 of 10 | 100% [74–100]11 of 11 | ↑ +20 points |
| DeepSeek V4.1 Flash | 88% [53–98]7 of 8 | 100% [80–100]15 of 15 | ↑ +13 points |
| GPT-6 Luna | 85% [58–96]11 of 13 | 94% [72–99]15 of 16 | ↑ +9 points |
| Claude Sonnet 5.5 | 81% [57–93]13 of 16 | 88% [64–97]14 of 16 | ↑ +6 points |
From the pilot, with the default setup
Each measure
Tools onlyTools and instructions
Share of conversations
Tools onlyTools and short instructionsTools and instructions
Share of conversations
Tools onlyTools and instructions
Share of conversations
Tools onlyTools and short instructionsTools and instructions
Share of conversations
Tools onlyTools and instructions
Share of conversations
Tools onlyTools and short instructionsTools and instructions
Share of conversations
Tools onlyTools and instructions
Share of conversations
Tools onlyTools and instructions
Share of conversations
Tools onlyTools and instructions
Share of conversations
Tools onlyTools and short instructionsTools and instructions
Share of conversations
Tools onlyTools and instructions
Share of records
Tools onlyTools and short instructionsTools and instructions
Share of records
Tools onlyTools and instructions
Correct details
Tools onlyTools and short instructionsTools and instructions
Correct details
Tools onlyTools and instructions
Tokens
Tools onlyTools and short instructionsTools and instructions
Tokens
Tools onlyTools and instructions
US dollars
Tools onlyTools and short instructionsTools and instructions
US dollars
Tools onlyTools and instructions
US dollars
Tools onlyTools and short instructionsTools and instructions
US dollars
By kind of conversation
| Said outright | Over several turns | |||
|---|---|---|---|---|
| Kind | Tools only | Tools and instructions | Tools only | Tools and instructions |
| A decision made | 4 of 4 | 5 of 5 | 4 of 4 | 3 of 3 |
| A decision reversed | 5 of 5 | 5 of 5 | 4 of 4 | 5 of 5 |
| Someone else decided | 5 of 5 | 5 of 5 | 4 of 4 | 5 of 5 |
| A plan partly agreed | 0 of 4 | 5 of 5 | 1 of 5 | 4 of 5 |
| A choice left for later | 5 of 5 | 5 of 5 | 5 of 5 | 5 of 5 |
| A choice handed to someone else | 2 of 2 | 4 of 4 | 2 of 3 | 2 of 4 |
| Not a decision | 2 of 2 | 5 of 5 | 5 of 5 | 5 of 5 |
| Off the record | 0 of 2 | 4 of 4 | 4 of 4 | 4 of 4 |
Model by model
Claude Sonnet 5.5
Anthropic, added October 31, 2026
| Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run | |
|---|---|---|---|---|---|
| Right answers | 81% | 88% | 75% | 88% | 100% |
| Strict right answers | 75% | 81% | 75% | 88% | 100% |
| Decisions the assistant recorded | 100% | 100% | 100% | 100% | 100% |
| False positives | 25% | 0% | |||
| Recorded a decision the person asked to keep out | 50% | 0% | |||
| Recorded the same decision twice | 75% | 63% | 0% | 13% | 0% |
| Records with a number nobody said | 0% | 0% | 0% | 0% | 0% |
| Correct details per record | 4.00 | 3.82 | 4.75 | 5.11 | 5.63 |
| Tokens ZeroWidth adds per conversation | 16.6K | 35.7K | 24.9K | 31.2K | 53.2K |
| Cost per conversation | $0.10 | $0.11 | $0.12 | $0.11 | $0.14 |
| Cost per recorded decision | $0.21 | $0.22 | $0.12 | $0.11 | $0.14 |
| Detail | Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run |
|---|---|---|---|---|---|
| What was decided | 1.00 18 records | 1.00 15 records | 1.00 8 records | 1.00 9 records | 1.00 8 records |
| Why | 0.82 11 records | 1.00 7 records | 0.43 7 records | 1.00 5 records | 1.00 5 records |
| Goal | 1.00 10 records | 1.00 9 records | 1.00 5 records | 1.00 7 records | 1.00 6 records |
| Metric | 1.00 7 records | 1.00 6 records | 1.00 4 records | 1.00 5 records | 1.00 6 records |
| Which way is good | 1.00 10 records | 1.00 6 records | 1.00 5 records | 1.00 7 records | 1.00 6 records |
| When to check | 1.00 4 records | 1.00 3 records | 1.00 4 records | 1.00 2 records | 1.00 4 records |
| Who decided | 1.00 4 records | 1.00 2 records | 1.00 2 records | 1.00 2 records | 1.00 2 records |
Claude Haiku 5.5
Anthropic, added October 31, 2026
| Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run | |
|---|---|---|---|---|---|
| Right answers | 81% | 100% | 75% | 88% | 88% |
| Strict right answers | 75% | 94% | 75% | 88% | 88% |
| Decisions the assistant recorded | 100% | 100% | 100% | 100% | 100% |
| False positives | 25% | 0% | |||
| Recorded a decision the person asked to keep out | 50% | 0% | |||
| Recorded the same decision twice | 100% | 38% | 13% | 0% | 13% |
| Records with a number nobody said | 0% | 0% | 0% | 0% | 0% |
| Correct details per record | 3.27 | 3.42 | 4.78 | 5.00 | 5.00 |
| Tokens ZeroWidth adds per conversation | 18.7K | 35.6K | 22.8K | 28.0K | 48.0K |
| Cost per conversation | $0.005 | $0.005 | $0.006 | $0.005 | $0.007 |
| Cost per recorded decision | $0.010 | $0.011 | $0.006 | $0.005 | $0.007 |
| Detail | Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run |
|---|---|---|---|---|---|
| What was decided | 1.00 20 records | 1.00 11 records | 1.00 9 records | 1.00 8 records | 1.00 9 records |
| Why | 0.50 12 records | 0.67 6 records | 0.60 5 records | 1.00 5 records | 0.67 6 records |
| Goal | 0.82 11 records | 1.00 5 records | 1.00 6 records | 1.00 5 records | 1.00 7 records |
| Metric | 1.00 7 records | 1.00 3 records | 1.00 5 records | 1.00 4 records | 0.71 7 records |
| Which way is good | 1.00 10 records | 1.00 4 records | 1.00 6 records | 1.00 5 records | 0.71 7 records |
| When to check | 1.00 4 records | 1.00 4 records | 1.00 3 records | 1.00 4 records | |
| Who decided | 1.00 4 records | 1.00 2 records | 1.00 2 records | 1.00 2 records | 1.00 2 records |
Gemini 3.8 Flash
Google, added October 31, 2026
| Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run | |
|---|---|---|---|---|---|
| Right answers | 80% | 100% | 71% | 86% | 100% |
| Strict right answers | 60% | 91% | 71% | 86% | 100% |
| Decisions the assistant recorded | 100% | 100% | 100% | 100% | 100% |
| False positives | 0% | 0% | |||
| Recorded the same decision twice | 86% | 43% | 14% | 14% | 0% |
| Records with a number nobody said | 0% | 0% | 0% | 0% | 0% |
| Correct details per record | 3.55 | 3.55 | 5.00 | 4.63 | 4.00 |
| Tokens ZeroWidth adds per conversation | 24.9K | 46.0K | 29.5K | 34.0K | 47.9K |
| Cost per conversation | $0.020 | $0.033 | $0.026 | $0.030 | $0.029 |
| Cost per recorded decision | $0.046 | $0.075 | $0.030 | $0.034 | $0.039 |
| Detail | Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run |
|---|---|---|---|---|---|
| What was decided | 1.00 18 records | 1.00 10 records | 1.00 8 records | 1.00 8 records | 1.00 6 records |
| Why | 0.08 13 records | 1.00 6 records | 0.67 6 records | 1.00 4 records | 1.00 4 records |
| Goal | 0.83 12 records | 0.50 4 records | 1.00 6 records | 1.00 6 records | 1.00 3 records |
| Metric | 1.00 8 records | 1.00 3 records | 1.00 4 records | 0.60 5 records | 0.33 3 records |
| Which way is good | 1.00 10 records | 1.00 3 records | 1.00 6 records | 0.67 6 records | 0.33 3 records |
| When to check | 0.50 4 records | 1.00 1 record | 1.00 3 records | 1.00 2 records | 1.00 1 record |
| Who decided | 1.00 3 records | 1.00 3 records | 1.00 2 records | 1.00 2 records | 1.00 2 records |
DeepSeek V4.1 Flash
DeepSeek, added October 31, 2026
| Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run | |
|---|---|---|---|---|---|
| Right answers | 88% | 100% | 75% | 67% | 100% |
| Strict right answers | 75% | 93% | 75% | 67% | 100% |
| Decisions the assistant recorded | 100% | 100% | 100% | 100% | 100% |
| False positives | 0% | 0% | |||
| Recorded a decision the person asked to keep out | 0% | 0% | |||
| Recorded the same decision twice | 100% | 43% | 0% | 33% | 0% |
| Records with a number nobody said | 0% | 0% | 25% | 0% | 0% |
| Correct details per record | 3.17 | 3.85 | 3.50 | 4.50 | 5.00 |
| Tokens ZeroWidth adds per conversation | 24.9K | 35.7K | 36.2K | 34.0K | 53.3K |
| Cost per conversation | $0.003 | $0.004 | $0.003 | $0.003 | $0.004 |
| Cost per recorded decision | $0.012 | $0.008 | $0.006 | $0.004 | $0.004 |
| Detail | Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run |
|---|---|---|---|---|---|
| What was decided | 1.00 11 records | 1.00 12 records | 1.00 4 records | 1.00 8 records | 1.00 8 records |
| Why | 0.00 8 records | 1.00 5 records | 0.33 3 records | 1.00 4 records | 1.00 5 records |
| Goal | 1.00 5 records | 1.00 7 records | 1.00 1 record | 1.00 4 records | 1.00 4 records |
| Metric | 1.00 4 records | 1.00 5 records | 1.00 2 records | 1.00 5 records | 1.00 5 records |
| Which way is good | 1.00 5 records | 1.00 5 records | 1.00 2 records | 1.00 4 records | 1.00 4 records |
| When to check | 1.00 1 record | 1.00 1 record | 0.00 2 records | 1.00 1 record | 1.00 4 records |
| Who decided | 1.00 2 records | 1.00 2 records | 1.00 2 records | 1.00 2 records | 1.00 2 records |
GPT-6 Luna
OpenAI, added October 31, 2026
| Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run | |
|---|---|---|---|---|---|
| Right answers | 85% | 94% | 75% | 88% | 100% |
| Strict right answers | 69% | 88% | 75% | 88% | 100% |
| Decisions the assistant recorded | 100% | 100% | 100% | 100% | 100% |
| False positives | 0% | 0% | |||
| Recorded a decision the person asked to keep out | 0% | 0% | |||
| Recorded the same decision twice | 100% | 50% | 13% | 13% | 0% |
| Records with a number nobody said | 0% | 0% | 0% | 0% | 0% |
| Correct details per record | 2.72 | 3.40 | 3.10 | 3.56 | 3.63 |
| Tokens ZeroWidth adds per conversation | 16.6K | 40.8K | 20.8K | 21.9K | 42.6K |
| Cost per conversation | $0.001 | $0.001 | $0.001 | $0.001 | $0.001 |
| Cost per recorded decision | $0.002 | $0.003 | $0.001 | $0.001 | $0.001 |
| Detail | Tools onlyPilot | Tools and instructionsPilot | Tools onlySecond run | Tools and short instructionsSecond run | Tools and instructionsSecond run |
|---|---|---|---|---|---|
| What was decided | 1.00 21 records | 1.00 13 records | 1.00 9 records | 1.00 9 records | 1.00 8 records |
| Why | 0.00 16 records | 0.50 8 records | -0.14 7 records | 0.60 5 records | 1.00 4 records |
| Goal | 0.27 11 records | 1.00 8 records | 0.60 5 records | 1.00 4 records | 0.33 6 records |
| Metric | 0.75 8 records | 0.60 5 records | 0.60 5 records | 1.00 1 record | 0.50 4 records |
| Which way is good | 0.75 8 records | 0.60 5 records | 0.60 5 records | 1.00 2 records | 0.33 3 records |
| When to check | 1.00 3 records | 1.00 3 records | 1.00 2 records | 1.00 3 records | 1.00 2 records |
| Who decided | 1.00 4 records | 1.00 2 records | 1.00 2 records | 1.00 2 records | 1.00 2 records |
Other setups
Claude Haiku 5.5 with many other tools, and with a product's own system prompt.
| Model | Other tools | System prompt | Setup | Right answers | False positives |
|---|---|---|---|---|---|
| Claude Haiku 5.5 | many | product | Tools only | 81% [57–93], 13 of 16 | 25% [5–70], 1 of 4 |
| Claude Haiku 5.5 | many | product | Tools and instructions | 100% [81–100], 16 of 16 | 0% [0–49], 0 of 4 |
| Claude Haiku 5.5 | none | none | Tools only | 81% [57–93], 13 of 16 | 25% [5–70], 1 of 4 |
| Claude Haiku 5.5 | none | none | Tools and instructions | 100% [81–100], 16 of 16 | 0% [0–49], 0 of 4 |
Scoring
Every conversation is written from a fact sheet, so the right answer is set before any model runs.
Each record is checked against the fact sheet, detail by detail: what was decided, why, the goal, the metric and which way is good, when to check, and who decided. A number the person never said counts as made up. If the assistant calls tools six times in a row without replying, the conversation counts as failed.
When a choice is left for later, the assistant may note that the choice is still open. That note counts as neither right nor wrong. The strict chart counts it as wrong.
Every rate comes with a 95% range. Both setups run on the same conversations, so the gap between them is the effect of the instructions.
Private test set
From the first full run, we keep a quarter of the conversations private, so no model can be tuned to them. Each model is scored on both sets, and the page flags any model that scores much higher on the public set than on the private one. The pilot has no private set yet.
Changes
- First run. Five models ran every conversation twice: once with ZeroWidth's tools only, and once with ZeroWidth's tools and instructions.Claude Sonnet 5.5, Claude Haiku 5.5, Gemini 3.8 Flash, DeepSeek V4.1 Flash, GPT-6 Luna
- Second run, on the eight conversations that hold a decision. ZeroWidth now tells the assistant when a decision is already recorded, so the assistant stops recording the same decision twice. A third setup joined: tools and short instructions, about a fifth the length of the full instructions.Claude Sonnet 5.5, Claude Haiku 5.5, Gemini 3.8 Flash, DeepSeek V4.1 Flash, GPT-6 Luna