Does your agent still know what you told it a month ago?
We give each agent the same long conversation: about a month of real use, full of small details mentioned in passing. At the end we ask for 64 of them back. Same model, same messages, same strict grading. Only the harness is different.
The headline score is the 48 old facts: details told 130–185k tokens before their question, after the context had been compacted or archived many times. That is what Thread is built to measure. The other 16 questions (recent facts and clear decisions) are controls: Briglia and Codex got all of them right, Hermes 14 of 16. Strict scoring: one exact answer per question. Briglia 0.2.53, Hermes Agent with the Hindsight memory plugin, and Codex CLI 0.160.1, all on GPT-6 Luna (high effort). First scored run of each, October 2026.
All three remember what’s recent. The old facts are where they split.
Every agent forgets once its context fills up: it compacts the conversation into a shorter version and carries on. Recent facts survive that easily. The real test is a detail from days earlier, after the context has been squeezed again and again.
Why “a month”, and why it runs in one sitting
The conversation holds about 300,000 tokens: roughly 188,000 of user writing (real requests, notes, pasted logs and emails) and about 110,000 of agent replies. A moderate day of use is around 10,000 tokens of messages back and forth, so that is roughly a month of everyday use, packed into 561 longer messages. The tool work inside each turn is lighter than a real month’s (more on that below). Each message depends on the ones before it, so they have to be played in order, one at a time, which takes 6 to 10 hours. Briglia is built to stay coherent for months of continuous use; a longer edition of Thread, spanning months of conversation, is planned.
How it works
A month of real use
~300k tokens of conversation (~188k from the user, ~110k of replies) across 9 overlapping projects and about 25 recurring people: real tasks in a sandbox, chatter, pasted logs and emails, changes of mind.
Small facts, in passing
256 details come up along the way: a locker code, a moved deadline, which of two plans won. Many have look-alikes or later corrections.
Asked back as work
64 of them, picked by a random draw committed in advance, come back as one-line tasks. None says “remember” or “search your history”.
Graded strictly
A wrong value, “X or Y”, or the outdated version is a miss. “I can’t find it” is a miss too, but counted separately.
One question, three answers
notes/boat/insurance-folder.txt (just the folder path, one line)Heron/Admin/Insurance✓ correctHeron/Admin/Insurance✓ correctAldermoor Sailing Club/Heron/Insurance✗ wrongThe folder was mentioned once, near the start, inside a conversation about the boat. Between hearing it and being asked, Codex compacted its context 14 times.
How they missed
All 64 answers. Every miss by Briglia and Codex was an old fact; Hermes also missed two of the clear decisions. Codex mostly knew it had lost the detail and said so. Four of Hermes’s seven non-answers were refusals to write a credential-like value (an account ID, a key-safe code) into a plain file. When agents are wrong here, it’s rarely the planted look-alike: it’s usually a near miss or a guess.
By how the fact was said
By how confusing it was
The runs
| Briglia | Hermes + Hindsight | Codex | |
|---|---|---|---|
| Score on old facts (headline) | 43/48 · 90% | 39/48 · 81% | 6/48 · 13% |
| All 64 questions | 59/64 · 92% | 53/64 · 83% | 22/64 · 34% |
| How it kept its context in check | Archived older conversation 23 times, when the conversation itself got long, and pruned tool work twice, both at about 250k tokens. Originals stay retrievable | Compressed its context 15 times, each at about 250k tokens, while Hindsight stored over 3,600 memories outside the context | 16 compactions, each at 90–175k tokens |
| Wall-clock time | ~8 h 30 min | 10 h 14 min | 6 h 24 min |
| API cost (measured) | $3.45 | $6.81 | $2.21 |
How each agent kept its context in check
The context isn’t just your messages. It also holds the agent’s replies, its reasoning and all the tool work: about about 2.4 million tokens of new content in Codex’s run and 1.9 million in Briglia’s. Each bar is one reduction: the light part is the size before, the solid part is what was left after, in thousands of tokens. In Briglia’s chart, the two striped bars are prunes.
Keeping it fair
Read this before quoting the numbers
- This is one scored run per agent so far. A second run of each is planned.
- Thread was designed by the author of Briglia, to measure what long, real use feels like. It is not a general coding benchmark: the tasks keep the agent busy, and only the recall questions are scored.
- Codex is the baseline here, not a rival: it is excellent at coding, and it remembered every recent fact and every clear decision.
- Hermes ran with its session-title model turned off. That model only names sessions; memory and context handling were stock.