Simplex VM0: grounding long-term memory in evidence
Results from EgoLifeQA, ATM-Bench and Mem-Gallery, and what they tell us about memory for persistent interactive systems.
Simplex AI Research & engineeringPreliminary SOTA results, with evaluation context
Based on the team's current evaluations and comparisons, Simplex VM0 achieves SOTA-level results on the evaluated long-term-memory tasks. The results below retain their evaluation context. This is a team-reported conclusion, not an official leaderboard certification; different inputs, splits and judges limit direct comparisons.
“What happened to the thing we talked about last time?”
For a robot or an assistant that stays with a user over time, this is an ordinary question. Answering it requires more than finding a similar sentence. The system must connect records of the same event, account for changes, and recover evidence that was available at the relevant time.
Simplex VM0 is our ongoing work on memory for long-running interactive systems. We recently evaluated it on EgoLifeQA, ATM-Bench and Mem-Gallery. Here we share the main results and what they tell us about persistent memory.
Remembering enough to answer
The difficult parts often sit between records. A name may appear across many conversations. An image may contain a detail missing from its description. A later message may replace an earlier plan. Retrieval is a starting point; the system also needs to establish which records belong together and which facts remain valid.
Our focus is on answers grounded in sources and time. This evaluation covers life-log question answering, information distributed across personal records, and multimodal conversational memory.
EgoLifeQA: questions about a recorded life
On our internally assembled manual-2.9K evaluation set, VM0 answered 2,150 of 2,905 questions correctly: 74.01% accuracy. One question with a malformed timestamp could not be answered normally and remains in the denominator.
On the official 500-question A1_JAKE set, the result was 421 / 500, or 84.20% accuracy. These sets overlap and cover different scopes. A1_JAKE concerns a single wearer, so the two scores should not be read as a direct comparison at equal difficulty.
This run used the released dense video descriptions and life-log text, with records cut off at the question time to exclude future information. It measures memory and question answering over those descriptions, rather than end-to-end understanding from raw video pixels.
ATM-Bench and Mem-Gallery: complementary tests
ATM-Bench places questions in a broader personal-information setting. Mem-Gallery evaluates long-term multimodal conversational memory. They test aspects of memory that a single multiple-choice accuracy number cannot capture.
| Evaluation scope | Size | Result |
|---|---|---|
| ATM-Bench ordinary set | 1,013 questions | QS 85.78% |
| ATM-Bench Hard | 31 questions | QS 66.91% |
| Mem-Gallery | 1,711 questions | Normalized judge mean 91.64% |
These are our internal system evaluations, not certified leaderboard submissions. ATM-Bench QS and the Mem-Gallery judge mean are different metrics, and neither should be treated as multiple-choice accuracy or averaged together.
Both evaluations used GLM-5.3-Flash as the judge, which affects comparability with reports using other judges. The ordinary and Hard ATM-Bench runs used different evaluation-pipeline versions; the Hard run added original-image verification. These datasets also informed development, so the results should be read as development evaluations rather than untouched independent tests.
What this means for persistent interaction
The results encourage us to keep connecting memory to running interactive systems: returning to an earlier topic, updating an understanding when records change, and recovering supporting evidence when needed.
They also leave clear work ahead. ATM-Bench Hard has substantial room for improvement. Errors can arise in retrieval, temporal reasoning, use of visual information or final answer generation. These are whole-system results; they do not establish how much any individual component contributes.
We will follow up with a fuller evaluation write-up and examples from interaction. The goal remains concrete: when a user returns to something from the past, the system should be able to pick up the thread and explain its answer.