We don't say it's you. We say how often it agrees with you.
One question, measured the same way for every engram: how often would you have done the same? The method is here now. The number comes when the test has run.
The question.
Take decisions you already made. Hide the answers. Replay each situation to your engram. Count how often it would have done what you did. That count, against the ceiling of how often you agree with yourself, is fidelity.
The method, in five steps.
- 01Hold outAt least three hundred of your own decisions, each later in time than every node in your engram. Any thread that mentions a held-out decision is removed from the corpus, so nothing leaks.
- 02StratifyBy decision type and by source stream, so a score is not carried by the easy kind.
- 03ReplayEach situation goes to the helper as it would arrive in life, without your answer. It decides, and cites what it used.
- 04Score against youAgreement, per decision type, reported as a fraction of your own test-retest rate: we re-ask you a sample later, because nobody agrees with themselves every time.
- 05State the marginAt three hundred decisions a seventy-five percent rate carries a ninety-five percent interval of about plus or minus five points. A thinner sample gets a wider interval, printed beside the score.
What a score means, and what it does not.
A score of one hundred percent of your consistency means the engram agrees with you as often as you agree with yourself. It cannot do better than that by construction. It is not a claim that it is you.
Voice is scored separately from decisions, because the research says tone is the weak dimension and we would rather you knew which part to trust. A decision score tells you how far to extend a standing rule; a voice score tells you how closely to read a draft.
We also run the persona card on and off. If voicing the output in your style lowers decision agreement, the card stays out of the decision call for good.
Our own result.
Not yet measured. The held-out test runs on our own founder's engram first. When it has, the score, its interval, its sample size and its date appear here, whatever they are.
What people ask first.
- Why not just show a benchmark score?
- Because a memory benchmark measures recall of facts in a conversation, not whether a helper does what you would do. The published scores are also vendor-run and disputed. Grading on your own held-out decisions is slower and means more.
- What if my record is thin?
- The interval is wider, and we print it. Below a sample that can carry a number at all, we tell you before the build, and the first months run with every action approved by you while the record grows.
- How often is it re-measured?
- Each time the engram changes substantially: new sources, a season of corrections, a change of model. The score has a date on it, and the history stays visible.
- Can I see the decisions it got wrong?
- Yes. Every disagreement is listed with the situation, what you did, what it would have done, and the nodes it used. That list is the most useful page in the product: it shows where a rule is missing or a correction is due.
- Can it score higher than you?
- No, by design. The ceiling is your own consistency: how often you make the same call when the same situation comes back. The score is a fraction of that ceiling, so a perfect score means it agrees with you as often as you agree with yourself, and never that it is you.
Ask how we would measure yours.
Ask how we would measure yours.
You'll talk to a senior engineer, not a sales rep, and leave knowing what your engram would be built from and how it would be measured.