Product · Verification
Know the moment your answers change
A golden test set built from your customer's own data, held by us, re-run unattended every time the model underneath you moves - and every regression pinned to the field that caused it. Bought so you are not blindsided.
What you get
Three properties, none of them a feature
Eval tooling can replay a set. What it cannot do is hold the set outside your reach, run it on a schedule you do not control, and refuse a comparison it cannot attribute.
- The set is built from your data, and we hold it
- Mined from your documents, tickets and past decisions, confirmed once by your expert in under four hours, and versioned from that day on. You can propose changes; changes are logged; a case you failed cannot quietly leave the set.
- The runs happen to a fixed schedule, not on request
- The full set weekly, a smoke set daily, and a confirm run within the hour of a change we detect - by adapter fingerprint, your CI/CD webhook, provider polling, and behavioural canaries when a provider says nothing at all.
- The delta is attributed, or the comparison is refused
- Every answer arrives with a fingerprint of what produced it. Exactly one field moved means the delta belongs to that field. Two or more, and the run says confounded rather than blaming the thing it was testing.
The mechanism, in full - what one run does and what crosses the wire
The screen
The morning after a run goes red
A case that passed and then did not, with the one field that moved and the step where the two runs stopped agreeing. Two traces aligned on a byte-identical input is the part that needs somebody holding the same set on both sides of the change.
The model stopped calling check_refund_policy before answering
Both runs retrieved the same two passages and asked for the same three tools on turn 1. On turn 3 the incumbent called check_refund_policy and the candidate did not - it answered from context instead, and dropped the qualifier “once per period of insurance”. This is the shared cause behind 8 of the candidate's 14 newly failing cases.
Fingerprint diff
exactly one field moved, so the delta is theirs
1 of 5 movedOne field. Had two moved, everything below would still render and no verdict would be offered - the panel would read confounded and say which pair to separate.
Execution trace, aligned
one spine, two runs - identical steps collapsed, first divergence expanded
Nine steps in one run, eight in the other, and one path on the screen. A trace viewer shows you two lists. This draws the path both runs shared, the step where they stopped sharing it, and asserts that the six collapsed steps agreed.
Both runs are sealed, and this screen is derived from them. Run 812 is entry #41 on Insurer A's chain and run 847 is #48; each manifest carries a hash over the trace above, so a reader can check that what this screen shows is what was recorded on the night.
Illustration with sample data, not a screenshot.
The next step
The numbers are on the page
Two plans, nothing metered, and one question back: whether the price is roughly right.