No. 1
Aletheia v3 — why we stopped scoring confidence and started banding it
A scalar confidence number is a lie wrapped in a decimal point. After a year of post-hoc analysis we replaced the model with three discrete bands and watched the conflict-resolution rate climb 38%.
A scalar confidence number is a lie wrapped in a decimal point. For the first
year of Gnostikon we shipped a confidence: 0.0–1.0 field on every gnosis
and let downstream agents threshold against it. It looked rigorous on paper.
In practice, the threshold was always either 0.7 (which nothing met) or
0.0 (which everything met).
The problem with scalars
A confidence number implies a population of repeated trials. Most of our gnoses come from one transcript, one author, one moment. There is no population. The "0.83" we returned was a number-shaped opinion. Worse, it moved every time we retrained the embedder, which made downstream behavior non-deterministic across model bumps.
What we shipped
Three bands, one promise per band:
| Band | Promise |
|---|---|
low | "Cite this if asked. Don't act on it." |
medium | "Act on it if no high contradicts it." |
high | "Act on it. If you're wrong, the source is wrong." |
The promise is what bound us. Every band gets a documented contract; every contract gets reviewed when it changes.
What changed
After three months on bands, conflict-resolution rate (the share of disputed
gnoses that get a clean cut on review) climbed 38%. Agents stopped shipping
the 0.7-something-confused-shrug decisions that had been our biggest source
of escalations.
What we got wrong
We initially shipped four bands (low, medium, high, certain).
certain collapsed back into high within a month — nobody could tell the
two apart in review, and the contracts ended up identical. Three is the
right number.
This is the first post on the new docs site. Welcome.