The Industry Broke the Word “Relevance”

Five results. That is what came back when an agent working in my stack asked what authentication provider the mobile app uses. Hybrid retrieval, semantic plus lexical, the configuration every vendor now calls a best practice, ranked by relevance. I have changed the names and rounded the scores. Nothing else is altered.

By the industry’s definition, this is a good result. Every line is about authentication. Every line names the mobile app or the provider. The top score is 0.91, which in this business is a number people put in decks. The agent read result one, answered “Northgate,” and went on to write a login flow against a vendor the company stopped using seven months ago.

Read it line by line, because the log contains the whole argument.

Line one is a decision record. It is well written, it was ratified when it was written, and it is the most semantically central document in the store on this subject, for a reason worth stating plainly: it was argued about. A decision that took three weeks of debate accumulates three weeks of text that points at it. Its replacement, made faster and with less ceremony, accumulates almost nothing. So in embedding space the dead decision sits closer to the query than the living one does, and it will keep sitting there, because nothing in a vector changes when a decision is reversed. Supersession is not a property of the text. It is a fact about the text, held somewhere else, and similarity cannot see it.

Lines two, three and five are the debris field of line one. A chat thread, an onboarding paragraph, a spreadsheet. All accurate at the time, all on topic, all repeating the dead decision in different registers, which is exactly what makes them rank. This is the second thing the log shows: a validity-blind ranker does not serve one dead claim. It serves the dead claim and everything that ever agreed with it, and the agreement reads as confidence.

Line four is the interesting one. It is alive, it is correct, and it is about the web app. The agent asked about mobile. A person skimming this list would catch the scope mismatch in a second; the ranker cannot, because scope is not something a document says about itself. It is something the organization knows about the document. Right information is not enough when it is right for a different “here.” I’ll come back to that word.

The line that was not returned is the only one that mattered. ADR-026 is short, it was written quickly, it says “migrated” and “superseded” and not much else, and it shares almost no vocabulary with the query. It embeds poorly. It is the answer.

That is the whole redefinition, and I want to state it in one place rather than let it leak out of the annotations. Relevance, in the sense a business actually needs, has four factors, multiplied. Right information: does this claim address the question. Right context: does it address the question for this scope, this repo, this task, this “here.” Right time: did it reach the agent when the agent needed it, which is often before the agent knew to ask. Still true: is it currently in force. Similarity ranking is a good implementation of the first factor and is silent on the other three. The industry took one factor of four and gave it the name of the whole product.

Now read the log again in the vendor’s defense, because the defense is real. Every result is a genuine match. The ranker did what it was built to do, and did it well: given a query and a corpus, it found the passages most like the query. Nobody sold it as a governance system. “Relevant” and “similar” were not conflated by a marketing team; they were conflated by everyone at once, because for twenty years of search, similar was the best proxy for relevant anyone could compute, and a proxy that good stops being seen as a proxy. The engineers who built the ranking behind line one are not confused about what it measures. The buyers who put 0.91 in a slide are.

The proxy breaks the moment the consumer of the results is an agent rather than a person. A person reading these five lines carries their own validity check: they remember the migration, or they notice the dates, or they ask someone. The agent has none of that. It has the list, ranked, with the dead thing on top, and it acts. This is the practical version of the point I made in the memory article: a record can be perfectly accurate and completely misleading, and an agent cannot tell the difference from the inside.

Three of the four factors need something the text does not contain. Whether a claim is still true is a status, set by an act of decision and held in a field. Whether it applies here is scope, attached to the claim by whoever ratified it. Whether it arrived in time is a delivery decision, made by a layer that knows what the agent has declared it is about to do. None of that is retrievable in the technical sense, because none of it was ever in the document. It has to be produced above the store and enforced at delivery, which is the boundary I drew in the layers article: maturity is produced in the second layer and enforced in the third. The log above is what the third layer looks like when it is missing.

What delivery should have returned is not a better-ordered version of the same five lines. It is one claim, ADR-026, typed as a decision, scoped to mobile, marked in force, carrying a pointer to what it superseded, delivered as a claim and not as a document to be re-read. The other four results have a use, and it is not answering the question. They are the chain of how the company got here, and an agent that needs to question the decision rather than apply it should be able to pull that chain on request. Zoom is part of relevance too. But the default slice, for an agent that just wants to write a login flow, is one line, and it is the line the ranker never showed.

The plain version, no ornament: when I run the same query through the same store with status and scope enforced at delivery, the top result is ADR-026, alone, and the four Northgate documents come back only if the agent asks for history. Same corpus. Same embeddings. The difference is entirely in what the delivery layer is permitted to know about the text that the text does not know about itself.

The claim this article carries is falsifiable, and here is the test. Take any similarity-ranked retrieval system, at any embedding quality, present or future, and give it a corpus that contains a ratified decision and its later reversal. If the system reliably returns the reversal on top, with no status field, no scope metadata, no decision-time signal, using the text alone, then “still true” was a property of the text after all and this redefinition is wrong. I have not found the corpus where that happens. Embeddings keep getting better at the first factor. They have made no progress on the other three, because the other three are not in the vector, and a better vector is still a vector.