How LLMs actually work · module 02 of 9 · ~50 min
What vector search can and can't see
An embedding turns text into a point in space, and cosine similarity ranks by the angle between points. That buys you topical resemblance and quietly loses negation, exact numbers, and rare names. Knowing the blind spots tells you when to reach for lexical search instead.
By the end you can explain
- — By the end you can explain what an embedding encodes and what cosine similarity ranks by.
- — By the end you can name the three failure classes where dense retrieval silently returns the wrong thing.
- — By the end you can say when BM25 beats embeddings and why hybrid plus reranking covers both.
- — By the end you can explain why chunking decides what your system can ever find.
You build semantic search over your docs. A user asks "which materials are not fire-rated," and the top result is a passage about materials that are fire-rated. The search didn't break. It did exactly what it was built to do, and what it was built to do is not what you assumed.
Vector search ranks by resemblance, and "X" resembles "not X" almost perfectly. Same words, same topic, opposite meaning. If you know what the geometry measures, that result stops being a mystery and starts being a predictable failure class you can design around. This module is about what the geometry sees, and the three things it's blind to.
What an embedding encodes
An embedding model maps a chunk of text to a list of numbers, a point in a high-dimensional space. A common one produces 1,536 numbers per input. The training objective, going back to word2vec, is roughly: text that shows up in similar contexts should land near each other. So "cardiologist" and "heart surgeon" end up close, because they appear in the same kinds of sentences.
That's the whole trick, and it's also the whole limit. The space is organized by usage, by what appears near what, not by logic and not by truth. The famous word2vec result, king minus man plus woman lands near queen, tells you the space captures relationships of usage. Hold onto that. It's about to cost us.
Predict first
'Smoking causes cancer' and 'cancer causes smoking.' How close are their embeddings?
Cosine is an angle, and only an angle
To search, you embed the query, embed every chunk, and rank chunks by how close they are to the query. The standard closeness measure is cosine similarity: the cosine of the angle between the two vectors. Two vectors pointing the same way score 1.0 even if one is three times longer. Point them apart and the score drops toward 0. That's the number your vector database sorts by.
Drag it before you read another word about it.
Cosine similarity
1.00
The angle between the vectors, nothing else. This is what pgvector's <=> cosine operator ranks by.
Dot product
7.32
Angle times both magnitudes. Grows when either vector gets longer.
Euclidean distance
3.12
Straight-line gap between the tips. On unit-normalized vectors (the dashed circle) it ranks the same as cosine.
Cosine says identical (1.00). Euclidean distance says far apart. Cosine can't see magnitude.
The thing to try to break: make two vectors that a human would call "the same meaning" score low, and two that clearly differ score high. You'll find cosine throws away magnitude entirely (the "same direction, different length" preset scores 1.00 while the straight-line distance is large), and that it only ever answers one question, how much do these two point the same way. That question is topical resemblance, which isn't the same as equivalence or relevance. I walked through the geometry and the real pgvector setup behind it in what cosine similarity actually measures.
The three things it can't see
From "the space is organized by usage," you can read off exactly where dense retrieval fails, and it's the same three every time.
Negation. "X" and "not X" share all their words and their topic, so they embed as near-neighbors. The benchmark paper NevIR showed that essentially every neural retriever ranks a document and its negation almost identically. Your fire-rating query from the top of this page is this failure.
Exact numbers and IDs. "10.0 metres" and "12.0 metres" are close in embedding space because they're the same kind of token in the same kind of sentence. The specific value, the part you actually care about, is barely encoded. Same for part numbers, case numbers, and regulation codes.
Rare entities. A name or identifier the model saw rarely in training has no rich neighborhood, so it collapses toward its category. A specific person's name drifts toward "a person," a specific statute toward "some regulation." The long tail is exactly where you most need precision and exactly where dense retrieval has the least.
The tool that has the opposite blind spots
There's a retrieval method that nails every case dense search misses: exact term matching, weighted well. BM25 scores a document by how many of the query's terms it contains, weighted so that rare terms count more than common ones. Search for "O. Reg. 176/26" and BM25 finds the one document with that exact string, because the string is rare and therefore high-signal.
Predict first
You need the document about 'O. Reg. 214/26' specifically. Which retriever should you trust more?
Watch the score come from the rare terms.
#1 · doc 2O. Reg. 176/26 amends the fee schedule for building permit applications effective July 1.
3.60o.: 1.03 (idf 1.03) · reg.: 1.03 (idf 1.03) · 176/26: 1.54 (idf 1.54)
#2 · doc 4O. Reg. 214/26 revokes the previous height exception for lots fronting on a laneway.
2.06o.: 1.03 (idf 1.03) · reg.: 1.03 (idf 1.03)
#3 · doc 1The maximum permitted height of a detached house is 10.0 metres in a residential zone.
0.00#4 · doc 3Maximum lot coverage in a residential zone is 33 percent of the lot area.
0.00#5 · doc 5A tall building near a residential area requires additional shadow studies before approval.
0.00#6 · doc 6Permit fees are set out in the schedule and reviewed annually by the municipality.
0.00Try the identifier query, then the concept query. BM25's score is carried by the rare exact terms, precisely what embedding similarity smears away. It's also blind to paraphrase: ask BM25 for "maximum building height" and it won't find a passage that says "tallest a structure may be" with no shared words. So the two methods fail in mirror-image ways, which is the setup for the fix.
Hybrid, then rerank
Run both. Dense retrieval catches paraphrase and topic, BM25 catches identifiers and exact terms, and you fuse the two ranked lists (reciprocal rank fusion is the common, boring, effective way). That covers both failure modes with one pipeline. A provider write-up on contextual retrieval measured hybrid contextual retrieval cutting the failure rate substantially, with a further drop when reranking was added on top, though the exact figures will depend on your corpus.
Reranking is the second lever. A cross-encoder reranker reads the query and a candidate document together, in one pass, so it can judge whether the document actually answers the query instead of just sitting near it in space. It's too expensive to run over your whole corpus, so the pattern is: retrieve wide with hybrid search, take the top fifty to a hundred, then rerank down to the handful you actually use.
Chunking decides what can be found
One thing happens before any of this, and it quietly sets the ceiling. You don't embed whole documents, you embed chunks, and how you cut the chunks decides what each vector means. Cut a sentence in half and neither half carries the fact. Pack two topics into one chunk and you get a mushy average vector that's a strong match for nothing. Retrieval quality is often decided at chunk time, before a single search runs.
Section 4. Building Height 4.1 In a residential zone, the maximum permitted height of a detached house is 10.0 metres where the lot frontage is 12.0 metres or greater. 4.2 Despite 4.1, where a site-specific exception a
pplies to the lot, the maximum permitted height is the height set out in the exception, and the exception prevails over the general limit in this section. Section 5. Lot Coverage 5.1 The maximum lot coverage in a resid
ential zone is 33 percent of the lot area. 5.2 A covered platform attached to the main wall is excluded from lot coverage where its area does not exceed 4.0 square metres.
Watch what happens to the two-part fact, the height limit and the exception that can override it. Slide the chunk size and change the strategy, and notice the state where the limit lands in one chunk and the exception in another. Retrieve only the first and you'll answer "the limit is 10 metres" with total confidence and be wrong, because the exception that changed the answer was never in the context. That's not a model failure. That's a chunk boundary drawn in the wrong place.
In production
The NoeticMap pipeline runs on exactly this stack over about 13,000 research papers in Postgres with pgvector. Two things from building it are worth stealing. First, it keeps a full-text index next to the vector index, because the corpus is full of exact strings, author names, specific measures, identifiers, that dense search alone loses. Second, it embeds the same corpus at two granularities: a paper-level vector for "what's been studied here" and a finding-level vector for "what did a study specifically conclude." The retrieval unit changed the results more than any model parameter, which is the chunking lesson wearing work clothes. And the rule that keeps the whole thing honest: never judge retrieval by eyeballing the model's final answer. Measure recall against a small labeled set, because a confident wrong answer and a correct one read identically until you check. Permit-check, the Toronto and Ontario permit tool, hits the identifier side of this constantly, where a query is a bylaw or regulation code and lexical matching is doing the real work.
Can you retrieve it?
Answer out loud before revealing. If one is fuzzy, the section it came from is the one to reread.
Go deeper — the best material on this, curated