The assignment
Meridian Industrial makes hydraulic presses and stamping lines for automotive parts plants. They have 400 field service technicians, thirty years of equipment manuals, a fault-code database going back to the 1990s, and five years of resolved support tickets sitting in three separate systems nobody wants to open at 2am from a factory floor.
You've been hired to build one thing: a single assistant a technician can ask a question, in whatever words they'd actually use, standing next to a machine that's making a noise it shouldn't be making.
That's the whole brief. Nobody at Meridian used the phrase "domain-agnostic," nobody mentioned Recall@k, and nobody asked for a retry loop with a bounded counter. Every architectural decision in this document is going to arrive because something in this very ordinary assignment broke in a very ordinary way — not because a textbook said so. That's deliberate. If you've never lived through a production RAG system falling over, the rules in most write-ups read like religious commandments: don't use a fixed threshold, don't couple two jobs into one node. Fine — but why not, concretely, today, with an actual technician standing at an actual machine? That's what each chapter here is going to show you before it tells you.
The demo that worked
You have two weeks before a stakeholder demo. The fastest path is obvious: embed all three sources — manuals, fault codes, tickets — into one vector store, embed the technician's question, pull back the top five matches, hand them to an LLM, done. You build it in four days. It's genuinely impressive in the demo: someone asks about a fault code, gets a clean answer, and the room is happy.
The rest of this document is the four months after that demo — the actual questions 400 technicians typed, and what each one broke.
Four promises you can't break
Before the first real bug report comes in, it's worth naming what Meridian's own document mix already guarantees you can't get away with — not as abstract theory, but as a direct read of the three sources you were handed on day one.
None of these were decisions you made. They were already true the moment Meridian handed you three folders. The architecture from here on is just taking that reality seriously instead of hoping the easy 20% from the demo generalizes.
Should the system even search?
"Thanks, that's the fix — appreciate it"
A technician, having just solved his own problem by scrolling further in the chat, types a closing "thanks" message out of habit. The system, which retrieves for every message with no exception, dutifully embeds "thanks, that's the fix — appreciate it," finds the five nearest documents (there's always a nearest five — cosine similarity never returns nothing), and generates a confident answer about gasket replacement torque values, because two of those documents happened to mention torque and the LLM stitched something plausible-sounding out of them. The technician, now confused, forwards the screenshot to his supervisor asking if the tool is broken.
Two witnesses
The fault code that returned everything except itself
A technician standing at a stamping line types exactly what's on the display: "F-2291". The bi-encoder embeds it as a short, meaning-thin string, and returns the five semantically nearest documents — which turn out to be general troubleshooting pages about faults, hydraulic faults, and electrical faults, none of which is the one manual page that defines F-2291 specifically. The embedding space has no special respect for an exact alphanumeric code; it just sees "fault" and "code"-shaped tokens and drifts toward anything fault-code-adjacent.
The same week, a different technician asks "how do I shut it down safely without the emergency stop" — and gets nothing useful either, because the one manual section that answers this is titled "Graceful Termination Procedure" and shares almost no vocabulary with the question at all.
That's the actual argument for hybrid search — not "more retrieval is better," but that BM25 (the literal witness) and a bi-encoder (the gist witness) fail on opposite queries, so running only one guarantees you fail half of Meridian's technicians by construction.
The net and the needle
Hybrid retrieval fixes the two-witness problem, but it introduces a new one almost immediately: casting a wide enough net that F-2291's real page is somewhere in the top eight isn't the same as knowing it's the best of those eight. A bi-encoder scores query and document independently and compares vectors afterward — fast, but blind to fine-grained interaction between the two texts. Meridian's technicians started getting answers built from the third-best document instead of the best one, close enough to sound right, wrong enough to send someone to the wrong valve.
What counts as enough
This is the incident that actually took the system down for over two weeks without anyone noticing — because unlike a crash, it never stopped answering. It just started answering wrong, fluently, on a schedule nobody could see.
Week 6: launch, tuned on one corpus
At launch, the corpus is just the equipment manuals — long, formal, technical prose. You spend a week testing sufficiency by hand and land on a rule: if the top retrieved passage scores above 0.60 cosine similarity against the query, treat it as enough evidence to answer; below that, say "I don't have enough information." It works well. Manual prose is dense and formal, so a genuinely relevant passage tends to land around 0.58–0.68, and irrelevant ones fall well below 0.45. The 0.60 line does real work. You ship it.
Someone on Meridian's side points out technicians keep re-solving problems that were already solved in old support tickets, so five years of resolved tickets get added as a second corpus, searched by the exact same pipeline, with the exact same 0.60 line. Nobody revisits the threshold, because nobody thinks of a threshold as a thing that could stop being true — it shipped, it worked, it's "done."
Support tickets are short, casual, and full of near-identical phrasing across unrelated incidents — "making a screeching noise again," "screeching sound, same as last time," "loud screech on startup." Short, colloquial, overlapping text pushes cosine similarity up, regardless of whether the underlying machines or problems are actually related. A ticket about a screeching cooling fan on a Model 40 scores 0.71 against a technician's question about a screeching hydraulic pump on a Model 60 — comfortably above the 0.60 line, so the system calls it "sufficient" and answers confidently with fan advice.
Meanwhile, a technician asks a question that is answered correctly in the manuals — but the manual's formal phrasing scores only 0.58 against the technician's casual wording. Below the line. The system says "insufficient" and declines to help with a question it could actually answer.
Both failures are invisible from outside. A confident wrong answer and a confident right answer are typographically identical. Nobody files a bug report for seventeen days, because nothing looks broken — it just looks like an assistant that's sometimes unhelpful, which technicians assume is normal for "one of these AI things."
A technician follows the fan-repair advice on a hydraulic pump. The wrong lubricant voids a seal warranty. That's the incident that finally gets the threshold looked at — and the postmortem reveals the number was never wrong on the corpus it was tuned on. It was wrong the moment a second, structurally different corpus started being compared against the same fixed line.
Here's the point worth sitting with: the threshold was correctly tuned. It wasn't guesswork or laziness — it was a genuinely careful, well-tested number, for the corpus that existed in week 6. The bug wasn't in the number. It was in the idea that any single number could survive a second, differently-shaped corpus arriving later — which, in a domain-agnostic system, it always eventually will.
One job, not two
This is the second incident worth living through in full, because the mistake that caused it feels completely reasonable in the moment — which is exactly why it's dangerous.
Week 10: a good idea, working well
A technician asks "how do I shut down the Model 60 safely." First retrieval pass comes back weak — the Model 40 and Model 60 manuals use overlapping language and the initial retrieval blends both models together. The sufficiency judge (Chapter 6's fix, already live) reports back: "insufficient — retrieved passage covers Model 40, not Model 60." A node called query_understanding takes that reason and the failed query, and rewrites it: "Model 60 emergency shutdown procedure, hydraulic press line." Retried, it works perfectly. The technician gets the right page. This becomes the pattern the whole retry loop is built on: one failed query in, one better query out — always a single string.
A regional manager asks: "What's different between the Model 40 and Model 60 shutdown sequences?" This isn't a reformulation problem — it's genuinely two separate lookups that need to be retrieved independently and then compared. Under deadline pressure, the fastest-looking fix is to teach the same query_understanding node to also detect comparison questions and split them into two sub-questions. It ships in an afternoon. It works, in the demo shown to the manager.
The retrieve node downstream was updated the same afternoon to "handle the new output" — since query_understanding could now return either a single string or a list of two sub-questions, someone wrote the retrieve node to expect the new shape: a list to iterate over. It wasn't tested against the old reformulation path, because nobody thought of it as a different path — it's "the same node," after all.
Here's the exact failure: when a plain reformulated string like "Model 60 emergency shutdown procedure, hydraulic press line" comes back from the reformulation path, the retrieve node iterates over it exactly as it would a list — except a string is iterable too, character by character. It doesn't crash. It happily retrieves for the query "M", then "o", then "d", then "e", then "l"… forty-some single-character "sub-queries," each one dutifully embedded and searched, each one returning some nearest document to the letter "e" or the letter "l" in vector space — nonsense, but fluent-sounding nonsense once an LLM is asked to write an answer from it.
For four days, the team debugs this as an LLM quality problem — "the model seems to be hallucinating more on retries lately" — because nothing in the logs says "type error." Nothing crashed. Nothing raised an exception. A silently wrong type assumption just quietly ruined every single retry in the system, while every first-pass answer kept working perfectly, which is exactly why it took four days: the bug only ever showed up one retry deep, in the one code path nobody was looking at.
The postmortem's actual finding, verbatim from the incident writeup: "the bug was never in the LLM. It was that one node's output type wasn't part of its contract, so nothing downstream could trust it." That's the whole argument for splitting decomposition from reformulation — not architectural purity for its own sake, but four real days that a type-shape guarantee would have prevented outright.
The brake
The retry loop that wouldn't stop
A technician asks about a fourth-generation press model Meridian sold to exactly one customer, whose manual was never digitized. Every retrieval pass comes back thin. The sufficiency judge, quite correctly, keeps saying "insufficient." The reformulation node, quite correctly, keeps trying a slightly different phrasing each time. Nothing is malfunctioning — the corpus genuinely doesn't contain the answer, and nothing in the loop knows to ever stop looking. The technician's request sits "in progress" for four minutes before someone kills the process by hand.
retry_count, lives in shared state and increments on every reformulation. Insufficient and under the limit → reformulate and retry. Insufficient and at the limit → a dedicated give_up exit. What that exit actually says — "here's the closest thing I found, with a caveat" versus a flat "I don't have this" — is a configuration value the node reads, not a branch it contains, because Meridian's answer for a safety-critical shutdown question should probably be a hard decline, while their answer for a minor cosmetic fault might be fine as a hedged best-guess. One node, one read of a config value, no rewritten logic between the two policies.
When one question is really two
Back to the regional manager's comparison question from Chapter 7 — now solved properly, with decomposition split cleanly out of reformulation.
"What's different between the Model 40 and Model 60 shutdown sequences?" triggers decompose, which returns exactly two sub-questions: "Model 40 shutdown sequence" and "Model 60 shutdown sequence." Each one runs the entire retrieve → rerank → grade cycle completely independently — its own retries, its own evidence, its own local answer — with zero shared state between the two branches until both finish. A synthesis step then reads both finished answers and writes the actual comparison.
This is the one place in the whole system where per-branch isolated state is correct, not a violation of the "no nested sub-state" discipline from Chapter 7 — two genuinely independent lookups deserve genuinely independent state, as long as the merge back into shared state happens through one explicit, named step, never silently.
What survives a retry
Trace the Model 60 shutdown question from Chapter 7 through the shared state, field by field, and the design principles stop being abstract:
Prompts and instrumentation
Two disciplines that don't show up as their own incident, because they're the reason several incidents above got caught in hours instead of weeks. Every LLM-calling node owns exactly one prompt template and reads only the state fields it needs — the sufficiency grader in Chapter 6 never needed the original raw query, only the active query and the candidates; the reformulator in Chapter 7 never needed the final answer field, because it doesn't exist yet at that point in the run. A shared "master prompt" with conditionals is the same coupling mistake as the merged query_understanding node, one layer down — and would have made the Chapter 7 bug even harder to isolate.
Latency and evaluation numbers are not core logic and must never look like it. A thin wrapper around every node records timing into a metrics field before returning control to the graph — no node ever branches on a timer. This is precisely what let Meridian's ops team separate "reranking is slow" from "retrieval is slow" in Chapter 5, instead of staring at one blended number with no way to tell which stage to fix.
Proving it's really domain-agnostic
Every incident in this document was found by testing the system against a real, differently-shaped part of Meridian's own content — not by imagining an edge case in the abstract. The manuals surfaced the sufficiency-threshold assumption in Chapter 6. The tickets' short colloquial phrasing is what actually broke it. The fault codes surfaced the exact-match blind spot in Chapter 4. If Meridian had only ever had one of these three document types, at least three of this document's chapters would never have been written, because the failure they describe would never have had a chance to occur.
The RAG, distilled
Strip away Meridian's presses and fault codes, and thirteen chapters of specific incidents distill into a general method for architecting any system like this: