A RAG pilot in four weeks: scope it so it can't lie to you

The dirty secret of RAG pilots is that most of them are built to succeed. Cherry-picked documents, demo questions rehearsed until they land, a happy-path UI — then production traffic arrives and the system meets questions nobody rehearsed. The pilot didn’t fail; it lied.

A pilot’s job is not to impress. It’s to predict production. Here’s the four-week structure we use to make a pilot that can’t lie to you, whoever builds it.

Week 1: Write the acceptance criteria before touching architecture

Before any code: a document, signed by both sides, that says what the pilot must do to earn a production budget. Ours has three sections:

The question set. 50–100 real questions, pulled from tickets, emails, and the actual humans who’d use this — not invented by the builders. Crucially, it includes questions the system should refuse: out-of-scope asks, questions whose answers aren’t in the corpus, questions where the honest answer is “ask legal.”

The numbers. Retrieval hit rate (did the right passages surface?), answer accuracy against a rubric, refusal correctness (did it say “I don’t know” when it didn’t know?), latency p95, and cost per query. Set thresholds now, while nobody’s invested in passing them. Teams that set thresholds after building always discover their system’s score is, remarkably, exactly the threshold.

The kill criteria. What result means stop — don’t proceed to production, don’t extend the pilot, stop. Writing this down up front is the single strongest honesty mechanism we know, because it converts “let’s keep iterating” (which can burn quarters) into a decision someone has to defend.

Week 2: Build the boring skeleton

Ingestion, chunking, retrieval, generation, and the evaluation harness running end to end on day one of week two — ugly, unstyled, scoring terribly. That’s fine. The point is that from here on, every change gets measured. The team stops arguing about chunking strategies in the abstract and starts reading eval diffs.

Two rules keep this week honest. First, the corpus is the real corpus — the messy wiki, the PDFs with tables, the docs with three contradictory versions — not a curated sample. Curation is how pilots learn to lie. Second, no UI work yet. A pilot that gets pretty before it gets accurate has chosen its priorities, and they’re the wrong ones.

Week 3: Iterate against the numbers

This is the week that looks like magic from outside and is actually plumbing: fixing chunk boundaries that split tables, adding metadata filters, tuning retrieval depth, tightening the system prompt’s refusal behavior, handling the documents that turned out to be scanned images. Each change lands with an eval run. The score graph over the week is the status report.

Somewhere mid-week, run the red-team hour: the domain experts try to break it in front of everyone. Prompt injections hiding in documents, questions premised on false facts, requests for advice the system must not give. Every break becomes an eval case. This hour is consistently the highest-value sixty minutes of the engagement — and the one most pilots skip because it’s uncomfortable.

Week 4: The readout nobody has to spin

Week four produces two artifacts. A thin real interface — enough for ten actual users to run their own questions for a few days (their unrehearsed usage is data no eval set replaces). And the readout: scores against the week-one thresholds, cost-per-query at realistic volume, the failure catalog with examples, and a scale/stop recommendation with reasons.

Because the criteria were written before the architecture, the readout defends itself. Either the system cleared bars both sides agreed on, or it didn’t — and when it didn’t, the readout says “here’s what would have to change, and here’s why we’d wait.” That conversation is uncomfortable for exactly one meeting, and then it saves a six-figure production budget from chasing a pilot that was performing rather than working.

The uncomfortable summary

You can have a RAG pilot that demos beautifully in two weeks. You can have one that predicts production in four. The two-week version usually costs more in the end, because its failure gets discovered after the production budget is spent instead of before.

Scope the honest version — ours is fixed-price, with the acceptance criteria written into the SOW — or take this structure and run it with whoever you already trust. Either way, write the kill criteria first. It’s the cheapest insurance in AI.