I wanted to understand retrieval-augmented generation at a level deeper than reading about it, so I built a small playground over a weekend. Reading about RAG gives you the high-level architecture; building it gives you the intuitions about what works and what doesn't — which chunks surface for which queries, which prompts produce grounded answers versus hallucinations, which evaluation methods actually catch problems. There is no substitute for building the thing yourself, even if what you build is a toy version of what production systems look like. The idea was simple: upload documents, ask questions, and see exactly how the system retrieves and generates answers. The "see exactly how" part was the most important design decision — every step of the pipeline is visible, letting you inspect retrieved chunks, ranking scores, and the prompt sent to the LLM.

What started as a weekend experiment turned into a tool I still reach for almost daily, both for debugging retrieval issues in production and for testing document sets before building something more permanent. The playground has become a laboratory for prompt and chunking experiments — when a production RAG system isn't performing well on a particular type of question, I can load the same documents into the playground and iterate without the overhead of deploying changes to production for each iteration.

Three components: a document ingestion pipeline that chunks text and generates embeddings, a vector store on PostgreSQL with pgvector, and a query pipeline that embeds the question, retrieves relevant chunks by vector similarity, and feeds them to an LLM. I chose pgvector because it runs inside the same PostgreSQL instance the rest of the application uses — no separate vector database to provision, monitor, or keep in sync. For a weekend project, that simplicity was non-negotiable; for a production system, the tradeoff is more nuanced. Dedicated vector databases like Pinecone or Weaviate offer better performance at scale and more sophisticated indexing algorithms, but they add operational complexity. For most applications up to a few million vectors, pgvector with appropriate indexes is fast enough, and the operational simplicity is worth the performance tradeoff.

The embedding model choice matters more than people new to RAG often realize. I started with OpenAI's text-embedding-ada-002 and later experimented with smaller open-source models like all-MiniLM-L6-v2 from sentence-transformers. The OpenAI model produces higher-quality embeddings but costs money per embedding and requires an API call for every chunk. The open-source model is free and runs locally but produces somewhat lower-quality embeddings. For the playground, the quality difference was acceptable in exchange for the cost and latency savings; for a production system with strict accuracy requirements, the OpenAI model would be the safer choice.

Through experimentation, chunks of roughly 500 tokens with a 100-token overlap between consecutive chunks produced the most reliable retrieval. The 500-token size is large enough to contain a complete idea but small enough that a single chunk doesn't contain too many unrelated ideas that would dilute the embedding. The 100-token overlap handles the case where a relevant idea spans a chunk boundary — without it, questions whose answers span a boundary frequently miss critical context. The overlap effectively gives every chunk a "halo" of context from the chunks around it, making retrieval more robust to imperfect chunk boundaries.

The generation prompt explicitly instructs the model to cite the specific chunks it drew its answer from and to include a confidence indicator, which lets the user check the cited source material rather than trusting the generated answer at face value. Without citations, the user has no way to distinguish a grounded answer from a hallucination; with citations, they can click through and verify. A subtle prompting issue I spent more time on than expected: what to instruct the model to do when the retrieved chunks don't contain the answer. The default behavior of most LLMs is to try to answer anyway, producing confident-sounding hallucinations. The fix is to explicitly instruct the model to say "I don't know based on the provided context" — which produces a less satisfying but more honest response. This is a clear case where the right behavior for production is the opposite of what produces the most impressive demo.