Retrieval augmented generation looks simple on paper: embed your documents, retrieve the nearest neighbours to a query, and hand them to a language model. In practice, naive implementations surface irrelevant chunks, invent citations, and degrade badly once real users arrive with queries you never anticipated.
Most failures trace back to retrieval, not generation. Pure vector search rewards semantic similarity even when it is the wrong kind of similarity, so a hybrid of keyword (BM25) and dense retrieval consistently outperforms either alone. Chunking strategy, embedding choice, and reranking matter far more than the model you put at the end of the pipeline.
The systems that hold up in production treat retrieval as an evaluable component with its own metrics, ground answers in retrieved context with explicit citations, and fail loudly when confidence is low rather than confidently returning something wrong.