Instead if the exam was open-book, looking at any question we will see our book's index and try to find that chapter which is related to our question and then in that we will search for our answer. This is what RAG is.

What RAG is and why it was introduced ?

Large Language Models (LLMs) are trained on large amount of data, enabling them to perform a wide range of language tasks. However, their knowledge is limited to the data available during training, meaning they cannot access newly published information or private documents. As a result, they may produce outdated or incorrect responses, a phenomenon known as hallucination.

Retrieval-Augmented Generation (RAG) was introduced to overcome this limitation. Instead of relying solely on the model's internal knowledge, RAG retrieves relevant information from an external knowledge source such as documents, databases, or knowledge bases and provides it to the LLM as context before it generates a response.

How a basic RAG pipeline works

Document Processing: The source documents are first divided into smaller, meaningful sections called chunks. Chunking makes it easier to retrieve only the most relevant portions of a document.

Embedding Generation: Each chunk is converted into a numerical representation, known as an embedding, which captures its semantic meaning. These embeddings are then stored in a vector database.

User Query: When a user submits a question, the query is also converted into an embedding using the same embedding model.

Retrieval: The vector database compares the query embedding with the stored document embeddings and retrieves the most relevant chunks based on semantic similarity.

Prompt Augmentation: The retrieved chunks are combined with the user's original query to create an enriched prompt. This provides the LLM with the necessary context to answer the question.

LLM Response: Finally, the augmented prompt is passed to the language model, which generates a response grounded in the retrieved information

Common scenarios where RAG works well

RAG is particularly useful in applications where responses need to be based on external or frequently updated information rather than the LLM's pre-trained knowledge. It is widely used in enterprise knowledge bases to answer questions about company policies, technical documentation, and employee handbooks. Similarly, customer support systems use RAG to retrieve information from product manuals, FAQs, and troubleshooting guides, enabling them to provide more accurate and context-specific responses.

RAG is also effective for document question answering, where users can query reports, research papers, contracts, or PDFs and receive answers grounded in the document content. In domains such as healthcare, legal services, and research, RAG helps retrieve relevant guidelines, regulations, or academic literature, allowing the language model to generate responses that are both context-aware and based on reliable external information.

Why RAG sometimes gives incorrect answers

Although RAG enhances the accuracy of LLMs by providing relevant external information, it does not guarantee correct responses. The quality of the generated answer depends on the relevance, completeness, and freshness of the retrieved information, as well as the LLM's ability to interpret it correctly. Issues such as poor retrieval, ineffective chunking, context window limitations, outdated knowledge bases, and model hallucinations can still lead to incorrect or misleading responses, making careful system design and maintenance essential for reliable performance. It also depends on the query of the user if it does not matches with the context LLM will make an answer by itself.

Limitations of RAG

1. Poor Retrieval and Missing Context
The effectiveness of a RAG system depends heavily on its retrieval component. If the retriever fails to fetch the most relevant documents or omits important information, the language model will generate responses based on incomplete or irrelevant context. As a result, the answer may be inaccurate or fail to fully address the user's query.

2. Poor Chunking and Its Impact on Responses
Documents are typically divided into smaller chunks before being stored in a vector database. If the chunks are too small, important contextual information may be lost. Conversely, overly large chunks may contain unrelated information, reducing retrieval precision. Choosing an appropriate chunk size and overlap is therefore essential for generating accurate responses.

3. Context Window Limitations
Large language models have a fixed context window that limits the amount of retrieved information they can process in a single request. When too many relevant documents are retrieved, some information may be truncated or excluded, causing the model to overlook important details and produce incomplete answers.

4. Hallucinations Even with RAG
Although RAG reduces hallucinations by grounding responses in retrieved documents, it does not eliminate them completely. The model may still generate unsupported or incorrect information, especially when the retrieved context is ambiguous, contradictory, or insufficient to answer the query accurately.

5. Keeping Knowledge Bases Up to Date
A RAG system is only as reliable as the knowledge base it retrieves from. If new documents are not regularly added or outdated information is not removed, the system may provide obsolete or inaccurate responses. Maintaining an up-to-date knowledge base requires continuous document ingestion, indexing, and re-embedding, which can be computationally expensive and operationally challenging.

When RAG is not the right solution

While Retrieval-Augmented Generation (RAG) is highly effective for applications that rely on external knowledge, it is not the ideal solution for every use case. If the required information is already contained within the model's training data and does not change frequently, a standard Large Language Model (LLM) is often sufficient and avoids the additional complexity of document retrieval. RAG may also be unnecessary for tasks such as creative writing, brainstorming, code generation, or conversational assistance, where responses depend more on reasoning and language generation than on retrieving factual information. Additionally, implementing and maintaining a RAG system involves extra infrastructure, including document preprocessing, vector databases, embedding generation, and regular knowledge base updates. For small-scale applications or domains with limited and static data, these costs may outweigh the benefits, making a standalone LLM a simpler and more practical choice.