Q&A using RAG: Possible problems and efficient evaluation

Brain John Aboze

November 11, 2024 | 9 mins

This blog post was written by Brain John Aboze as part of the Deepchecks Community Blog. If you would like to contribute your own blog post, feel free to reach out to us via blog@deepchecks.com. We typically pay a symbolic fee for content that's accepted by our reviewers.

Introduction

Question-answering is a central task in NLP, which is the design of systems capable of answering questions expressed by a human in natural language. More broadly, QA systems can be divided into two categories: open-domain QA, where the questions could be on practically any topic, and closed-domain QA, where they are limited to very specialized domains of knowledge. In fact, the challenge with question answering involves deducing the intention of a question and retrieving relevant information that will supply a precise answer.

Photo by Max Fischer

While Large Language Models (LLM) have been applied to QA tasks natively, they often struggle with issues like hallucinations, knowledge cut-off, and a lack of domain-specific expertise. To resolve this, Retrieval-Augmented Generation (RAG) combines external knowledge retrieval with generative capabilities, making it a cost-effective solution that often outperforms standalone LLMs. Yet, despite their benefits, RAG models present their own set of challenges.

In this article, we’ll explore the possible problems of RAG-based QA systems. Also, we will shed light on why efficient evaluation is important with respect to both retrieval and response evaluation to ensure they meet the high standards required for effective QA.

Understanding the three paradigms of RAG

It is pertinent to understand various paradigms that illustrate the evolution of RAG architecture. These can be viewed from three key phases: the early stages, the developed stage, and the stage of flexible design. Each phase marks an important new step in the capability of RAG systems.

Three paradigms of RAG, Arxiv

Before we delve further into the evolution of the RAG architecture, let’s break down and understand some core terminologies of the RAG model:

• Indexing: Indexing is a term not unique to RAGs or LLMs but associated with databases. In traditional databases, indexing involves creating a data structure that can speed up data retrieval operations by acting as a pointer to data in one or several tables within the database. Think of it like the book index that helps quickly find information without reading the entire book. In the context of RAG, indexing is a process whereby documents in various formats are extracted and converted into plain text, which is then chunked and encoded into vectors-numerical representations-using an embedding model and stored in a vector database, which will later support efficient search and retrieval.
• Retrieval: When a user provides a query, it is converted to a vector using the same encoding model that was used at indexing time. Also, the system computes the similarity of this query vector to the vectors of the indexed chunks, usually using approximate nearest neighbors algorithms. It retrieves the top K chunks that are most relevant to the given query. These chunks become the expanded context within which a response is generated.
• Generation: Finally, the query and the retrieved text chunks are combined into a single prompt in a process called retrieval fusion. An LLM then uses this prompt to generate a response/output. Depending on the task, the model might rely upon its own stored knowledge, or it might strictly constrain its attention to the information provided in the retrieved text chunks. Furthermore, if the interaction occurs within an ongoing conversation context, the model can include previous conversations/exchanges, which provides the capability for multi-turn conversation.

RAG core components, Author

Naive RAG

During the early development of RAG, the workflow was built around a straightforward process of indexing, retrieval, and generation, often known as the “retrieve-read” framework. However, this had some inherent issues. The system’s reliance on basic similarity calculations to match the query with relevant text can lead to mismatches. This approach often fails to capture the deeper meaning or context of the query, resulting in suboptimal information retrieval. The retrieval phase can retrieve redundant or noisy information or even overlook crucial information, increasing the risk of errors and hallucinated responses. It is also prone to augmentation hurdles in efficiently combining the relevant parts of the retrieved information and the given query required for generation.

Advanced RAG

Advanced RAG extends the Naive RAG baseline by changes that are actually made to maximize the quality of retrieval. Therefore, in the light shed above, the quality of retrieval would hence mean relevance in retrievals. Advanced RAG employs a two-pronged approach to optimize the retrieval phase: pre-retrieval and post-retrieval strategies to refine the input query and retrieved results, respectively, ultimately strengthening the overall retrieval workflow.

Advanced RAG pre-retrieval and post-retrieval processes, Author

Modular RAG

Modular RAG is a leap forward in the evolution of RAG architecture as it breaks down the RAG model into modules, offering a more flexible, scalable, and customizable approach to fit various requirements. The new modules include search, predict, fusion, memory, demonstration, and routing.

Modular RAG modules, Author

The search module empowers the RAG system to tap into the different data sources, spanning search engines and databases, enabling the retrieval of relevant information. The predict module renders the likelihood of redundancy or noise remote by the capability of the LLM itself to generate appropriate context, ideally when no proper fit retrieval is found. This essentially decreases irrelevance or repetition of content. The fusion module enhances the retrieval fusion process with strategies like a multi-query approach to expand user queries into diverse perspectives to be fused with the retrieved context. The memory module creates short, long, and contextual memory pools that align more closely with the query and retrieve information at hand. The demonstration module enables the RAG system to dynamically adapt to different tasks or domains by adjusting its retrieval, generation, or processing strategies. Lastly, intelligent routing allows the RAG system to navigate the best pathway for each query.

Possible Problems of RAG

Despite the enhancement to native LLMs, some challenges can impact their reliability and effectiveness.

Retrieval Challenges

• Query dependence: The naive RAG often uses the user’s query directly. If the query is ambiguous or complex, this can result in bad retrieval. Ambiguity, special vocabulary, and bad language structure may retrieve irrelevant or insufficient data.
• Chunking strategy: There are three types of text chunking: fixed length, semantic, and content-based. The simplest technique would be fixed-length chunking, wherein documents are successively divided into segments based on a given length parameter. Semantic chunking splits the document semantically. Lastly, content-based chunking parses documents according to unique structural features. There is no golden rule about choosing the size of chunks, but semantic completeness and context length should be balanced. While larger chunks capture more context and noise, smaller chunks might miss the critical information.
• Text splitting problems: During indexing, data formats being unstructured, semi-structured, and structured can be consequential as splitting text into chunks can sometimes separate important data, like tables, causing corruption during retrieval. Furthermore, tables and images that may exist in the data complicate semantic searches.
• Granularity issues: The size of the retrieved information-whether large chunks (coarse-grained) or smaller pieces (fine-grained)-can significantly impact the model’s performance. That means that coarse-grained retrieval might pull a lot of irrelevant data or noise that confuses the system, and fine-grained risks missing important context. Finding the right balance is crucial but challenging.
• Choice of embedding: The choice of the embedding model required for a RAG system is very crucial, as it directly influences the quality of information retrieval and, consequently, the generated output. Low-quality embeddings will result in bad retrieval, which goes on to produce irrelevant or inaccurate responses. When selecting an embedding model, the following factors should be considered: vector dimension, retrieval performance, model size, and whether it is public or private. Another important criterion is cost, including those associated with querying, indexing, and storage. Moreover, as the search latency grows with the dimension of embeddings, choosing lower-dimensional embeddings can help avoid some delay. It’s also essential to ensure that the embedding model supports the required languages. All these factors have to be balanced in developing an optimized RAG system to retrieve the needed information efficiently and accurately.

Retrieval challenges, Author

Generation Challenges

Not surprisingly, just like humans can be distracted by a surfeit of information, the performance of LLMs is worsened if the context is redundant or too long, causing the model to lose focus-a phenomenon known as the “lost in the middle” problem. Therefore, much care should be taken in curating the context to ensure the quality of the generated responses, prioritizing the most relevant information and minimizing excessive content.

Augmentation Challenges

Naive RAG systems often rely on a single retrieval step followed by generation, which is insufficient for complex tasks that require multi-step reasoning. This approach can lead to limited information scope, hence causing insufficient or shallow responses and over-reliance on retrieved data, leading to less creativity or generalization of responses. To solve this problem, one should use the looping flow model, which comes in various forms: iterative, recursive, and adaptive.

Loop flow pattern for retrieval augmentation process, Arxiv

Latency and Scalability Challenges

As RAG systems evolve and grow, they encounter substantial challenges related to latency and scalability. The increasing size of the document database can lead to slower retrieval processes, hindering efficient operations. Furthermore, scaling the model to accommodate more complex queries or larger datasets necessitates significant computational resources, which may not be feasible in all environments. It is essential to balance performance optimization and scalability to address these demands.

Ethical and Legal Concerns

As RAG systems grow and process more data, privacy and data security considerations become very important. Accessing sensitive data, such as personally identifiable information (PII), requires strict precautions and guardrails to avoid violating privacy laws. Failing to do so can result in substantial fines, loss of customer trust, and reputational damage. Additionally, unreliable data sources can lead to false or biased outputs, undermining the model’s credibility.

Integration and Maintenance Complexity

Maintaining and managing the various integrations within a RAG model can be challenging, especially as the system grows more complex. Integrating different vector stores, embedding models, and LLMs, along with frameworks like building RAG with LangChain and LlamaIndex, requires careful coordination to ensure seamless operation. Keeping these components in sync, updating them as needed, and troubleshooting any compatibility issues are crucial tasks requiring significant time and resources. This complexity can lead to operational inefficiencies and an increased risk of system errors if not properly managed.

RAG Evaluation

RAG evaluation targets retrieval quality and generation quality. Evaluating retrieval quality depends on determining the efficiency of the context obtained by the retriever component using metrics such as hit rate, mean reciprocal rank (MRR), mean average precision (MAP), and normalized discounted cumulative gain (NDCG). The generation quality evaluation mainly relies on whether a generator can synthesize coherent and relevant answers from its retrieval context. This evaluation examined the faithfulness (groundedness), relevance, non-harmfulness, and accuracy of the information produced by the RAG model.

To effectively evaluate RAG systems, three primary quality scores and four essential abilities are widely used in current RAG model evaluation processes. These factors provide insight into evaluating the two main RAG model targets (retrieval and generation quality). The three primary quality scores, also known as the RAG triads, include context relevance, groundedness, and answer relevance. Required abilities cover RAG evaluation on its adaptability and efficiency, including noise robustness, negative rejection, information integration, and counterfactual robustness.

RAG triads, TruLens

• Noise robustness evaluates the capacity to handle noise from the retrieval process.
• Negative rejection evaluates the model’s ability to detect when to stop answering a query when the information retrieved is insufficient to address it.
• Information Integration assesses how well the model can combine data from several documents to answer challenging queries.
• Counterfactual robustness measures the model’s capacity to identify and ignore documented document errors, even when given instructions on possible misinformation.

Evaluation Frameworks

A variety of benchmark tests and tools have been proposed to assist in the evaluation of RAG. These benchmark tools offer quantitative measures for quality assessment in diverse evaluation dimensions, as seen in the table below:

Final Notes

The core task of RAG remains question-answering (QA), which is a fundamental task in NLP. It requires a system that can understand questions as well as pick out the relevant information to give accurate answers. However, some issues are pertinent, which ongoing research in this space is trying to resolve. Some tips include optimization of user query refinement in reducing ambiguity, efficient chunking strategies for balancing the preservation of context, and superior precision of retrieval can be applied. Also, active management of context through prioritization and multistep reasoning will avoid issues like “lost in the middle.” Using scalable architecture and optimized retrieval processes can mitigate latency issues. Ethical compliance is important to building trust and credibility, especially when processing sensitive information. Efficient assessment based on clear metrics and benchmarks would be relevant to guarantee the systems’ reliability and performance.