How to build a RAG application with Langchain

Brain John Aboze

October 28, 2024 | 5.5 mins

This blog post was written by Brain John Aboze as part of the Deepchecks Community Blog. If you would like to contribute your own blog post, feel free to reach out to us via blog@deepchecks.com. We typically pay a symbolic fee for content that's accepted by our reviewers.

Introduction

The rise of intelligent applications in information retrieval, question-answering, text classification, dialogue systems, fact-checking, and summarization highlights the growing importance of advanced, responsive systems. Retrieval-augmented generation (RAG) is emerging as a powerful solution across these domains. Despite its potential, building effective RAG systems can be complex. However, frameworks like LangChain make the process more manageable and efficient. This article explores building naive and advanced RAG applications using LangChain and evaluates these systems to help you harness the full potential of this powerful technology.

What is LangChain?

LangChain is an open-source framework designed to help developers create applications with large language models (LLMs). Available in Python and JavaScript (TypeScript), it’s designed to bridge the gap between powerful AI models and real-world data. LangChain allows adjustments of LLMs through integrating external data sources, so developers can come up with applications that are not only intelligent but also contextually relevant and customized to meet particular requirements. LangChain provides a toolkit of essential components, which includes:

Document loaders to bring in your data
Text splitters to break down large documents into smaller chunks
Embedding models to convert text into a numeric format known as embeddings that LLM can work with
Vector store for efficient storage, organization, and retrieval of embeddings
Indexing ensures that everything in the vector store is organized and accessible
Retrievers to find the most relevant data for each query

LangChain retrieval components, LangChain

If you’re looking for a deeper dive into retrieval-augmented generation (RAG), I highly recommend checking out our previous article, “ Q&A Using RAG: Possible Problems and Efficient Evaluation.” That article investigates the core features of RAG architecture, teaching its fundamental ideas. It also explains different types of RAG, emphasizing that efficient evaluation is crucial for performance and reliability.

Prerequisites & Getting Started

Prerequisites

Python 3.8 and higher
API keys: You will need API keys from the following services:

GROQ: Obtain your API key from GroqCloud. We will utilize open-source LLM models hosted on the platforms.

Nomic: Sign up at Nomic’s Atlas platform and generate an API key. We will use its embedding model.

Getting started

Create and activate a Python virtual environment.

python -m venv env

Activate the virtual environment:

Windows:

.\env\Scripts\activate

macOS/Linux:

source env/bin/activate

Next, install the necessary packages:

pip install langchain langchain-groq langchain-chroma langchain-nomic langchain-community arxiv pymupdf flashrank streamlit

Ensure the following environment variables are set with your specific values:

export NOMIC_API_KEY=<your_nomic_api_key>
export GROQ_API_KEY=<your_groq_api_key>

Next, we import the necessary packages for the application.

from langchain_groq import ChatGroq
from langchain_community.document_loaders import ArxivLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_nomic import NomicEmbeddings
from langchain_chroma import Chroma
import langchain.hub as hub
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import FlashrankRerankimport streamlit as st

Step-by-step guide to building a RAG application

Setting up the data loader

The first crucial step in building a RAG application is to prepare and structure the data that will be used for retrieval and generation. In this example, we’ll use the ArxivLoader, a tool designed to pull data from arXiv, an open-access archive containing over 2 million scholarly articles.

def arxiv_loader(query):
    loader = ArxivLoader(
        query=query
    )
    docs = loader.load()
    return docs

Text-splitting strategy

With the documents loaded, we need to break the documents into small chunks so the RAG system can process and retrieve relevant information more efficiently. Smaller chunk sizes make it easier to search and match but may not provide sufficient context, while larger chunk sizes can capture more relevant information as well as noise. Choosing the best strategy can be challenging. LangChain provides different text-splitting strategies, but using the RecursiveCharacterTextSplitter is highly recommended; we will use it as follows:

def get_documents_splits(documents, chunk_size=1000, chunk_overlap=200):
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap, add_start_index=True)
    splits = text_splitter.split_documents(documents)
    return splits

Integrating embedding model and vector store

With our document chunked, the next step is to convert the text chunks to a numeric/vector representation (embeddings) to aid efficient storage and retrieval in the vector store. We will use the nomic-embed-text-v1.5 embedding, since it is frequently used with open-source LLMs and has cost benefits. It comes in different dimensions; for our purposes, 765 dimensions are sufficient to capture enough semantic details and maintain efficient processing. After embedding the text chunks, the next step is to store these embeddings in a vector store. Chroma is a popular vector store that is designed to get started quickly and provides efficient storage and retrieval.

def embed_and_store(splits):
    vectorstore = Chroma.from_documents(
        documents=splits,
        embedding=NomicEmbeddings(
            model="nomic-embed-text-v1.5",
            dimensionality=768,
            inference_mode='remote'
        )
    )
    return vectorstore

Setting up the Retriever

LangChain provides a wrapper around the vector store class to make a vector store a retriever that performs queries in the vector store using similarity searches. In this case, it is going to retrieve 20 text chunks for every given query.

def get_retriever(vectorstore):
    retriever = vectorstore.as_retriever(
        search_type="similarity",
        search_kwargs={"k": 20}
    )
    return retriever

Retriever optimization

We can perform post-retrieval optimization using FlashRank to perform contextual compression. FlashRank summarizes or filters the documents based on the specific query and reduces the amount of irrelevant information added to the context for generation.

def get_contextual_compression(retriever):
    compressor = FlashrankRerank()
    compression_retriever = ContextualCompressionRetriever(
        base_compressor=compressor, base_retriever=retriever
    )
    return compression_retriever

Create the RAG chain

Next, we chain the query, retrieve documents and prompts, and pass them through the LLM (Meta’s llama-3.1-8b) to generate the respective output. This can be achieved using the LCEL runnable protocol to define the chain. We can define our own prompt for the RAG pipeline or use a predefined one from LangChain Hub— rag-prompt.

def create_rag_chain(retriever):
    prompt = hub.pull("rlm/rag-prompt")
    llm = ChatGroq(model='llama-3.1-8b-instant')
    rag_chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )
    return rag_chain

Define the main function

This function will handle the entire process, from loading and processing the data to retrieving relevant information and generating a response using your RAG system.

def main(query):
    docs = arxiv_loader("LLM for requirement engineering")
    docs_splits = get_documents_splits(docs, 1000, 200)
    vectorstore = embed_and_store(docs_splits)
    retriever = get_retriever(vectorstore)
    compression_retriever = get_contextual_compression(retriever)
    rag_chain = create_rag_chain(compression_retriever)
    llm_response = rag_chain.invoke(query)
    print(llm_response)

Query the RAG pipeline.

We can query our RAG pipeline by simply passing a query to the main function and receiving a response as follows:

main(query="How can LLMs be used in requirement analysis and specification? Please provide a list with brief descriptions of each application")

Which yields the following output:

Building a simple application with Streamlit

Streamlit is an open-source Python framework that allows you to build interactive web applications quickly and easily. We can quickly wrap the code we have written in Streamlit for a research query application. All we need to do is update the main function and define the interface and inputs of our application as follows:

def main(query, question):
    docs = arxiv_loader(query)
    docs_splits = get_documents_splits(docs, 1000, 200)
    vectorstore = embed_and_store(docs_splits)
    retriever = get_retriever(vectorstore)
    compression_retriever = get_contextual_compression(retriever)
    rag_chain = create_rag_chain(compression_retriever)
    llm_response = rag_chain.invoke(question)
    return llm_response

if __name__ == "__main__":
    st.title("Research Paper Query & Analysis")
    # User input
    research_query = st.text_input("Enter your research query:", value="LLM for requirement engineering")
    research_question = st.text_input("Enter your research question:", value="How can LLMs be used in requirement analysis and specification? Please provide a list with brief descriptions of each application")
    if st.button("Run Query"):
        with st.spinner("Fetching and processing data..."):
            result = main(research_query, research_question)
            st.success("Query complete!")
            st.write(result)

Now, run the app using the command as my script is named rag.py:

python -m streamlit run rag.py

Final Notes

The rise of intelligent applications for information retrieval, question-answering, and summarization in decision-making portends a radical shift to more advanced and responsive systems. While it is not easy to develop a RAG system as it has many moving parts, LangChain has today made it easier through the implementation of a powerful framework connecting LLMs to real-world data.

For a more in-depth perspective on RAG’s problems and evaluation methods, see our prior publication, “ Q&A Using RAG: Possible Problems and Efficient Evaluation.” LangChain makes it really easy for you to create strong, contextually aware apps that fully utilize RAG technology.