Building a RAG Chatbot from Scratch

In this comprehensive tutorial, we’ll walk through building a complete Retrieval Augmented Generation (RAG) chatbot using Vellum’s Workflow Builder. RAG systems combine the power of Large Language Models with external knowledge sources to provide more accurate, context-aware responses while reducing hallucinations.

By the end of this tutorial, you’ll have built a chatbot that:

• Maintains conversation context across multiple turns
• Searches through your document knowledge base
• Provides responses grounded in your specific content
• Gracefully handles questions outside its knowledge scope

What You’ll Learn

Part 1: Building the Foundation

Step 1: Create Your Workflow

Start by creating a new Workflow in Vellum. You’ll begin with an empty canvas containing just an Entrypoint node.

Step 2: Set Up Workflow Inputs

First, configure your workflow to accept chat history as input:

1. Click on the Inputs tab in the left sidebar
2. Click Add and select Chat History from the dropdown
3. Add a test message to simulate user input

Step 3: Add and Configure the Prompt Node

1. Drag from the Entrypoint node to create a connection and select Prompt from the node panel
2. In the Prompt Node configuration, update the system prompt to include context constraints:

Please answer the user's questions, but only use the <context> you're given below. If you can't answer their questions with the <context>, please say "Sorry, I'm unable to answer that."
<context>
</context>

3. Add a Chat History input variable and connect it to your workflow’s chat history
4. Connect the Prompt Node to the Final Output Node

Step 4: Test the Basic Setup

Before adding RAG capabilities, test your basic chatbot:

1. Ensure your Chat History contains only one user message (remove any assistant responses from previous tests)
2. Run the workflow

You should see output similar to this, where the chatbot correctly responds that it cannot answer without context:

Part 2: Creating Your Knowledge Base

Step 5: Create a Document Index

Now we’ll create a Document Index to store your knowledge base:

1. Navigate to the Document Indexes page
2. Click Create Index in the top right corner

3. Name your index (e.g., “Countries”) and configure the settings:

4. Click Save to create your index

Step 6: Upload Documents

After creating your index, you’ll see the upload interface:

Vellum supports various file types including:

• PDF documents
• Word documents (DOCX)
• Text files (TXT)
• CSV files
• PowerPoint presentations (PPTX)
• HTML files

Drag and drop your documents or click to select files for upload.

Step 7: Preview Your Documents

Once uploaded, you can preview your documents to ensure they were processed correctly:

Part 3: Implementing RAG Functionality

Step 8: Restructure Your Workflow

Now we’ll modify the workflow to include document search:

1. Delete the existing connection between Entrypoint and Prompt Node:
   • Click on the edge between nodes
   • Press Backspace or click the Delete icon

Step 9: Add a Templating Node

We’ll use a Templating Node to extract the most recent user message for our search query:

1. Create a Templating Node between the Entrypoint and Prompt Node

2. Configure the Templating Node:
   • Connect chat_history as an input variable
   • Use this template to extract the latest user message:

1
{{ chat_history[-1].text }}

3. Optional: Rename the node to “Current User Message” for clarity

4. Test the Templating Node by running the workflow:

Step 10: Add a Search Node

1. Drag from the Templating Node to create a new connection and select Search Node
2. Configure the Search Node:
   • Document Index: Select your “Countries” index
   • Search Query: Connect to the “Current User Message” output

Step 11: Connect Search Results to the Prompt

1. Connect the Search Node to your Prompt Node
2. In the Prompt Node, add a new input variable of type String
3. Connect this input to the Search Node output

Step 12: Insert Search Results into the Prompt

1. Place your cursor between the <context> tags in your prompt
2. Press the / key to open the variable insertion dropdown
3. Select the Search Node results to insert them into the context

Part 4: Testing Your RAG Chatbot

Step 13: Run the Complete RAG Workflow

Now test your complete RAG implementation:

1. Ensure your chat history contains a question that can be answered from your documents
2. Run the workflow
3. Observe how the chatbot now provides context-aware responses based on your document content

Step 14: Test Edge Cases

Test your chatbot with various scenarios:

Advanced Enhancements

Adding Source Citations

To make your RAG chatbot even more useful, consider implementing source citations. This will be covered in a future tutorial, but you can explore:

• Using metadata from search results
• Formatting citations in responses
• Providing document references

Optimizing Search Performance

Fine-tune your RAG system by:

• Adjusting chunking strategies in your Document Index
• Experimenting with different search weights
• Implementing metadata filtering for more precise results

Evaluation and Monitoring

Consider setting up:

• RAG-specific evaluation metrics
• Online evaluations for production monitoring
• Cost tracking for your workflow executions

Key Takeaways

Next Steps

Now that you have a working RAG chatbot, consider exploring:

• Evaluating RAG Pipelines - Learn how to measure and improve your RAG system’s performance
• Advanced Chunking Strategies - Optimize how your documents are processed and stored
• Metadata Filtering - Add more sophisticated search capabilities
• Workflow Deployment - Deploy your chatbot to production

Additional Resources

• RAG System Architecture
• Search Node Documentation
• Templating Node Guide
• Document Index Management
• Prompt Engineering Best Practices