Retrieval-Augmented Generation

Enhance your agent with large knowledge bases using RAG.

Overview

Retrieval-Augmented Generation (RAG) enables your agent to access and use large knowledge bases during conversations. Instead of loading entire documents into the context window, RAG retrieves only the most relevant information for each user query, allowing your agent to:

  • Access much larger knowledge bases than would fit in a prompt
  • Provide more accurate, knowledge-grounded responses
  • Reduce hallucinations by referencing source material
  • Scale knowledge without creating multiple specialized agents

RAG is ideal for agents that need to reference large documents, technical manuals, or extensive knowledge bases that would exceed the context window limits of traditional prompting. RAG adds on slight latency to the response time of your agent, around 500ms.

How RAG works

When RAG is enabled, your agent processes user queries through these steps:

  1. Query processing: The user’s question is analyzed and reformulated for optimal retrieval.
  2. Embedding generation: The processed query is converted into a vector embedding that represents the user’s question.
  3. Retrieval: The system finds the most semantically similar content from your knowledge base.
  4. Response generation: The agent generates a response using both the conversation context and the retrieved information.

This process ensures that relevant information to the user’s query is passed to the LLM to generate a factually correct answer.

When RAG is enabled, the size of knowledge base items that can be assigned to an agent is increased from 300KB to 10MB

Guide

Prerequisites

1

Enable RAG for your agent

In your agent’s settings, navigate to the Knowledge Base section and toggle on the Use RAG option.

Toggle switch to enable RAG in the agent settings
2

Configure RAG settings (optional)

After enabling RAG, you’ll see additional configuration options:

  • Embedding model: Select the model that will convert text into vector embeddings
  • Maximum document chunks: Set the maximum amount of retrieved content per query
  • Maximum vector distance: Set the maximum distance between the query and the retrieved chunks

These parameters could impact latency. They also could impact LLM cost which in the future will be passed on to you. For example, retrieving more chunks increases cost. Increasing vector distance allows for more context to be passed, but potentially less relevant context. This may affect quality and you should experiment with different parameters to find the best results.

RAG configuration options including embedding model selection
3

Knowledge base indexing

Each document in your knowledge base needs to be indexed before it can be used with RAG. This process happens automatically when a document is added to an agent with RAG enabled.

Indexing may take a few minutes for large documents. You can check the indexing status in the knowledge base list.

4

Configure document usage modes (optional)

For each document in your knowledge base, you can choose how it’s used:

  • Auto (default): The document is only retrieved when relevant to the query
  • Prompt: The document is always included in the system prompt, regardless of relevance, but can also be retrieved by RAG
Document usage mode options in the knowledge base

Setting too many documents to “Prompt” mode may exceed context limits. Use this option sparingly for critical information.

5

Test your RAG-enabled agent

After saving your configuration, test your agent by asking questions related to your knowledge base. The agent should now be able to retrieve and reference specific information from your documents.

API implementation

You can also implement RAG through the API:

1import requests
2
3# First, index a document for RAG
4
5document_id = "your-document-id"
6response = requests.post(
7  f"https://api.elevenlabs.io/v1/conversational-ai/knowledge-base/{document_id}/rag-index",
8  headers={"xi-api-key": "your-api-key"},
9  json={"model": "e5_mistral_7b_instruct"}
10)
11
12# Check indexing status
13while True:
14  if response.json()["status"] in ["SUCCEEDED", "FAILED"]:
15      break
16  time.sleep(5) # wait 5 seconds before checking status again
17  response = requests.get(
18    f"https://api.elevenlabs.io/v1/conversational-ai/knowledge-base/{document_id}/rag-index",
19    headers={"xi-api-key": "your-api-key"}
20  )
21
22
23# Then update agent configuration to use RAG
24
25agent_id = "your-agent-id"
26agent_config = requests.get(
27  f"https://api.elevenlabs.io/v1/conversational-ai/agents/{agent_id}",
28  headers={"xi-api-key": "your-api-key"}
29).json()
30
31# Enable RAG in the agent configuration
32
33agent_config["agent"]["prompt"]["rag"] = {
34    "enabled": True,
35    "embedding_model": "e5_mistral_7b_instruct",
36    "max_documents_length": 10000
37}
38
39# Update document usage mode if needed
40
41for i, doc in enumerate(agent_config["agent"]["prompt"]["knowledge_base"]):
42    if doc["id"] == document_id:
43        agent_config["agent"]["prompt"]["knowledge_base"][i]["usage_mode"] = "auto"
44
45# Update the agent configuration
46
47requests.put(
48    f"https://api.elevenlabs.io/v1/conversational-ai/agents/{agent_id}",
49    headers={"xi-api-key": "your-api-key"},
50    json=agent_config
51)
Built with