This tutorial guides you through creating a powerful Retrieval-Augmented Generation (RAG) agent using Prisme.ai’s AI Knowledge product. You’ll learn how to build an AI that can intelligently access and leverage your organization’s documents to provide accurate, contextual responses—all without writing a single line of code.

​
What You’ll Build

A complete RAG system with:

  • A customizable knowledge base containing your documents
  • Vector embedding and intelligent retrieval capabilities
  • An interactive chat interface for querying your documents
  • Automated testing for continuous quality assurance
  • Analytics for performance monitoring

AI Knowledge functions as a RAG-as-a-Service offering, creating a collaborative environment where Data Scientists, Data Engineers, Developers, and Business Teams can efficiently work together. The platform features a unique methodology called Gen.AI Test-Driven Building, ensuring everyone operates from a single source of truth.

​
Prerequisites

Before starting this tutorial, make sure you have:

  • Access to the AI Knowledge product (self-hosted or Cloud version)
  • Basic understanding of prompting concepts
  • Documents to integrate (web pages, PDFs, Word documents, PowerPoint presentations, etc.)

​
Step 1: Understanding the AI Knowledge Interface

Let’s familiarize ourselves with the AI Knowledge product’s interface and main components:

1

Access AI Knowledge

Log in to your Prisme.ai account and navigate to the AI Knowledge product

2

Explore the Navigation

The interface consists of several key sections:

  • Agent: Configure AI models, select embedding models, and set parameters
  • Test: Configure and manage automated tests for your AI
  • Analytics: Monitor KPIs like token usage, generated answers, and daily usage
  • Help: Access embedded product documentation

​
Step 2: Creating Your RAG Project

Now, let’s set up your first RAG project in AI Knowledge:

1

Access Your Dashboard

Navigate to the AI Knowledge dashboard

2

Create a New Agent

Click the “Create Agent” button to start building your RAG agent

3

Configure Project Settings

Enter a name and description for your project that reflects its purpose

​
Step 3: Understanding the RAG Architecture

AI Knowledge uses a Retriever-Augmented Generation architecture that combines document retrieval with language generation:

This architecture allows your agent to:

  1. Process and store documents as vector embeddings
  2. Retrieve relevant document chunks based on user queries
  3. Generate contextually accurate responses using LLMs
  4. Provide sources and citations for transparency

AI Knowledge is designed to be model-agnostic, ensuring compatibility across various LLMs for both embedding and generative tasks. This flexibility allows you to switch models as needed without rebuilding your entire knowledge base.

​
Step 4: Integrating Documents

Let’s add documents to your knowledge base:

1

Access Document Integration

In your project, navigate to the document integration section

2

Upload Documents

Click the “Add Document” button and upload files from your computer, or provide URLs to web content

3

Configure Document Processing

Adjust the document processing settings:

  • Text Splitter: Choose between static (fixed-size chunks) or dynamic (content-aware) segmentation
  • Chunk Size: Set the optimal size for document segments
  • Overlap: Configure how much content overlaps between chunks to maintain context
4

Process Documents

Start the document processing, which includes:

  • Document loading and parsing
  • Text extraction and cleaning
  • Segmentation into chunks
  • Vector embedding generation
  • Metadata extraction

​
Step 5: Testing Your RAG Agent

Now that your documents are integrated, you can start querying your knowledge base:

1

Access the Chat Interface

Click on the “Chat” button to access the interactive interface

2

Ask Questions

Enter questions related to your documents to see how the AI responds

3

Review Sources and Chunks

Examine the sources and chunks used to generate each response

AI Knowledge provides full transparency by showing which document chunks were used to generate each response. This feature allows you to verify the AI’s reasoning and ensure accuracy.

​
Step 6: Implementing Test-Driven Building

A unique aspect of AI Knowledge is its support for Test-Driven Building (TDB), which ensures your RAG agent performs consistently over time:

1

Access the Testing Module

Navigate to the “Test” section in your project

2

Create a Test Suite

Click “Add Test” to create a new test suite for your RAG agent

3

Define Test Cases

Add test cases with:

  • Questions: Queries that users might ask your agent
  • Expected Answers: The correct information that should be provided
4

Configure Test Settings

Set how often tests should run (daily, weekly, etc.) and whether they should trigger automatically

5

Run Tests

Execute the tests to evaluate your agent’s performance

6

Review Results

Analyze the test results, which include:

  • Prompt and Context: The exact prompt and context used for generating responses
  • Generated Response: The AI’s actual response
  • Response Evaluation: Rating of response quality (poor, correct, good)
  • Context Evaluation: Rating of context relevance and accuracy

Business experts can manually assess responses, context, and instances of hallucination. We recommend running tests after each modification to the model, prompt, AI parameters, chunk size, image inclusion settings, self-query configuration, and other adjustments.

​
Step 7: Advanced Customization Options

After setting up your basic RAG agent, you can explore advanced customization options:

1

Access the Builder

Navigate to the Builder product on the Prisme.ai Platform

2

Implement Custom Logic

Create automations with Python or NodeJS using frameworks like LlamaIndex or Langchain

3

Connect to AI Knowledge

Copy the automation URL and go to Advanced > API & Webhook in AI Knowledge

4

Enable Webhook

Enter your automation URL and subscribe to events like questions asked, documents added/deleted/updated, and test runs

​
Step 8: Handling PII and Safety Controls

For organizations dealing with sensitive information, AI Knowledge offers options for privacy and safety:

1

Implement PII Controls

Use the Webhook functionality to detect and handle personally identifiable information

2

Configure Data Anonymization

Decide whether to anonymize personal data based on your use case

3

Add Custom Safety Controls

Integrate specific safety measures tailored to your organization’s compliance requirements

​
Step 9: Monitoring and Maintenance

Keep your RAG agent performing optimally with regular monitoring:

1

Review Analytics

Check the Analytics section to monitor usage patterns, token consumption, and response quality

2

Set Up Alerts

Use the Webhook feature with Builder to implement alerts on platforms like Slack, Teams, or Jira

3

Regular Maintenance

Update your knowledge base as new documents become available or existing information changes

4

Performance Optimization

Based on analytics and test results, adjust settings to improve response quality and efficiency

​
Best Practices for High-Performance RAG

To maximize the effectiveness of your RAG agent, consider these recommendations:

Knowledge Architecture

Design a well-organized structure for easy access and interpretation by AI systems

Data Quality

Ensure data is accurate, well-formatted, and relevant—this accounts for ~80% of success

Hybrid Technologies

Consider combining NLU for tag extraction with LLMs for generation to improve efficiency

Continuous Optimization

Regularly monitor response times, accuracy, and retrieval effectiveness

​
Wrapping Up

By following this tutorial, you’ve created a powerful RAG agent that can access and leverage your organization’s knowledge without requiring any coding. Your AI can now provide accurate, contextual responses based on your documents, with full transparency about its sources and reasoning.

The Test-Driven Building approach ensures your agent maintains high performance over time, while the flexible architecture allows you to adapt and extend its capabilities as your needs evolve.

​
Next Steps

Document Classification

Learn how to automatically classify and organize your documents with AI

AI Contact Routing

Create an intelligent contact form that routes inquiries to the right department

Webhook Integration

Build advanced integrations between your RAG agent and other systems

Website to RAG

Turn website content into a comprehensive knowledge base for your RAG agent

Was this page helpful?

Raise issue