Category: RAG

Learn AI with Rails: AI Bootcamp for Developers – RAG, Embeddings & Vector Databases – Day 3

RAG is one of the first things we’d understand. Most AI products are not just “ChatGPT wrappers.” They become valuable because they answer questions about company-specific data.

Examples:

  • Internal documentation
  • HR policies
  • Product manuals
  • Customer support articles
  • Legal contracts
  • Medical records
  • Source code
  • Jira tickets
  • Slack messages
  • GitHub repositories

ChatGPT doesn’t know these documents. That’s where RAG comes in.

Goal

By the end of today, you should confidently answer:

  • What is RAG?
  • Why do we need RAG?
  • What are embeddings?
  • Why can’t we just send an entire PDF to the LLM?
  • What is semantic search?
  • What is a vector database?
  • Why is pgvector popular in Rails?
  • How would you build a document chat system?

Part 1 – Why LLMs Alone Are Not Enough

Imagine you build an HR chatbot.

The user asks:

“How many annual leave days do employees receive?”

Your company’s HR policy says:

24 days.

But the LLM was trained months ago and has never seen your HR document.

Without access to your data, it has to guess—or say it doesn’t know.

This is the fundamental problem RAG solves.

Part 2 – What is RAG?

RAG = Retrieval-Augmented Generation

Break it down:

  • Retrieval → Find relevant information.
  • Augmented → Add that information to the prompt.
  • Generation → The LLM generates the final answer using that context.

The key idea:

The LLM isn’t expected to know everything—it is given the right information at request time.

High-Level Flow

User Question
      │
      ▼
Retrieve Relevant Documents
      │
      ▼
Add Documents to Prompt
      │
      ▼
LLM Generates Answer
      │
      ▼
User

Notice that the LLM doesn’t search your database directly.

Your Rails application retrieves the data first.

Int. Question

What is RAG?

A strong answer:

Retrieval-Augmented Generation is a technique where relevant external information is retrieved first and then supplied to the language model as context, allowing it to answer questions using current or private data.

Part 3 – Why Not Paste the Entire PDF?

A common beginner idea is:

“I’ll upload the whole manual to ChatGPT.”

Let’s say your PDF is:

  • 800 pages
  • 350,000 words

Problems:

1. Context Window Limits

The entire document may not fit into the model’s context window.

2. Cost

More tokens = higher API cost.

3. Speed

Larger prompts take longer to process.

4. Noise

Most of the document is irrelevant to the user’s question.

If someone asks:

“How do I reset my password?”

Why send 800 pages?

You only need the page that explains password resets.

Part 4 – The RAG Pipeline

This is one of the most important diagrams to remember.

PDF
↓
Extract Text
↓
Split into Chunks
↓
Generate Embeddings
↓
Store in Vector Database
──────────────
User Question
↓
Generate Query Embedding
↓
Similarity Search
↓
Top Matching Chunks
↓
LLM
↓
Answer

Every production RAG system follows a variation of this flow.

Part 5 – What Are Chunks?

Large documents are split into smaller pieces.

Example:

Instead of:

Employee Handbook
(350 pages)

Split into:

Chunk 1
Company Introduction
---------------
Chunk 2
Leave Policy
---------------
Chunk 3
Medical Insurance
---------------
Chunk 4
Travel Policy

Now retrieval becomes efficient.

Why Not One Sentence Per Chunk?

Very small chunks:

  • lose context

Very large chunks:

  • increase cost
  • contain unrelated information

Chunk size is a trade-off.

Part 6 – What Are Embeddings?

This is the concept that many developers initially find abstract.

Think of an embedding as a numeric representation of meaning.

The model converts text into a list of numbers.

For example (illustrative only):

"Ruby on Rails"
↓
[0.12, -0.44, 0.91, ...]

Another phrase:

"Rails Framework"
↓
[0.13, -0.43, 0.90, ...]

Even though the wording is different, the vectors end up close together because they have similar meaning.

The exact numbers don’t matter—you just need to know that similar meanings produce similar vectors.

Think of a Map

Imagine a map.

Nearby cities are close.

Faraway cities are distant.

Embeddings work similarly.

Ruby
      Rails
          Sinatra
Python
Cooking
Football

Ruby and Rails are “near” each other.

Cooking is far away.

The model has learned semantic relationships.

Int. Question

What is an embedding?

Good answer:

An embedding is a numerical vector that represents the semantic meaning of text, allowing similar concepts to be located near each other in vector space.

Part 7 – Semantic Search

Traditional SQL search:

WHERE title LIKE '%Rails%'

This only matches literal text.

Suppose your document says:

Ruby web framework

The user searches:

Rails

A keyword search may miss it.

Semantic search compares meaning, not exact words.

Example:

Document:

Ruby web framework

Query:

Rails

Keyword search: ❌ No match (depending on the implementation)

Semantic search: ✅ High similarity because the concepts are closely related.

Rails Analogy

Traditional search:

LIKE
ILIKE

Semantic search:

Embedding
↓
Vector Similarity
↓
Closest Meaning

That’s the major difference.

Part 8 – Vector Databases

Where do we store embeddings?

Inside a vector database.

Popular options:

  • pgvector (PostgreSQL extension)
  • Pinecone
  • Qdrant
  • Weaviate
  • Milvus

Why pgvector Is Popular in Rails

Because many Rails applications already use PostgreSQL.

Instead of introducing another database, you can extend PostgreSQL with vector support.

Benefits:

  • One database
  • Familiar tooling
  • ActiveRecord support
  • Simpler backups
  • Easier deployment

For many Rails applications, pgvector is an excellent first choice.

How Similarity Search Works

Suppose the user asks:

Password reset

The query becomes an embedding.

The database compares it with stored document embeddings.

Password Policy
0.98
-----------
Leave Policy
0.31
-----------
Travel Policy
0.22
-----------
Insurance
0.12

The most similar chunks are returned.

Those chunks are added to the prompt.

Part 9 – Complete Rails Architecture

A production Rails application might look like this:

Browser
↓
Rails Controller
↓
Question Service
↓
Embedding API
↓
pgvector Search
↓
Top 5 Chunks
↓
Prompt Builder
↓
LLM API
↓
Answer
↓
Store Conversation
↓
Browser

Notice that Rails coordinates every step.

The LLM is only responsible for generating the final answer.

Part 10 – RAG vs Fine-Tuning

A very common interview question.

RAG

  • External knowledge
  • Easy to update
  • Great for company documents
  • No model retraining

Fine-Tuning

  • Changes model behaviour
  • Expensive
  • Longer process
  • Better for specialised tasks or consistent output style

Rule of thumb:

If the knowledge changes frequently (documentation, policies, support articles), use RAG.

Part 11 – Example: Company Wiki Chatbot

Suppose your company has:

  • 2,000 documentation pages

The user asks:

“How do I deploy staging?”

Flow:

User
↓
Embedding
↓
Vector Search
↓
Deployment Guide
↓
LLM
↓
Answer

The LLM answers using your company’s deployment guide rather than guessing.

Part 12 – Where Does Sidekiq Fit?

Another practical interview topic.

Generating embeddings for thousands of documents can take time.

A common approach:

PDF Uploaded
↓
Active Job / Sidekiq
↓
Extract Text
↓
Split Chunks
↓
Generate Embeddings
↓
Store in pgvector

Keep the upload request fast and process indexing asynchronously.

Part 13 – Common RAG Mistakes

Sending Entire Documents: Slow and expensive.

Tiny Chunks: Not enough context.

Huge Chunks: Too much irrelevant information.

Never Updating Embeddings: If documents change, regenerate the affected embeddings.

Blind Trust: Retrieved text can also be outdated or incorrect.

Validate your data sources and refresh them when needed.

Imp. Questions

Practice answering these.

Fundamentals

  1. What is RAG?
  2. Why do we need RAG?
  3. Why can’t ChatGPT answer company-specific questions by default?
  4. Why not send an entire PDF?

Embeddings

  1. What is an embedding?
  2. Why are embeddings useful?
  3. What is semantic search?

Databases

  1. What is a vector database?
  2. Why use pgvector?
  3. How does similarity search work?

Rails

  1. Where would Sidekiq fit?
  2. How would you build a document chatbot?
  3. Would you store conversations?
  4. How would you update embeddings when documents change?

Practical Exercise 1

Think about a support portal.

The documents include:

  • Refund policy
  • Shipping policy
  • Returns
  • Coupons
  • Warranty

Now answer:

“My order arrived damaged.”

Which document(s) should your RAG system retrieve before asking the LLM to generate a response?

Explain why.

Practical Exercise 2

Design the Rails models for a document chat system.

For example, think about models such as:

  • Document
  • DocumentChunk
  • Conversation
  • Message

What responsibilities should each have?

Practical Exercise 3

Sketch a background job flow.

When a user uploads a PDF:

  1. What happens immediately?
  2. What should Sidekiq handle?
  3. When are embeddings created?
  4. When are they stored?
  5. What happens if embedding generation fails?

Think in terms of a production-ready system rather than just happy-path code.

Homework

  1. Draw the complete RAG pipeline from memory.
  2. Explain embeddings in your own words without using AI jargon.
  3. Explain semantic search versus keyword search.
  4. Explain why pgvector is a good fit for many Rails applications.
  5. Describe how Sidekiq helps during document ingestion.
  6. Answer all 14 interview questions aloud.

Int. Challenge

Imagine you’re asked this in an interview:

“We have a Rails application with 500,000 product manuals. Users should be able to ask questions about any manual. Design the system.”

A strong answer would include:

  • Rails as the orchestration layer
  • Background jobs for document ingestion
  • Chunking strategy
  • Embedding generation
  • pgvector (or another vector database)
  • Similarity search
  • Prompt construction
  • LLM generation
  • Conversation storage
  • Caching and monitoring
  • Security and access control (users should only retrieve documents they are authorized to access)

This kind of end-to-end system design discussion is what distinguishes a senior engineer from someone who has only experimented with AI APIs.

Day 4 Preview

Tomorrow we move from concepts to implementation:

Building AI Features in Ruby on Rails

We’ll cover:

  • AI architecture in Rails
  • Choosing Ruby AI libraries and SDKs
  • Service objects for AI integration
  • Streaming AI responses
  • Background jobs with Sidekiq
  • Conversation storage
  • Cost optimization
  • Error handling
  • Designing a production-ready AI service layer
  • A complete Rails AI project structure suitable for real-world applications

From Day 4 onward, the bootcamp becomes much more code-focused and closely aligned with the kinds of AI features senior Rails developers build in production.


Happy AI Learning! 🚀