RAG is one of the first things we’d understand. Most AI products are not just “ChatGPT wrappers.” They become valuable because they answer questions about company-specific data.
Examples:
- Internal documentation
- HR policies
- Product manuals
- Customer support articles
- Legal contracts
- Medical records
- Source code
- Jira tickets
- Slack messages
- GitHub repositories
ChatGPT doesn’t know these documents. That’s where RAG comes in.
Goal
By the end of today, you should confidently answer:
- What is RAG?
- Why do we need RAG?
- What are embeddings?
- Why can’t we just send an entire PDF to the LLM?
- What is semantic search?
- What is a vector database?
- Why is
pgvectorpopular in Rails? - How would you build a document chat system?
Part 1 – Why LLMs Alone Are Not Enough
Imagine you build an HR chatbot.
The user asks:
“How many annual leave days do employees receive?”
Your company’s HR policy says:
24 days.
But the LLM was trained months ago and has never seen your HR document.
Without access to your data, it has to guess—or say it doesn’t know.
This is the fundamental problem RAG solves.
Part 2 – What is RAG?
RAG = Retrieval-Augmented Generation
Break it down:
- Retrieval → Find relevant information.
- Augmented → Add that information to the prompt.
- Generation → The LLM generates the final answer using that context.
The key idea:
The LLM isn’t expected to know everything—it is given the right information at request time.
High-Level Flow
User Question │ ▼Retrieve Relevant Documents │ ▼Add Documents to Prompt │ ▼LLM Generates Answer │ ▼User
Notice that the LLM doesn’t search your database directly.
Your Rails application retrieves the data first.
Int. Question
What is RAG?
A strong answer:
Retrieval-Augmented Generation is a technique where relevant external information is retrieved first and then supplied to the language model as context, allowing it to answer questions using current or private data.
Part 3 – Why Not Paste the Entire PDF?
A common beginner idea is:
“I’ll upload the whole manual to ChatGPT.”
Let’s say your PDF is:
- 800 pages
- 350,000 words
Problems:
1. Context Window Limits
The entire document may not fit into the model’s context window.
2. Cost
More tokens = higher API cost.
3. Speed
Larger prompts take longer to process.
4. Noise
Most of the document is irrelevant to the user’s question.
If someone asks:
“How do I reset my password?”
Why send 800 pages?
You only need the page that explains password resets.
Part 4 – The RAG Pipeline
This is one of the most important diagrams to remember.
PDF↓Extract Text↓Split into Chunks↓Generate Embeddings↓Store in Vector Database──────────────User Question↓Generate Query Embedding↓Similarity Search↓Top Matching Chunks↓LLM↓Answer
Every production RAG system follows a variation of this flow.
Part 5 – What Are Chunks?
Large documents are split into smaller pieces.
Example:
Instead of:
Employee Handbook(350 pages)
Split into:
Chunk 1Company Introduction---------------Chunk 2Leave Policy---------------Chunk 3Medical Insurance---------------Chunk 4Travel Policy
Now retrieval becomes efficient.
Why Not One Sentence Per Chunk?
Very small chunks:
- lose context
Very large chunks:
- increase cost
- contain unrelated information
Chunk size is a trade-off.
Part 6 – What Are Embeddings?
This is the concept that many developers initially find abstract.
Think of an embedding as a numeric representation of meaning.
The model converts text into a list of numbers.
For example (illustrative only):
"Ruby on Rails"↓[0.12, -0.44, 0.91, ...]
Another phrase:
"Rails Framework"↓[0.13, -0.43, 0.90, ...]
Even though the wording is different, the vectors end up close together because they have similar meaning.
The exact numbers don’t matter—you just need to know that similar meanings produce similar vectors.
Think of a Map
Imagine a map.
Nearby cities are close.
Faraway cities are distant.
Embeddings work similarly.
Ruby Rails SinatraPythonCookingFootball
Ruby and Rails are “near” each other.
Cooking is far away.
The model has learned semantic relationships.
Int. Question
What is an embedding?
Good answer:
An embedding is a numerical vector that represents the semantic meaning of text, allowing similar concepts to be located near each other in vector space.
Part 7 – Semantic Search
Traditional SQL search:
WHERE title LIKE '%Rails%'
This only matches literal text.
Suppose your document says:
Ruby web framework
The user searches:
Rails
A keyword search may miss it.
Semantic search compares meaning, not exact words.
Example:
Document:
Ruby web framework
Query:
Rails
Keyword search: ❌ No match (depending on the implementation)
Semantic search: ✅ High similarity because the concepts are closely related.
Rails Analogy
Traditional search:
LIKEILIKE
Semantic search:
Embedding↓Vector Similarity↓Closest Meaning
That’s the major difference.
Part 8 – Vector Databases
Where do we store embeddings?
Inside a vector database.
Popular options:
- pgvector (PostgreSQL extension)
- Pinecone
- Qdrant
- Weaviate
- Milvus
Why pgvector Is Popular in Rails
Because many Rails applications already use PostgreSQL.
Instead of introducing another database, you can extend PostgreSQL with vector support.
Benefits:
- One database
- Familiar tooling
- ActiveRecord support
- Simpler backups
- Easier deployment
For many Rails applications, pgvector is an excellent first choice.
How Similarity Search Works
Suppose the user asks:
Password reset
The query becomes an embedding.
The database compares it with stored document embeddings.
Password Policy0.98-----------Leave Policy0.31-----------Travel Policy0.22-----------Insurance0.12
The most similar chunks are returned.
Those chunks are added to the prompt.
Part 9 – Complete Rails Architecture
A production Rails application might look like this:
Browser↓Rails Controller↓Question Service↓Embedding API↓pgvector Search↓Top 5 Chunks↓Prompt Builder↓LLM API↓Answer↓Store Conversation↓Browser
Notice that Rails coordinates every step.
The LLM is only responsible for generating the final answer.
Part 10 – RAG vs Fine-Tuning
A very common interview question.
RAG
- External knowledge
- Easy to update
- Great for company documents
- No model retraining
Fine-Tuning
- Changes model behaviour
- Expensive
- Longer process
- Better for specialised tasks or consistent output style
Rule of thumb:
If the knowledge changes frequently (documentation, policies, support articles), use RAG.
Part 11 – Example: Company Wiki Chatbot
Suppose your company has:
- 2,000 documentation pages
The user asks:
“How do I deploy staging?”
Flow:
User↓Embedding↓Vector Search↓Deployment Guide↓LLM↓Answer
The LLM answers using your company’s deployment guide rather than guessing.
Part 12 – Where Does Sidekiq Fit?
Another practical interview topic.
Generating embeddings for thousands of documents can take time.
A common approach:
PDF Uploaded↓Active Job / Sidekiq↓Extract Text↓Split Chunks↓Generate Embeddings↓Store in pgvector
Keep the upload request fast and process indexing asynchronously.
Part 13 – Common RAG Mistakes
Sending Entire Documents: Slow and expensive.
Tiny Chunks: Not enough context.
Huge Chunks: Too much irrelevant information.
Never Updating Embeddings: If documents change, regenerate the affected embeddings.
Blind Trust: Retrieved text can also be outdated or incorrect.
Validate your data sources and refresh them when needed.
Imp. Questions
Practice answering these.
Fundamentals
- What is RAG?
- Why do we need RAG?
- Why can’t ChatGPT answer company-specific questions by default?
- Why not send an entire PDF?
Embeddings
- What is an embedding?
- Why are embeddings useful?
- What is semantic search?
Databases
- What is a vector database?
- Why use pgvector?
- How does similarity search work?
Rails
- Where would Sidekiq fit?
- How would you build a document chatbot?
- Would you store conversations?
- How would you update embeddings when documents change?
Practical Exercise 1
Think about a support portal.
The documents include:
- Refund policy
- Shipping policy
- Returns
- Coupons
- Warranty
Now answer:
“My order arrived damaged.”
Which document(s) should your RAG system retrieve before asking the LLM to generate a response?
Explain why.
Practical Exercise 2
Design the Rails models for a document chat system.
For example, think about models such as:
DocumentDocumentChunkConversationMessage
What responsibilities should each have?
Practical Exercise 3
Sketch a background job flow.
When a user uploads a PDF:
- What happens immediately?
- What should Sidekiq handle?
- When are embeddings created?
- When are they stored?
- What happens if embedding generation fails?
Think in terms of a production-ready system rather than just happy-path code.
Homework
- Draw the complete RAG pipeline from memory.
- Explain embeddings in your own words without using AI jargon.
- Explain semantic search versus keyword search.
- Explain why
pgvectoris a good fit for many Rails applications. - Describe how Sidekiq helps during document ingestion.
- Answer all 14 interview questions aloud.
Int. Challenge
Imagine you’re asked this in an interview:
“We have a Rails application with 500,000 product manuals. Users should be able to ask questions about any manual. Design the system.”
A strong answer would include:
- Rails as the orchestration layer
- Background jobs for document ingestion
- Chunking strategy
- Embedding generation
pgvector(or another vector database)- Similarity search
- Prompt construction
- LLM generation
- Conversation storage
- Caching and monitoring
- Security and access control (users should only retrieve documents they are authorized to access)
This kind of end-to-end system design discussion is what distinguishes a senior engineer from someone who has only experimented with AI APIs.
Day 4 Preview
Tomorrow we move from concepts to implementation:
Building AI Features in Ruby on Rails
We’ll cover:
- AI architecture in Rails
- Choosing Ruby AI libraries and SDKs
- Service objects for AI integration
- Streaming AI responses
- Background jobs with Sidekiq
- Conversation storage
- Cost optimization
- Error handling
- Designing a production-ready AI service layer
- A complete Rails AI project structure suitable for real-world applications
From Day 4 onward, the bootcamp becomes much more code-focused and closely aligned with the kinds of AI features senior Rails developers build in production.
Happy AI Learning! 🚀