A comprehensive course teaching data-driven approaches to building and improving Retrieval-Augmented Generation (RAG) systems. This repository contains course materials, code examples, and a companion book.

All of this material is supported by the Systematically Improving RAG Course.

Click here to get 20% off →

This course teaches you how to systematically improve RAG applications through:

• Data-driven evaluation and metrics
• Embedding fine-tuning and optimization
• Query understanding and routing
• Structured data integration
• Production deployment strategies

The core philosophy centers around the "RAG Flywheel" - a continuous improvement cycle that emphasizes:

1. Measure: Establish benchmarks and evaluation metrics
2. Analyze: Understand failure modes and user patterns
3. Improve: Apply targeted optimizations
4. Iterate: Continuous refinement based on real-world usage

.
├── cohort_1/        # First cohort materials (6 weeks)
├── cohort_2/        # Second cohort materials (weeks 0-6)
├── latest/          # Current course version with latest updates
│   ├── week_0/      # Getting started with Jupyter, LanceDB, and evals
│   ├── week_1/      # RAG evaluation foundations
│   ├── week_2/      # Embedding fine-tuning
│   ├── week_4/      # Query understanding and routing
│   ├── week_5/      # Structured data and metadata
│   ├── week_6/      # Tool selection and product integration
│   └── capstone/    # Comprehensive WildChat project
├── docs/            # MkDocs book content
├── data/            # CSV files from industry talks
├── md/              # Markdown conversions of notebooks
└── mkdocs.yml       # Documentation configuration

The course follows a 6-week structure where each week corresponds to specific workshop chapters in the companion book:

• Book Coverage: Chapter 0 (Introduction) + Chapter 1 (Starting the Flywheel with Data)
• Topics:
  • Shifting from static implementations to continuously improving products
  • Overcoming the cold-start problem through synthetic data generation
  • Establishing meaningful metrics aligned with business goals
  • RAG as a recommendation engine wrapped around language models

• Book Coverage: Chapter 2 (From Evaluation to Product Enhancement)
• Topics:
  • Transforming evaluation insights into concrete improvements
  • Fine-tuning embeddings with Cohere and open-source models
  • Re-ranking strategies and targeted capability development

• Book Coverage: Chapter 3 (UX - 3 parts)
  • Part 1: Design Principles
  • Part 2: Feedback Collection
  • Part 3: Iterative Improvement
• Topics:
  • Building interfaces that delight users and gather feedback
  • Creating virtuous cycles of improvement
  • Continuous refinement based on user interaction

• Book Coverage: Chapter 4 (Topic Modeling - 2 parts)
  • Part 1: Analysis - Segmenting users and queries
  • Part 2: Prioritization - High-value opportunities
• Topics:
  • Query classification with BERTopic
  • Pattern discovery in user queries
  • Creating improvement roadmaps based on usage patterns

• Book Coverage: Chapter 5 (Multimodal - 2 parts)
  • Part 1: Understanding different content types
  • Part 2: Implementation strategies
• Topics:
  • Working with documents, images, tables, and structured data
  • Metadata filtering and Text-to-SQL integration
  • PDF parsing and multimodal embeddings

• Book Coverage: Chapter 6 (Architecture - 3 parts)
  • Part 1: Intelligent routing to specialized components
  • Part 2: Building and integrating specialized tools
  • Part 3: Creating unified product experiences
• Topics:
  • Tool evaluation and selection
  • Performance optimization strategies
  • Streaming implementations and production deployment

A comprehensive project using the WildChat dataset that covers:

• Data exploration and understanding
• Vector database integration (ChromaDB, LanceDB, Turbopuffer)
• Synthetic question generation
• Summarization strategies
• Complete test suite implementation

• LLM APIs: OpenAI, Anthropic, Cohere
• Vector Databases: LanceDB, ChromaDB, Turbopuffer
• ML/AI Frameworks: Sentence-transformers, BERTopic, Transformers
• Evaluation Tools: Braintrust, Pydantic-evals
• Monitoring: Logfire
• Data Processing: Pandas, NumPy, BeautifulSoup
• Visualization: Matplotlib, Seaborn, Streamlit

The /docs directory contains a comprehensive book built with MkDocs that serves as the primary learning resource:

• Introduction & Core Concepts: The RAG Flywheel philosophy and product-first thinking
• Workshop Chapters (0-6): Detailed guides that map directly to each course week
• Office Hours: Q&A summaries from Cohorts 2 and 3 with real-world implementation insights
• Industry Talks: Expert presentations including:
  • RAG Anti-patterns in the Wild
  • Semantic Search Over the Web
  • Understanding Embedding Performance
  • Online Evals and Production Monitoring
  • RAG Without APIs (Browser-based approaches)

1. Product-First Thinking: Treating RAG as an evolving product, not a static implementation
2. Data-Driven Improvement: Using metrics, evaluations, and user feedback to guide development
3. Systematic Approach: Moving from ad-hoc tweaking to structured improvement processes
4. User-Centered Design: Focusing on user value and experience, not just technical capabilities
5. Continuous Learning: Building systems that improve with every interaction

To build and view the documentation locally:

mkdocs serve

1. Clone the repository
2. Install dependencies: pip install -e .
3. Start with /latest/week_0/ for the most up-to-date content
4. Follow the notebooks in sequential order within each week
5. Reference the corresponding book chapters in /docs for deeper understanding

This course emphasizes:

• Systematic Improvement: Data-driven approaches over guesswork
• Product Thinking: Building RAG systems that solve real problems
• Practical Application: Real-world datasets and examples
• Evaluation-First: Measure before and after every change
• Continuous Learning: The field evolves rapidly; the flywheel helps you adapt

• Industry talk transcripts in /data/
• Office hours recordings summaries in /docs/office_hours/
• Advanced "Kura" notebooks in /latest/kura/ for specialized topics

This is educational material for the "Systematically Improving RAG Applications" course.

Want to learn more about RAG? Take our free email course and get the latest news and information about RAG techniques and best practices.

Sign up for the free RAG Crash Course →

IF you want to get discounts and 6 day email source on the topic make sure to subscribe to