Split View: RAGAS 완전 가이드: RAG 시스템을 어떻게 정량적으로 평가하는가

Full RAG pipeline:

[User Question]
        |
[Retrieval Step] <- Evaluated by Context Precision, Context Recall
        |
[Generation Step] <- Evaluated by Faithfulness, Answer Relevancy
        |
[Final Answer]

Question: "What is the battery life of this product?"
Context: "The battery lasts up to 12 hours. Charging time is 2 hours."
Answer: "The battery lasts 12 hours and the device is also waterproof."

Claim analysis:
- "battery lasts 12 hours" -> supported by context CHECK
- "device is also waterproof" -> not in context CROSS

Faithfulness = 1/2 = 0.5  (low! hallucination occurred)

Original question: "How long does the battery last?"
Generated answer: "This product is made with premium materials and comes in various colors."

Synthetic questions generated:
- "What materials is this product made of?"
- "What colors does this come in?"

Similarity to original question: very low
Answer Relevancy ≈ 0.1  (low! answer ignores the question)

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

# Prepare evaluation dataset
# Each sample: question + system-generated answer + retrieved contexts + ground_truth
eval_data = {
    "question": [
        "What is the battery life of this product?",
        "What is the return policy?",
        "How long does shipping take?"
    ],
    "answer": [
        "The battery lasts up to 12 hours.",
        "You can request a refund within 30 days of purchase.",
        "Standard shipping takes 3-5 business days."
    ],
    "contexts": [
        # Retrieved context chunks for each question
        [
            "The battery lasts up to 12 hours. Charging time is 2 hours.",
            "Product dimensions are 15cm x 10cm x 2cm."
        ],
        [
            "Customers can request a refund within 30 days of purchase.",
            "Refunds are processed within 5-7 business days."
        ],
        [
            "Standard shipping: 3-5 business days. Express shipping: 1-2 business days.",
            "Shipping fee of $5 applies to orders under $50."
        ]
    ],
    "ground_truth": [
        "Battery life is up to 12 hours, and charging takes 2 hours.",
        "Returns are accepted within 30 days of purchase and processed in 5-7 business days.",
        "Standard shipping takes 3-5 business days, express takes 1-2 business days."
    ]
}

dataset = Dataset.from_dict(eval_data)

# Run evaluation
results = evaluate(
    dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    ]
)

print(results)
# Sample output:
# {
#   'faithfulness': 0.95,
#   'answer_relevancy': 0.88,
#   'context_precision': 0.82,
#   'context_recall': 0.91
# }

# Convert to DataFrame for detailed analysis
df = results.to_pandas()
print(df.to_string())

from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.embeddings import LangchainEmbeddingsWrapper

# Use gpt-4o-mini to reduce evaluation costs
judge_llm = LangchainLLMWrapper(
    ChatOpenAI(model="gpt-4o-mini", temperature=0)
)

# Embeddings used for Answer Relevancy calculation
embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=judge_llm,
    embeddings=embeddings,
)

# evaluate_rag.py - evaluation script run in CI/CD

import json
import sys
from datetime import datetime
from pathlib import Path
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset


def load_test_dataset(path: str) -> Dataset:
    """Load saved test dataset from disk"""
    with open(path, 'r', encoding='utf-8') as f:
        data = json.load(f)
    return Dataset.from_dict(data)


def run_rag_on_testset(testset: Dataset, rag_chain) -> Dataset:
    """Run each test question through RAG, collect answers and contexts"""
    answers = []
    contexts_list = []

    for question in testset["question"]:
        result = rag_chain.invoke({"query": question})
        answers.append(result["result"])
        contexts_list.append([
            doc.page_content
            for doc in result["source_documents"]
        ])

    return testset.add_column("answer", answers).add_column("contexts", contexts_list)


def evaluate_and_gate(
    dataset: Dataset,
    thresholds: dict,
    save_path: str = None
) -> bool:
    """
    Run evaluation and apply pass/fail gate based on thresholds.

    thresholds example:
    {
        "faithfulness": 0.8,
        "answer_relevancy": 0.75,
        "context_precision": 0.7,
        "context_recall": 0.75
    }
    """
    results = evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )

    print("\n" + "="*60)
    print(f"RAG Evaluation Results - {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
    print("="*60)

    all_passed = True
    for metric_name, threshold in thresholds.items():
        score = results[metric_name]
        passed = score >= threshold
        status = "PASS" if passed else "FAIL"
        print(f"{metric_name:25s}: {score:.3f} (threshold: {threshold}) [{status}]")
        if not passed:
            all_passed = False

    # Save results for history tracking
    if save_path:
        result_data = {
            "timestamp": datetime.now().isoformat(),
            "scores": {k: float(v) for k, v in results.items()},
            "passed": all_passed
        }
        Path(save_path).parent.mkdir(parents=True, exist_ok=True)
        with open(save_path, 'a') as f:
            f.write(json.dumps(result_data) + "\n")

    return all_passed


if __name__ == "__main__":
    testset = load_test_dataset("tests/rag_testset.json")
    # rag_chain = load_your_rag_chain()
    # testset = run_rag_on_testset(testset, rag_chain)

    thresholds = {
        "faithfulness": 0.80,
        "answer_relevancy": 0.75,
        "context_precision": 0.70,
        "context_recall": 0.75,
    }

    passed = evaluate_and_gate(
        testset,
        thresholds=thresholds,
        save_path="results/rag_eval_history.jsonl"
    )

    sys.exit(0 if passed else 1)  # Fail the build in CI/CD if below threshold

# .github/workflows/rag-evaluation.yml
name: RAG Quality Gate

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  evaluate-rag:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: pip install ragas langchain openai datasets
      - name: Run RAG Evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python evaluate_rag.py

from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_community.document_loaders import DirectoryLoader

# Load your documents
loader = DirectoryLoader("./docs", glob="**/*.md")
documents = loader.load()

# Initialize test set generator
generator = TestsetGenerator.with_openai()

# Generate test set with specified question types
testset = generator.generate_with_langchain_docs(
    documents,
    test_size=50,
    distributions={
        simple: 0.5,        # Simple factual questions 50%
        reasoning: 0.25,    # Questions requiring reasoning 25%
        multi_context: 0.25 # Questions needing multiple docs 25%
    }
)

# Save to file
testset_df = testset.to_pandas()
testset_df.to_json("tests/rag_testset.json", orient='records')

print(f"Generated {len(testset_df)} questions")
print("\nSample questions:")
print(testset_df[['question', 'ground_truth']].head())

1. Measure baseline (run RAGAS)
        |
2. Identify lowest scoring metric
        |
3. Improve that metric (adjust chunking/retrieval/prompt)
        |
4. Measure again, confirm improvement
        |
5. Add Quality Gate to CI/CD
        |
6. Automated evaluation on every code change