Split View: Vector Database 엔지니어 커리어 가이드: Pinecone·Weaviate·Milvus·pgvector 완전 비교와 RAG 시대의 필수 역량

Traditional databases (RDBMS) are optimized for exact value matching and range queries, using B-tree and Hash indexes.

Vector databases are optimized for similarity search among high-dimensional vectors, using ANN algorithms like HNSW and IVF, returning the "most similar" results.

Key differences:

• Query type: Exact match vs similarity search
• Indexes: B-tree/Hash vs HNSW/IVF
• Results: Exact results vs approximate results
• Data: Structured scalars vs high-dimensional vectors

HNSW is a hierarchical graph structure:

1. Build phase: Create multi-layer graphs. Upper layers contain fewer nodes (like a skip list), lower layers contain all nodes.

2. Search phase: Start from the top layer, find the nearest node, and descend layer by layer narrowing the search scope.

Key parameters:

• M: Maximum connections per node. Higher means more accurate but more memory
• ef_construction: Search range during building. Affects index quality
• ef_search: Search range during query. Accuracy-speed tradeoff

Small World property: Any two nodes in the graph can be reached through a small number of hops, making search efficient.

Cosine similarity: Compares only vector direction. Best for text embeddings. Unaffected by document length, enabling fair comparison between short sentences and long documents.

Euclidean distance: Measures absolute distance between vectors. Suitable when vector magnitude carries meaning (e.g., image feature vectors).

Dot product: Considers both direction and magnitude. Equivalent to cosine similarity for normalized vectors. Useful in recommendation systems where "intensity" of user/item matters.

Selection: Cosine similarity is the best choice for most text search use cases.

PQ is a vector compression technique:

1. Split original vectors into m sub-vectors
2. Learn codebooks via K-means in each sub-vector space
3. Replace each sub-vector with the nearest code ID
4. Store only m code IDs instead of original vectors

Example: 1536-dim vector, m=96, nbits=8

• Original: 1536 x 4 bytes = 6,144 bytes
• After PQ: 96 bytes (64x compression)

Tradeoffs:

• Pros: 10-100x memory savings, enables 1B+ vector processing
• Cons: 5-15% Recall drop, codebook learning cost
• Mitigation: OPQ (Optimized PQ), re-ranking to compensate accuracy

Vector DBs are the core component of RAG, quickly retrieving semantically relevant documents for user queries.

Optimization methods:

1. Chunking optimization: Appropriate chunk size (256-1024 tokens), overlap for context preservation
2. Embedding model selection: Choose/fine-tune embedding models suitable for the domain
3. Hybrid search: Combine vector + BM25 to complement keyword matching
4. Re-ranking: Re-sort initial search results with Cross-Encoder
5. Metadata filtering: Limit search scope by date, source, category
6. Index tuning: Adjust HNSW parameters for accuracy-speed balance

Design approach:

1. Index: IVF-PQ for minimal memory usage

   • 1B x 1536dim x 4bytes = 5.7TB (original)
   • After PQ: ~90GB (64x compression)

2. Distributed Architecture: Distribute data across 8-16 shards

   • ~6-12GB memory per shard
   • Add replicas per shard for availability

3. Query Routing: Route queries only to relevant shards for efficiency

4. Caching: Cache results for frequently searched queries

5. Batch Processing: Process vector insertions in batches to minimize index rebuild frequency

Recommended: Milvus (distributed support) + IVF-PQ index + multi-node cluster

Embedding drift occurs when embedding model outputs change over time, or data distribution shifts, degrading search quality.

Causes:

• Embedding model updates/changes
• Emergence of new domain terminology
• Data distribution changes

Response strategies:

1. Monitoring: Periodically measure Recall@K and search satisfaction
2. Benchmark sets: Regular quality measurement with fixed evaluation sets
3. Re-indexing: Regenerate all vectors when the model changes
4. Gradual updates: Test new embeddings with shadow indexes before switching
5. Version management: Record embedding model version in metadata

Hybrid search combines vector search (semantic similarity) with keyword search (BM25).

Implementation methods:

1. RRF (Reciprocal Rank Fusion): Fuse rankings from both search results
2. Weighted sum: Adjust vector/keyword weight with alpha parameter
3. Pipeline approach: First filter by keyword, then vector search

Advantages:

• Supports both exact keyword matching and semantic search simultaneously
• 10-15% Recall improvement over single-method approaches
• Handles diverse search intents

Optimal alpha values vary by domain and data, so A/B testing is recommended.

Choose pgvector when:

• Under 10 million vectors
• Already using PostgreSQL
• Need JOINs between relational data and vectors
• ACID transactions are required
• Want to minimize operational infrastructure

Choose dedicated vector DB when:

• Over 10 million vectors
• Need real-time high-performance search (P95 latency under 10ms)
• Need advanced features like hybrid search, re-ranking
• Need distributed processing
• Have a dedicated vector DB operations team

Many startups begin with pgvector and migrate to dedicated vector DBs as they scale.

Recall@K: Proportion of all relevant documents included in the top K results

• "How well did we find the documents we needed to find?"
• Key metric for vector DB index quality

Precision@K: Proportion of top K results that are actually relevant

• "How accurate are the search results?"

In vector DBs:

• ANN algorithm quality is primarily measured by Recall (how well ANN approximates exact KNN)
• In RAG systems, Recall is more important (missed documents mean the LLM cannot answer)
• After re-ranking, Precision becomes important (LLM context window is limited)

Situations requiring index rebuilds:

1. Embedding model change: When the new model has different dimensions or characteristics
2. Bulk data changes: When 30%+ of total data is deleted/modified
3. Performance degradation: When search latency or Recall falls below thresholds
4. Parameter changes: When adjusting HNSW M or ef values
5. Index type change: Switching from HNSW to IVF-PQ, etc.

Rebuild strategies:

• Blue-Green deployment: Build new index in parallel, then switch
• Gradual migration: Progressively move traffic to new index
• Offline rebuild: Batch rebuild during maintenance windows

Multimodal search maps data from multiple modalities (text, image, audio) into the same vector space for search.

Implementation methods:

1. Shared embedding model: Use CLIP, Imagebind to map text and images to the same space
2. Separate embeddings + fusion: Generate embeddings per modality, concatenate or average
3. Cross-attention based: Combine modalities via cross-attention

Vector DB implementation:

• Named vectors: Store multiple vectors per Point in Qdrant
• Separate collections: Split by modality, fuse results
• Metadata: Manage modality information as metadata

Key security considerations:

1. Access control: API key management, RBAC, multi-tenancy isolation
2. Data encryption: In-transit (TLS) and at-rest (AES-256) encryption
3. Embedding inversion: Defense against attacks that infer original text from vectors
4. Prompt injection: Defense against malicious documents being retrieved to manipulate the LLM in RAG systems
5. Data leakage prevention: Preprocessing to prevent sensitive information from being embedded
6. Audit logging: Audit trail for search queries and results

In multi-tenancy environments, data isolation between tenants is especially critical.

Cost optimization strategies:

1. Dimension reduction: Reduce dimensions with Matryoshka embeddings (1536 to 256)
2. Quantization: PQ, Binary Quantization for memory savings
3. Appropriate index selection: Index type matching scale
4. Serverless utilization: Usage-based pricing like Pinecone Serverless
5. Caching: Cache frequently searched query results
6. Batch processing: Batch non-real-time operations
7. Data lifecycle: Archive/delete old data
8. pgvector consideration: Use existing PostgreSQL for smaller scales

Chunking strategies:

1. Fixed size: Split by constant token/character count. Simple but may break context
2. Recursive splitting: Separator-based (paragraph - sentence - word). Most common
3. Semantic splitting: Split based on embedding similarity changes. High quality but expensive
4. Document structure: Leverage Markdown headers, HTML tags

Optimal chunk sizes:

• General RAG: 256-512 tokens (optimal input size for embedding models)
• Question answering: 512-1024 tokens (more context needed)
• Code search: Split by function/class units
• Legal/contracts: Split by clause units

Key: There is no fixed answer. Determine experimentally based on domain and use case.

Filtering implementations:

1. Pre-filtering: Apply metadata filter first, then ANN search on filtered vectors

   • Pro: Accurate filter application
   • Con: If few vectors remain after filter, ANN efficiency drops

2. Post-filtering: ANN search first, then apply metadata filter to results

   • Pro: ANN performance maintained
   • Con: Top K results may shrink after filtering

3. In-filter: Apply filter simultaneously during HNSW traversal

   • Pro: Balanced performance
   • Con: Complex implementation

Qdrant and Weaviate support in-filter approach for superior filtering performance.

Vector DB CRUD characteristics:

Create (Insert):

• Index update required on vector insertion
• HNSW: O(M * log(N)) per insertion — supports dynamic inserts
• IVF: Add to nearest cluster — fast but cluster imbalance possible

Read (Search):

• ANN search: O(log(N)) to O(sqrt(N))
• Metadata filter + search: varies by implementation

Update:

• Most vector DBs implement as Delete + Insert
• In-place updates difficult in HNSW

Delete:

• Soft delete followed by compaction
• Heavy deletes degrade index quality — periodic rebuilds needed

Streaming data loading strategies:

1. Micro-batch: Collect data from Kafka/Kinesis and periodically load to vector DB

   • Latency: seconds to minutes
   • Pro: Efficient, minimizes index load

2. Real-time insertion: Insert to vector DB immediately on event

   • Latency: milliseconds
   • Con: Index update overhead

3. Change Data Capture (CDC): Detect source DB changes and sync to vector DB

   • Debezium + Kafka + Vector DB pipeline

Key considerations:

• Balance between index rebuild frequency and search performance
• Duplicate detection and handling
• Retry on failure and consistency guarantees

Migration strategies:

1. Blue-Green Migration:

   • Build new vector DB in parallel
   • Gradually shift traffic to new system
   • Immediate rollback on issues

2. Phased Migration:

   • Phase 1: Replicate data to new vector DB (dual write)
   • Phase 2: Switch read traffic to new system
   • Phase 3: Switch write traffic as well
   • Phase 4: Decommission old system

3. Considerations:

   • Embedding compatibility: Full re-embedding needed if models differ
   • Schema mapping: Handle metadata structure differences
   • Performance validation: Compare Recall/Latency before and after
   • Downtime minimization: Design migration without service interruption

Key directions:

1. Native multimodal: Vector DBs that naturally integrate text/image/audio
2. Serverless universalization: Usage-based pricing and auto-scaling become standard
3. Graph + Vector integration: Native combination of knowledge graphs and vector search
4. On-device vector search: Lightweight vector search on mobile/edge devices
5. Auto-optimization: AI-based automatic index parameter tuning
6. Streaming indexing: Zero-downtime index updates for real-time data changes
7. Quantization advances: Extreme compression techniques like 1-bit quantization
8. SQL integration strengthening: pgvector performance improvements closing the gap with dedicated DBs

M determines the maximum number of connections per node. Increasing M:

• Pros: Search Recall improves (denser graph connections enable more accurate traversal)
• Cons: Memory usage increases (more edges to store)
• Build time: Index construction time increases

Generally M=16 is a good default. Increase to 32 for better accuracy, decrease to 8 to save memory.

A cosine similarity of 1 means the two vectors point in exactly the same direction. This is interpreted as the two texts (or data points) being semantically identical.

Conversely, -1 means opposite direction, and 0 means orthogonal (unrelated).

Note: A cosine similarity of 1 does not mean the vector values are identical. Vectors with different magnitudes but the same direction will have cosine similarity of 1.

pgvector is advantageous when:

1. Under 10 million vectors
2. Already using PostgreSQL, no additional infrastructure needed
3. Need JOINs between relational data and vectors
4. ACID transactions are essential
5. No dedicated vector DB operations team
6. Need a quick MVP/prototype

Key advantage: Leveraging the existing PostgreSQL ecosystem (monitoring, backup, replication) without additional infrastructure.

Chunks too large:

• Multiple topics mixed in embedding, diluting semantic representation
• Inefficient use of LLM context window (irrelevant content included)
• Precision degradation in search

Chunks too small:

• Insufficient context information degrades embedding quality
• Cannot provide complete information needed for answers
• Fragmented search results

Optimal range: 256-512 tokens is a good starting point for most RAG systems. Adjust experimentally by domain.

alpha=0.75 means vector search (semantic similarity) contributes 75% and BM25 keyword search contributes 25% to the final search score.

• alpha=1.0: Vector search only (pure semantic search)
• alpha=0.0: BM25 only (pure keyword search)
• alpha=0.75: Primarily semantic search + keyword complement

alpha=0.7-0.8 is known to be optimal for most RAG systems. It maintains semantic search advantages while supplementing with exact keyword matching.