Skip to content
Published on

Why Langfuse Puts Traces in ClickHouse — How the Storage Layer Splits the Work

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Opening — There Is Not One Datastore

Most people stall at the same point the first time they self-host Langfuse: there is not one database. You need Postgres, ClickHouse, Redis, and an S3-compatible object store. Four datastores for one application looks excessive.

Recall the data model from the first post and the reason appears. Organization and project settings are low in volume and high in consistency requirements. Traces arrive by the millions per day and have to be aggregated across a month. A single raw prompt can run to several megabytes. Push all three into the same engine and none of them work well.

Component and configuration names here were verified against the official documentation on 2026-08-15. Langfuse's architecture differs by version, so re-check the docs for the version you are running. This post works from the self-hosting v4 documentation, and ClickHouse became the trace store starting with v3. If you are reading an article that describes the v2 structure, it diverges from here.

Four Datastores and Two Containers

The self-hosting overview lists six components: two application containers and four datastores.

ComponentRole as the documentation states it
Langfuse WebThe main web application serving the Langfuse UI and APIs
Langfuse WorkerA worker that asynchronously processes events
PostgreSQLThe main database for transactional workloads
ClickHouseHigh-performance OLAP database which stores traces, observations, and scores
Redis/ValkeyA fast in-memory data structure store used for queue and cache operations
S3/Blob StorageObject storage to persist incoming events, multi-modal inputs, and large exports

An LLM API gateway attaches as an optional component for the playground and for evals. According to the containers documentation, web runs the langfuse/langfuse:4 image on port 3000 and worker runs langfuse/langfuse-worker:4 on port 3030.

Stitch the roles each page assigns together and you get this picture.

                      ┌───────────────────────────────┐
   SDK / OTLP  ─────▶ │  langfuse-web        :3000    │
                      │  console and API              │
                      └──────┬──────────────────┬─────┘
                             │                  │
              persist raw    │                  │  enqueue
              events         ▼                  ▼
                  ┌────────────────────┐   ┌──────────────┐
                  │ S3 / object store  │   │ Redis/Valkey │
                  │  raw events        │   │  queue+cache │
                  │  multi-modal input │   └──────┬───────┘
                  │  batch exports     │          │
                  └────────────────────┘          ▼
                                        ┌───────────────────────┐
                                        │ langfuse-worker :3030 │
                                        │ async processing      │
                                        └──────┬────────┬───────┘
                                               │        │
                                               ▼        ▼
                              ┌────────────────────┐  ┌──────────────────┐
                              │ ClickHouse         │  │ PostgreSQL       │
                              │  traces            │  │  orgs, projects  │
                              │  observations      │  │  datasets        │
                              │  scores            │  │  encrypted keys  │
                              └────────────────────┘  └──────────────────┘

The Ingestion Path — The Order an Event Travels

The key to this structure is one sentence in the cache documentation: Redis is used to accept new events quickly on the API and defer their processing and insertion, which is what lets the system handle request peaks gracefully.

So an event the SDK sends does not land in ClickHouse on arrival. Web receives it and enqueues it; the worker dequeues, processes, and inserts. Two consequences follow.

  • A small delay before a trace shows up in the UI is normal. The problem is a delay that keeps growing, and what you look at then is worker throughput.
  • If the worker is down, the API still returns 200. Data piles up in the queue. Without monitoring on the worker container, it backs up silently.

The object storage documentation gives the reasons for persisting raw events as retry jobs, replay and disaster recovery, and optional record assembly. Processing can fail and the original is still there to retry — at the cost of steadily consuming bucket capacity. The fifth post returns to that.

On the Redis side there is one setting you must get right. The documentation states that maxmemory-policy must be set to noeviction on all Redis or Valkey instances so that queue jobs are not evicted. Leave it at the default and queued events quietly vanish under memory pressure.

Why ClickHouse — Columnar Storage and Trace Queries

The ClickHouse documentation describes ClickHouse as the main OLAP storage solution within Langfuse for Trace, Observation, and Score entities, optimized for high write throughput and fast analytical queries.

Why that pairing works is clearest in ClickHouse's own explanation. The performance documentation gives four reasons.

  • Column-oriented storage. Values of the same type and data distribution sit together, which suits compression particularly well.
  • Primary key indexes define the sort order of the table data. A well-chosen primary key lets filters be evaluated with fast binary searches instead of full-column scans.
  • Vectorized execution. Query plan operators pass intermediate rows in batches rather than one at a time, which improves CPU cache utilization and allows SIMD instructions.
  • MergeTree background merges. Additional data transformations happen during the background merge rather than at query time, making user queries significantly faster.

Overlay those four on the storage shape from the first post and the picture completes. Langfuse conceptually keeps one observations table, and each row holds the observation data plus a copy of the trace-level attributes.

To a relational instinct that is waste — user_id copied twenty times across the twenty observations of one trace. In a columnar store the story changes. A column where the same value repeats compresses extremely well, and a query filtering on user_id reads only that one column. No join means no shuffle. The cost of denormalization all but disappears on this engine, and what you get back is a simpler query path.

What ClickHouse Holds — traces, observations, scores

The main tables the documentation names are traces, observations, and scores. The object storage documentation adds a separate blob_storage_file_log table that tracks files uploaded to blob storage.

The documentation is explicit about access patterns too: tracing data is partitioned monthly, and access patterns center on project and time filters. So a query narrowed by project and period gets the full benefit of partition pruning, while a query that leaves the period open and scans everything touches every partition. Set a wide default range on a dashboard and you pay that cost on every load.

User permissions are spelled out precisely as well.

-- Grants Langfuse requires on its ClickHouse user
GRANT INSERT, SELECT, ALTER UPDATE, ALTER DELETE,
      ALTER DROP INDEX, CREATE, DROP TABLE
  ON langfuse.*
  TO langfuse;

Note that ALTER UPDATE and ALTER DELETE are in there. This is not a read-only analytics store; it is a store where retention-driven deletes and updates happen, which connects directly to the retention discussion in the fifth post.

Cluster topology carries a constraint. The documentation states that Langfuse does not currently support multi-shard clusters and that the shard count must be 1, while recommending a minimum of three replicas in production. In other words, you scale by replication and vertically, not by sharding. If you want to separate read load, CLICKHOUSE_READ_ONLY_URL exists; the documentation describes it as an optional read-only endpoint used for UI and public-API queries.

What Postgres and Redis Carry

The Postgres documentation names users, organizations, projects, datasets, encrypted API keys, and settings. Low volume, consistency-critical, transactional. Traces are not here.

Redis takes caching on top of the queue role above. The documentation names API keys and prompts as the cached objects, each with a default time to live of 300 seconds. API keys are never stored in plain text; only hashed or encrypted forms are cached.

Here is the condition that trips self-hosters most often. The self-hosting overview states that all infrastructure components must operate with the UTC timezone, and that non-UTC configurations will cause queries to return incorrect or empty results. The same warning appears in both the Postgres and ClickHouse pages.

Where Large Payloads Go

This is where LLM tracing diverges decisively from ordinary distributed tracing: the data hanging off one span is large. A prompt with a long context, the raw text of twenty retrieved documents, and image inputs all attach to a single observation.

The object storage documentation groups this store's purposes into three: raw events, multi-modal content, and batch exports. On the multi-modal side there is a configurable size ceiling. LANGFUSE_S3_MEDIA_MAX_CONTENT_LENGTH defaults to 1,000,000,000 bytes, that is 1 GB. Download URL validity is set by LANGFUSE_S3_MEDIA_DOWNLOAD_URL_EXPIRY_SECONDS, default 3600 seconds.

Officially supported providers are Amazon S3, Google Cloud Storage, Azure Blob Storage, MinIO/AIStor, and Cloudflare R2, with OCI Object Storage, Tigris, and other S3-compatible services under community support. The minimum permissions are s3:PutObject, s3:ListBucket, and s3:GetObject on both the bucket and its objects, plus s3:DeleteObject if you want data retention.

The Schema Is Not a Stable Contract

Once the data is in ClickHouse a natural thought follows: why not just query it directly? The documentation attaches a clear warning. The ClickHouse schema is not a stable API contract. Major upgrades, background migrations, and performance work can alter tables, columns, deduplication behavior, and join patterns, so custom direct queries have to be re-validated with every Langfuse upgrade.

The judgment splits accordingly. Metrics your application depends on come from the public API; direct ClickHouse queries belong to one-off investigations and internal analysis where a human can fix breakage. The metrics API covered in the sixth post is the front half of that.

If you need the actual column definitions, read the migration files rather than guessing. The ClickHouse documentation points at the repository path ./packages/shared/clickhouse/migrations/ while describing the manual migration procedure. The following shows how to check the schema and is illustrative.

-- Illustrative: confirm the real column definitions with your own eyes
SHOW TABLES FROM langfuse;
SHOW CREATE TABLE langfuse.observations;

-- See how the partitions are actually cut
SELECT partition, sum(rows) AS rows, formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE database = 'langfuse' AND table = 'observations' AND active
GROUP BY partition
ORDER BY partition DESC;

The second query is safe without guessing at the schema, because system.parts is a system table ClickHouse provides rather than something Langfuse defines. It returns rows and disk usage per partition, so you can see immediately which month is eating the cost.

Where the Versions Diverge

This is the section where version labelling matters most in this series, because the storage-layer requirements differ by major version.

ComponentLangfuse v3Langfuse v4
ClickHouse24.3 or newer25.12 or newer, 26.4 recommended
PostgreSQL12 or newerMinimum 15, 16 recommended
Redis7 or newer7.2 recommended
Valkey—8 or newer officially supported

The documentation gives the reason the ClickHouse requirement rose: v4 relies on lightweight updates, the JSON type, and full-text search. These values can change, so re-check the docs for the version you are running before you deploy.

Per the versioning documentation, v2 is end of life, v3 is deprecated, and v4 is generally available. Moving from v3 to v4 replaced legacy batch ingestion with OpenTelemetry, removed the legacy read APIs, and introduced the Observations API v2 and the Metrics API v2. Whenever you find a Langfuse architecture article online, check which version it describes first.

Closing — Knowing the Storage Layer Speeds Up Diagnosis

To summarize the split: trace bodies in ClickHouse, organizations and settings in Postgres, queue and cache in Redis, raw events and large payloads in object storage. With that map, where you look during an incident changes.

  • Traces show up late in the UI → the worker and the Redis queue
  • Login works but traces are empty → the ClickHouse connection or migrations
  • Traces render but attachments do not open → object storage permissions or presigned URL expiry
  • Queries return empty results → the timezone setting

The check you can run today is pulling per-partition sizes. Without knowing how much each month occupies, there is no basis for setting a retention policy.

Try It Yourself

  • DuckDB Data Playground — handling the same data row-wise and column-wise makes the difference in compression and scan cost tangible.
  • PostgreSQL Playground — experiment with putting transactional and analytical data in one engine and see what buckles first.

If the design of observability data in ClickHouse itself interests you, Putting Observability Data Into ClickHouse goes deeper on sort keys and TTL design.

Series

References

Discuss on Twitter • View on GitHub