Skip to content
Published on

After You Have Collected It — Langfuse Dashboards, the Metrics API, and Scores on Traces

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Opening — Collection Is the Start, Questions Are the Point

By this stage the data is arriving. You know the data model, you have instrumented, you understand the storage layer, you have stood up the system, and you have tuned the volume. But the point of all of it is answering questions.

The problem is that opening a dashboard first does not make questions occur to you. When the chart is already drawn, you only ask the question the chart answers. Flip the order: write down the question you have to answer, then work out which measure it needs and which dimension it slices by.

Component and configuration names here were verified against the official documentation on 2026-08-15. Langfuse's architecture differs by version, so re-check the docs for the version you are running. The metrics API discussed here is the v2 form introduced in v4.

The Three Axes a Dashboard Shows

The metrics overview groups the metrics into three.

  • Quality, measured through user feedback, model-based scoring, human-in-the-loop scored samples, and custom scores sent through the SDKs and API. It can be assessed across time, prompt versions, LLMs, and users.
  • Cost and latency, which the documentation says can be broken down by user, session, geography, feature, model, and prompt version.
  • Volume, calculated based on the ingested traces and tokens used.

What matters is that the three explain each other. If quality dipped over some window, you have to look at how latency and cost moved over the same window to narrow the cause. Latency alone cannot tell you whether the model got slower, the prompt got longer, or retrieval added a step.

What You Slice By

The dimensions matter more than the measures. The documentation names trace name, user ID, tags, and release and version numbers as the analysis dimensions, with model and prompt version added on the cost and latency side.

Translating real questions into those dimensions looks like this.

QuestionMeasureDimension to slice by
Which feature costs the most?CostTrace name, tags
Did we get slower after the last deploy?LatencyRelease or version
Is one model failing more than the rest?Failure countModel
Do heavy users have a different cost profile?Cost, tokensUser ID
Did the new prompt raise quality?ScorePrompt version

The first post comes back here. These dimensions only exist if the values were present at instrumentation time. Without user_id, per-user cost queries are impossible from the outset, and without tags there is no per-feature breakdown. That is why most dashboard work stalls.

The Metrics API — Writing Queries as Code

Separately from dashboards you build by clicking, there is a path for pulling metrics programmatically. The metrics API documentation defines the endpoint as GET /api/public/v2/metrics.

The query is expressed as a single object. The fields the documentation lists are:

  • view — which data to look at
  • metrics — an array of measure and aggregation pairs
  • dimensions — an array of objects carrying field
  • filters — an array of filter conditions
  • fromTimestamp and toTimestamp — the window, in ISO 8601
  • orderBy — an array with field and direction
  • config — an object with row_limit, default 100 and max 1,000

The example request the documentation gives looks like this.

{
  "view": "observations",
  "metrics": [{ "measure": "totalCost", "aggregation": "sum" }],
  "dimensions": [{ "field": "providedModelName" }],
  "filters": [],
  "fromTimestamp": "2025-12-01T00:00:00Z",
  "toTimestamp": "2025-12-16T00:00:00Z",
  "orderBy": [{ "field": "sum_totalCost", "direction": "desc" }],
  "config": { "row_limit": 1000 }
}

Note the field name inside orderBy. It is the aggregation and the measure joined together, meaning the result column names are determined by the query structure.

Calls use the ordinary public API authentication.

# Illustrative: passing the query object as a query parameter
curl -sS -u "${LANGFUSE_PUBLIC_KEY}:${LANGFUSE_SECRET_KEY}" \
  --get "https://cloud.langfuse.com/api/public/v2/metrics" \
  --data-urlencode "query@./query.json"

Check the exact parameter encoding and the list of available measures, dimensions, and aggregations in the API documentation for the version you are running. That list grows and gets renamed across versions.

What Changed in v2 — There Is No traces View

This is where the version difference is most visible. The documentation names four views available in v2: observations, scores-numeric, scores-categorical, and scores-boolean. And it states that the traces view is no longer available in v2.

That change lines up exactly with the storage shape from the third post. Langfuse conceptually keeps one observations table where each row carries a copy of the trace-level attributes. Since trace-level attributes are already on the observation row, the observations view alone can answer trace-axis questions.

It also explains why an integration built from v3-era material breaks on v4. The versioning documentation states that moving from v3 to v4 removed the legacy read APIs and introduced the Observations API v2 and the Metrics API v2. If you built a dashboard or a reporting pipeline on top of the API, that item belongs in your upgrade plan.

And the principle emphasized in the third post takes effect here. Pull the metrics your application depends on through this API rather than poking ClickHouse directly. The schema is not a stable contract, but the public API is subject to the versioning policy.

How Scores Get Attached

The quality axis does not appear by itself; you have to create scores and attach them. The scores documentation lists five creation paths.

  1. LLM-as-a-Judge automated evaluators
  2. Code evaluators, in Python and TypeScript
  3. Manual scoring in the UI
  4. Annotation queues
  5. Programmatic addition through the API or SDK

The places they attach are the four from the first post: traces, observations, sessions, and dataset runs. The sessions documentation states that sessions can be annotated with human evaluations through scores and that session-level scores can be added programmatically through the SDK or API. The quality of a multi-turn conversation cannot be judged from one turn, which is why that axis exists.

The data type you choose decides what you can aggregate later. NUMERIC, CATEGORICAL, and BOOLEAN map straight onto the metrics API view names. TEXT scores, by contrast, cannot be used in experiments, LLM-as-a-judge evaluators, or analytics, as the documentation states. If you want to keep free-form prose, use the score's comment field and keep a separate CATEGORICAL score as the aggregation axis.

Check the exact method names and arguments for creating scores in the reference for the SDK version you are running. Python SDK v4 and JS/TS SDK v5 are the generally available SDKs on server v4.

Evaluators Moving Down to Observation Level

Of the v4 changes, this one has the largest practical impact. The versioning documentation states that the move from v3 to v4 shifted evaluators from trace level to observation level.

What that changes is clearest against the RAG tree from the first post. When only trace-level evaluation existed, you could ask "is this answer good?" and nothing else. With observation-level evaluation you can ask "did retrieval fetch the right documents?" of the retrieval step alone.

That makes a real difference in diagnosis. When final answer quality drops but the retrieval score holds steady, the problem is on the generation side. When the retrieval score dropped too, look at the index or the embeddings. A single number does not give you that distinction.

If you want to go deeper on evaluation design itself, LLM Evaluation and Observability covers eval harnesses, LLM-as-judge, and regression prevention.

Enabling Code Evaluators When Self-Hosting

Code evaluators run user-authored code on the server, so self-hosting means choosing the execution environment. The relevant variables from the configuration documentation are as follows.

# choose the execution dispatcher
LANGFUSE_CODE_EVAL_DISPATCHER=aws-lambda          # aws-lambda or insecure-local

# enable the worker consumer for the execution queue (default false)
QUEUE_CONSUMER_CODE_EVAL_EXECUTION_QUEUE_IS_ENABLED=true

# Lambda function names (documented defaults shown)
LANGFUSE_CODE_EVAL_AWS_LAMBDA_NODE_FUNCTION_NAME=code-based-eval-executor-node
LANGFUSE_CODE_EVAL_AWS_LAMBDA_PYTHON_FUNCTION_NAME=code-based-eval-executor-python

The name insecure-local is itself the warning. Of the two options the documentation offers, one is an isolated Lambda and the other is local execution; the name already tells you which belongs in production. Beyond these there are separate variables for queue shard count, worker concurrency, and the local execution timeout.

Designing Dashboards From Questions

Putting it together, the design order goes like this.

  1. Write down the questions you actually received last quarter. Received, not imagined.
  2. Translate each one into a measure and dimension combination. A question that will not translate means instrumentation is missing.
  3. Confirm the required dimension is present on the trace. If it is not, go back to the second post and fix instrumentation.
  4. Turn that combination into a dashboard or a metrics API query.
  5. For questions that need the quality axis, pick one path for attaching scores.

Follow that and the number of charts nobody looks at goes down. If dashboards keep multiplying while incident reviews still involve digging through logs, that is the signal the order is inverted.

Closing — One Sentence Across Six Posts

Reduced to one sentence, this series says: the questions you will ask later determine the shape of the data you record now.

The data model settled what could be recorded, instrumentation settled what actually is, the storage layer settled which queries can retrieve it, and retention and sampling settled how long it survives. Querying and evaluation in this final post are where you see the result of all those decisions.

The last check is the same as the first. Write down three questions you have to answer and see whether your current structure can answer them. If it cannot, the place to fix is not this post but the second one.

Try It Yourself

  • SLO & Error Budget Calculator — put a target on quality and latency and calculate how much monthly headroom that buys.
  • DuckDB Data Playground — writing measure and dimension combinations as queries by hand shows you what a dashboard is really doing.
  • Harness Engineering RPG — play through observability and release decisions to see which metrics actually change a decision.

Series

References

Discuss on Twitter • View on GitHub