The Questions Distributed Tracing Actually Answers — Spans, Sampling, and Where the Time Went

• Introduction — When the Reports Say Slow and You Do Not Know Which Service
• The Structure of Traces, Spans, and Context Propagation
• The Questions Logs and Metrics Cannot Answer
• Instrumentation — The Boundary of Auto-Instrumentation and Where Manual Spans Belong
• Sampling — The Trap of Head Sampling and the Value of Tail Sampling
• Span Attributes and Async Boundaries — Cardinality, Queues, Links
• In Practice — What You Are Actually Looking For on a Trace Screen
  • What OpenTelemetry Has Standardized and What It Has Not
• Closing — Tracing Is the Tool That Answers "Where"

Introduction — When the Reports Say Slow and You Do Not Know Which Service

A report comes in that the checkout screen takes 2 seconds. You open the dashboard and the gateway p99 has risen to 1.8 seconds. There are ten services behind it, so you open each of their p99 graphs one by one. All normal.

This situation is common. And in this situation, metrics cannot give you an answer in principle. The fact that each service is individually fast and the fact that one request passing through them in sequence was slow are not contradictory. What you need is not per-service statistics but the full path of a single request.

This post is about building and reading that path.

The Structure of Traces, Spans, and Context Propagation

A trace is a tree of spans sharing one trace ID. Each span has a name, a start time, an end time, a parent span ID, attributes, events, and a status. It also has a kind, and that matters more in practice than people expect.

• SERVER: the side receiving and handling a request
• CLIENT: the side calling another service
• PRODUCER / CONSUMER: the side sending a message and the side receiving it
• INTERNAL: a logical segment inside a process

The time difference between a CLIENT span and a SERVER span is the time consumed in the network and in queueing. You need both spans to know that value. If only one side is instrumented, you know only that "the call took a long time" and cannot tell whether the reason was the network or the other service.

A completed trace looks like this.

Trace 4bf92f3577b34da6a3ce929d0e0e4736                 total 1,847ms
├─ SERVER    api-gateway   POST /v1/orders                    1847ms
│  ├─ CLIENT    auth-svc     GET /verify                         38ms
│  ├─ CLIENT    checkout     POST /orders                      1782ms
│  │  ├─ SERVER   checkout    POST /orders                     1776ms
│  │  │  ├─ INTERNAL  cart.validate                               6ms
│  │  │  ├─ CLIENT    inventory  GET /stock                      41ms
│  │  │  ├─ CLIENT    pricing    POST /quote                     52ms
│  │  │  ├─ CLIENT    db.query   SELECT coupons WHERE ...      1612ms   <-- here
│  │  │  └─ CLIENT    payment    POST /charge                    64ms
│  │  └─ (network + queueing 6ms)
│  └─ INTERNAL  response.serialize                                9ms

Context propagation is the only mechanism that builds this tree. The caller puts the current trace ID and its own span ID into a header, and the receiver reads them and adopts them as the parent of its own span. The standard header is the W3C traceparent.

# The header that actually goes out when the gateway calls checkout
curl -sD - -o /dev/null http://checkout.internal/v1/orders \
  -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01'

# 00                                 version
# 4bf92f3577b34da6a3ce929d0e0e4736   trace ID — identical across the whole request
# 00f067aa0ba902b7                   parent span ID — changes at each call hop
# 01                                 sampling flag — if 00, this trace will not be stored

The places where propagation breaks are almost always the same. A hand-rolled HTTP client that auto-instrumentation cannot wrap, code that hands work off to a thread pool or a worker, message queues, and third-party proxies that strip headers. If a trace looks suspiciously short, start with those four.

The Questions Logs and Metrics Cannot Answer

Explaining why you need tracing as "because it is the third signal" is how adoption fails. You have to know exactly which question it answers.

First, where did one request spend its time. Metrics are aggregates, so they cannot reconstruct the path of an individual request. There is no guarantee that the p99 of service A and the p99 of service B belong to the same request. The coupon lookup query that burned 1,612ms in the trace above will never show up on that database's average query time dashboard. If requests like that are 1 percent of several million a day, the average does not budge.

Second, the repeated call problem. N+1 queries are invisible in metrics, because each individual query is a fast 4ms. The problem is that it runs 340 times within one request, and that only becomes visible when you count spans.

├─ SERVER  order-api  GET /v1/orders/A-99183                     1421ms
│  ├─ CLIENT  db.query  SELECT * FROM orders WHERE id = ?           5ms
│  ├─ CLIENT  db.query  SELECT * FROM order_items WHERE oid = ?     4ms
│  ├─ CLIENT  db.query  SELECT * FROM products WHERE id = ?         4ms
│  ├─ CLIENT  db.query  SELECT * FROM products WHERE id = ?         4ms
│  ├─ ... (the same query 340 times) ...

Third, conditional paths. Cases that only get slow for a particular tenant, a particular feature flag, or a particular combination of cache misses. Logs show you only a cross-section of each service, and stitching those cross-sections together by time is guesswork. A trace shows you that combination as a single object and lets you filter on attributes.

Conversely, what tracing cannot answer is equally clear. When it started getting worse and how many users are affected are questions for metrics. Which values arrived inside that span and which branch they took is a question for logs. Traces answer "where", metrics answer "when and how much", and logs answer "why".

Instrumentation — The Boundary of Auto-Instrumentation and Where Manual Spans Belong

Start with auto-instrumentation. Spans appear at I/O boundaries without touching the code.

npm i @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node

OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES=service.version=1.42.3,deployment.environment=prod \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4318 \
OTEL_TRACES_SAMPLER=parentbased_traceidratio \
OTEL_TRACES_SAMPLER_ARG=1.0 \
node --require @opentelemetry/auto-instrumentations-node/register server.js

parentbased_traceidratio follows the parent span's decision as-is and only makes a probabilistic judgement when there is no parent. This should be your default. If each service makes an independent probabilistic decision, traces get chopped off in the middle.

What auto-instrumentation gives you is exactly one thing. Network boundaries. HTTP servers and clients, gRPC, database drivers, Redis, and message queue clients are the targets.

What auto-instrumentation does not see is also one thing. Everything that happens inside the process. In-memory sorting, template rendering, encryption, serialization, lock waits, and event loop delay never appear as any span. They surface only as the self time of the parent span, that is, the time not accounted for by child spans.

├─ SERVER  report-api  GET /v1/reports/monthly              3204ms
│  ├─ CLIENT  db.query   SELECT ... FROM ledger              412ms
│  └─ CLIENT  s3.upload  PUT report-2026-07.pdf              260ms
│     self time = 3204 - 412 - 260 = 2532ms  <-- uninstrumented segment

When you find a span with large self time, put manual spans inside it. There are five places where you should.

1. Loop and batch boundaries — record the iteration count as an attribute.
2. Cache lookups and miss paths — record hit or miss as an attribute and cache performance becomes visible directly in the trace.
3. Third-party SDK calls with no auto-instrumentation
4. Lock waits, queue waits, connection pool waits
5. CPU segments such as serialization, compression, and image processing

import { trace, SpanStatusCode } from '@opentelemetry/api'

const tracer = trace.getTracer('checkout', '1.42.3')

export async function applyPromotions(cart, tenantId) {
  return tracer.startActiveSpan('checkout.applyPromotions', async (span) => {
    span.setAttribute('cart.item_count', cart.items.length)
    span.setAttribute('promotion.engine', 'rules-v3')
    span.setAttribute('tenant.id', tenantId)

    try {
      const cached = await rules.fromCache(tenantId)
      span.setAttribute('promotion.cache_hit', Boolean(cached))

      const result = await (cached ?? rules.compile(tenantId)).evaluate(cart)
      span.setAttribute('promotion.rules_evaluated', result.evaluated)
      span.setAttribute('promotion.matched_count', result.matched.length)
      return result
    } catch (err) {
      span.recordException(err)
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message })
      throw err
    } finally {
      span.end()
    }
  })
}

Span names must be low cardinality. Not GET /v1/orders/A-99183 but GET /v1/orders/:id. Backends group by span name, so putting an ID in the name collapses every aggregate view. Send the specific values as attributes.

Sampling — The Trap of Head Sampling and the Value of Tail Sampling

Storing everything does not add up financially for most organizations. The question is what you throw away.

Head sampling decides at the moment the root span is created, that is, when nothing has happened yet. Since it does not know the outcome, it has no choice but to discard at random. And discarding at random means discarding rare events exactly in proportion to their rarity.

python3 - <<'PY'
rate = 0.01              # head sampling at 1%
errors_per_hour = 5      # the real frequency of the error you want to investigate
kept = errors_per_hour * rate
print(f"error traces retained: {kept:.2f} per hour")
print(f"average wait to see one: {1/kept:.0f} hours")
PY
# error traces retained: 0.05 per hour
# average wait to see one: 20 hours

That is the practical ending of head sampling. When you need to investigate, the trace is not there. And the traces that are there are all successful requests, so there is no reason to look at them.

Tail sampling buffers spans until the trace completes and then decides based on the outcome. Keep it if there was an error, keep it if it was slow, keep only a small share of the rest.

# otel-collector-tailsampler.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 2000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 800

      - name: keep-vip-tenants
        type: string_attribute
        string_attribute:
          key: tenant.tier
          values: [enterprise]

      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 2

That tail sampling is not free is something that has to be said out loud. There are three costs.

First, the volume from agent to collector does not go down. Every span has to reach the collector before a decision can be made. What you save is only backend storage and indexing cost. The collector's own CPU and network actually increase.

Second, memory. You have to hold every span for the duration of decision_wait.

python3 - <<'PY'
traces_per_sec, wait_sec, spans, span_kb = 2000, 10, 12, 1.2
mb = traces_per_sec * wait_sec * spans * span_kb / 1024
print(f"buffer memory roughly {mb:.0f} MB (allow 2x headroom: {mb*2:.0f} MB)")
PY
# buffer memory roughly 281 MB (allow 2x headroom: 563 MB)

Third, and this is where people make the most mistakes. Every span of the same trace must arrive at the same collector instance. Scale the collector out across several nodes with an ordinary load balancer in front and one trace's spans scatter across instances, each judging a fragmented trace on its own. The result is randomly truncated traces. The solution is a gateway layer that routes by trace ID.

# otel-collector-gateway.yaml — route by trace ID at the front
exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-tailsampler.observability.svc.cluster.local
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [loadbalancing]

In practice you combine the two. Services with very high traffic get cut down to somewhere between 10 and 50 percent with head sampling first, and tail sampling is layered on top to fish out errors and slow requests. Anything already discarded at the head cannot be revived at the tail, so as a principle you set the head ratio as high as you can afford.

Span Attributes and Async Boundaries — Cardinality, Queues, Links

The advice "watch out for cardinality in traces too" is only half right. High cardinality in span attributes is not an explosion problem the way it is in metrics. Order IDs, user IDs, and query parameters are exactly the values traces exist to hold. Without them a trace is a picture you cannot filter.

Problems arise in two places.

First, when you convert spans into metrics. Build RED metrics from spans with the spanmetrics connector and the attributes you nominate as dimensions become metric labels directly. Put a user ID in there and the time series explode.

connectors:
  spanmetrics:
    # only what is listed here becomes a metric label — put low cardinality only
    dimensions:
      - name: http.route
      - name: http.request.method
      - name: deployment.environment
    histogram:
      explicit:
        buckets: [10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s]

Second, the size of the span itself. The OpenTelemetry SDK caps the attribute count per span by default (128), but there is no default cap on attribute value length. Put a whole request body in an attribute and one span becomes hundreds of KB, with transport and storage cost following right along. Set explicit limits if you need to.

OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT=64 \
OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT=2048 \
node --require @opentelemetry/auto-instrumentations-node/register server.js

Async boundaries are trickier. In a message queue, the consumer runs after the producer span has already ended. Use a parent-child relationship as-is and the parent is already closed, which makes the trace's time axis strange, and a batch consumer becomes impossible to represent at all. Process 100 messages at once and there are 100 parents, but a span can only have one parent.

The solution is links.

import { propagation, context, trace, SpanKind } from '@opentelemetry/api'

// Producer: inject the context into the message headers
export async function publish(order) {
  return tracer.startActiveSpan('orders.publish', { kind: SpanKind.PRODUCER }, async (span) => {
    const headers = {}
    propagation.inject(context.active(), headers)
    await producer.send({ topic: 'orders', messages: [{ value: JSON.stringify(order), headers }] })
    span.end()
  })
}

// Batch consumer: attach each message context as a link rather than a parent
export async function consumeBatch(messages) {
  const links = messages
    .map((m) => trace.getSpan(propagation.extract(context.active(), m.headers))?.spanContext())
    .filter(Boolean)
    .map((spanContext) => ({ context: spanContext }))

  return tracer.startActiveSpan(
    'orders.processBatch',
    { kind: SpanKind.CONSUMER, links },
    async (span) => {
      span.setAttribute('messaging.batch.message_count', messages.length)
      await Promise.all(messages.map(handle))
      span.end()
    }
  )
}

For a consumer handling only one message, parent-child is acceptable. Note, though, that if queue wait times are long, one trace ends up lasting hours, which backends handle badly. For pipelines with long waits it is more practical to cut with links and record the queue wait time as an attribute instead.

In Practice — What You Are Actually Looking For on a Trace Screen

The biggest reason people leave the tool switched on and never use it is that there is no investigation order. The following order works in most cases.

1. Start from metrics. Pin down which route and which time range got worse. Starting by rummaging through traces at random only burns time.
2. Find the edge in the service graph where latency or error rate rose. Whether the call relationships themselves changed is also visible here.
3. Look at the list of traces for that route and time range with the highest duration. Most backends provide a query that filters by duration and attributes.
4. Open one trace and find the span with the largest self time. Not the longest span, the largest self time. The longest span is usually the root and carries no information.
5. Look at that span's attributes. Cache miss, particular tenant, how many retries.
6. Query the logs by the same trace ID. This is where "why" comes from.

One discipline is required here. Do not draw conclusions from one slow trace. A single case may be coincidence. Open several traces under the same conditions to confirm a common pattern, and compare them side by side with a normal trace. Good backends provide per-span duration distributions or trace comparison views for exactly this.

What OpenTelemetry Has Standardized and What It Has Not

This affects adoption decisions, so you should know where the boundary is.

The parts that are standardized and safe to lean on are the structure of the API and SDK, the OTLP transport protocol, W3C-based context propagation, and the collector. Write your instrumentation code against the OpenTelemetry API and switching backends will rarely require touching it. Semantic conventions for core areas such as HTTP have also stabilized.

Parts that are still moving remain, however. Semantic conventions in several domains are still experimental or in the middle of being renamed, and maturity varies widely across language SDKs. How metrics are derived from spans and how service graphs are generated differ per backend, and there is no standard language for querying traces. Design on the assumption that dashboards and alerts depending on that area are tied to the backend.

Closing — Tracing Is the Tool That Answers "Where"

There is one sentence to remember. Tracing is the tool that answers where a single request spent its time, and that answer exists only when the context was not broken and the trace was not thrown away.

That fixes the adoption order too. First check propagation. If traces are breaking at service boundaries, every other investment is meaningless. Then look at sampling. If you are running head sampling only, the trace will not be there at the moment you need to investigate, so introduce tail sampling together with trace ID routing. Finally, put manual spans into the segments with large self time to fill in the blanks auto-instrumentation leaves.

The very first thing to do is open one trace in production right now. If you have ten services and see only three spans, that is your entire agenda for next week.

Further reading.

• OpenTelemetry: Traces concepts
• OpenTelemetry: Sampling
• OpenTelemetry Collector: Tail Sampling Processor
• W3C Trace Context specification