필사 모드: Instrumenting Your App With OpenTelemetry — From Auto-Instrumentation to Manual Spans, and Why You Put a Collector in Front

• Introduction — When You Have Dashboards but Don't Know Why It's Slow
• Decide the Order First
• Stage 1 — How Far Auto-Instrumentation Alone Gets You
  • What Auto-Instrumentation Never Sees
• Stage 2 — You Can't Fix Resource Attributes Later
• Stage 3 — The Four Places Context Propagation Breaks
  • Break 1 — Thread Pools and Executors
  • Break 2 — Message Queues
  • Break 3 — Background Jobs and Schedulers
  • Break 4 — Intermediate Layers That Strip Headers
• Stage 4 — Manual Spans Only Where Self Time Is Large
• Stage 5 — Why You Put a Collector Between the App and the Backend
• Ways Instrumentation Breaks the App
  • The Bar for Saying Instrumentation Is Done
• Closing — Instrumentation's Value Comes From a Trace's Completeness, Not Its Span Count

Introduction — When You Have Dashboards but Don't Know Why It's Slow

The order API's p99 is 1.4 seconds. Grafana already has panels for CPU, memory, request count, and error rate, and they're all green. The database dashboard is normal too. But users say it's slow, and we don't know which code spent that 1.4 seconds.

What's needed in this state isn't one more panel — it's instrumentation. Instrumentation is the work of making the code itself say "what happened, when, and how long it took," and OpenTelemetry is the project that standardizes the format and transport protocol for that telling.

This post instruments one service from start to finish. It uses a Python FastAPI service as the example, but the order is language-agnostic. Verified as of July 2026, against OpenTelemetry Collector v0.157.0, Semantic Conventions v1.43.0, and the Python SDK 1.3x line. Since there are still areas of the semantic conventions where names keep changing, it's safer to always check attribute names in the Semantic Conventions registry for the version you're on.

Decide the Order First

Teams that fail at instrumentation fail almost the same way every time. They start by planting manual spans in the code, and two weeks later they have 300 spans, while traces are still breaking at service boundaries.

Here's an order that actually works.

The key is that step 3 comes before step 4. Adding manual spans while propagation is broken just breaks an already-fragmented trace into finer fragments.

Stage 1 — How Far Auto-Instrumentation Alone Gets You

In Python, the opentelemetry-instrument launcher hooks in installed instrumentation packages at process start. There's no code change.

pip install \
  'opentelemetry-distro[otlp]' \
  opentelemetry-instrumentation-fastapi \
  opentelemetry-instrumentation-sqlalchemy \
  opentelemetry-instrumentation-requests \
  opentelemetry-instrumentation-redis \
  opentelemetry-instrumentation-logging

# Auto-detects and attaches whichever instrumentation packages are installed
opentelemetry-bootstrap --action=install

Execution is controlled purely through environment variables. This is auto-instrumentation's core advantage. Since the instrumentation config lives in the deployment manifest rather than in code, changing an endpoint or a sampling rate doesn't need a code review.

export OTEL_SERVICE_NAME=checkout-api
export OTEL_RESOURCE_ATTRIBUTES=service.version=2.7.1,deployment.environment.name=prod,service.namespace=commerce
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_TRACES_SAMPLER=parentbased_always_on
export OTEL_PYTHON_LOG_CORRELATION=true

opentelemetry-instrument uvicorn app.main:app --host 0.0.0.0 --port 8000

Starting with parentbased_always_on is recommended. Turn on rate-based sampling from the start, and when a trace doesn't show up, you can't tell whether it's an instrumentation problem or the sampling. Attach sampling in the collector after you've confirmed data is flowing.

What you get from this state is exactly one thing: network boundaries.

SERVER    checkout-api   POST /v1/orders                       1421ms
├─ CLIENT    GET http://auth.internal/verify                      31ms
├─ CLIENT    SELECT carts WHERE id = ?                             6ms
├─ CLIENT    redis GET promo:rules:t-8871                          2ms
├─ CLIENT    POST http://payment.internal/charge                  74ms
└─ (the remaining 1308ms belongs to no span)

The last line is everything. Auto-instrumentation tells you where the problem isn't, through a 1308ms gap. That gap is called self time, and in stage 4, this is the only place you add manual spans.

What Auto-Instrumentation Never Sees

• CPU work inside the process — serialization, compression, template rendering, encryption, image processing
• Waiting on a lock and waiting on a connection pool — the wait time isn't included in the query span that starts after the connection is acquired
• GIL contention and event-loop delay
• Third-party SDK calls with no instrumentation package
• Branches in business logic — which rules got evaluated, and how many

Stage 2 — You Can't Fix Resource Attributes Later

A resource is a set of attributes describing "what produced this telemetry." Unlike span attributes, resource attributes attach to every signal that process emits. And once set, they're hard to change. The moment you change service.name, the link to dashboards, alerts, the service graph, and past data all breaks at once.

# Minimal set — without these three, you can't tell where the data came from
OTEL_SERVICE_NAME=checkout-api
OTEL_RESOURCE_ATTRIBUTES=service.version=2.7.1,deployment.environment.name=prod

# Additionally useful in practice
OTEL_RESOURCE_ATTRIBUTES=service.version=2.7.1,\
deployment.environment.name=prod,\
service.namespace=commerce,\
service.instance.id=checkout-api-7d9f4b-x2k9m

In Kubernetes, don't hardcode the instance identifier — inject it via the Downward API.

# deployment.yaml
env:
  - name: OTEL_SERVICE_NAME
    value: checkout-api
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: >-
      service.version=2.7.1,
      deployment.environment.name=prod,
      service.namespace=commerce,
      service.instance.id=$(POD_NAME),
      k8s.namespace.name=$(POD_NAMESPACE)

Here are two things people frequently get wrong about naming conventions.

First, the name of the environment attribute is deployment.environment.name. The old name, deployment.environment, is no longer used. Different names become two separate attributes, and a dashboard variable only reads one of them.

Second, service.name should be scoped to the service, not the deployment unit. If you've stood up the same codebase twice for a canary, both are checkout-api, and you distinguish them with service.version or a separate attribute. The moment you name the canary checkout-api-canary, a phantom node appears in the service graph.

service.instance.id is high-cardinality, but since it's a resource attribute, it's not a problem for traces and logs. However, if you promote this attribute directly into a metric label, your time series get multiplied by the pod count. It's common to strip it in the collector, but only for the metrics pipeline.

Stage 3 — The Four Places Context Propagation Breaks

Propagation is the sole mechanism that creates a trace. The caller puts the current trace ID and span ID into the W3C traceparent header, and the callee reads it and uses it as the parent. Verifying this takes just one command.

# Impersonate the gateway by injecting the header directly, then look this trace ID up in the backend
curl -sS -o /dev/null -w '%{http_code}\n' \
  http://checkout-api.internal/v1/orders \
  -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' \
  -H 'content-type: application/json' \
  -d '{"cart_id":"c-1"}'

# If a span doesn't end up under this trace ID, it's one of the four cases below

Break 1 — Thread Pools and Executors

This is the type you run into most. Since the context lives in thread-local storage (or an asyncio contextvar), handing work off to a different thread doesn't carry that context along.

# Breaks — the worker thread has no context, so a new trace starts
from concurrent.futures import ThreadPoolExecutor

pool = ThreadPoolExecutor(max_workers=8)

def enrich_all(items):
    return list(pool.map(fetch_details, items))

# Survives — capture the current context and reactivate it inside the worker
from concurrent.futures import ThreadPoolExecutor
from opentelemetry import context as otel_context

pool = ThreadPoolExecutor(max_workers=8)

def _with_context(ctx, fn, *args):
    token = otel_context.attach(ctx)
    try:
        return fn(*args)
    finally:
        otel_context.detach(token)

def enrich_all(items):
    ctx = otel_context.get_current()
    futures = [pool.submit(_with_context, ctx, fetch_details, it) for it in items]
    return [f.result() for f in futures]

In Java, Context.current().wrap(runnable); in Go, passing a context.Context as a goroutine argument; in Node.js, AsyncLocalStorage — these play the same role. The name differs by language, but the principle is identical. Context doesn't follow the unit of execution, so you have to move it explicitly.

Break 2 — Message Queues

A queue is both a process boundary and a time boundary. Headers don't flow automatically the way they do with HTTP, so you have to inject them into the message directly.

from opentelemetry import propagate, trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("checkout", "2.7.1")

def publish_order(producer, order):
    with tracer.start_as_current_span(
        "orders publish", kind=SpanKind.PRODUCER
    ) as span:
        span.set_attribute("messaging.system", "kafka")
        span.set_attribute("messaging.destination.name", "orders")
        headers = {}
        propagate.inject(headers)          # put traceparent into the dict
        producer.send(
            "orders",
            value=order.to_bytes(),
            headers=[(k, v.encode()) for k, v in headers.items()],
        )

On the consumer side, use the extracted context as the parent. But if you're processing several messages at once as a batch, there's only one parent, so use links instead.

from opentelemetry import propagate, trace
from opentelemetry.trace import SpanKind, Link

def consume_batch(messages):
    links = []
    for m in messages:
        headers = {k: v.decode() for k, v in (m.headers or [])}
        ctx = propagate.extract(headers)
        sc = trace.get_current_span(ctx).get_span_context()
        if sc.is_valid:
            links.append(Link(sc))

    with tracer.start_as_current_span(
        "orders process", kind=SpanKind.CONSUMER, links=links
    ) as span:
        span.set_attribute("messaging.batch.message_count", len(messages))
        for m in messages:
            handle(m)

If queue wait time runs several minutes or more, consider using a link even for a single message. Chain it as parent-child, and one trace's duration stretches out by the wait time, which makes it unwieldy for the backend to handle.

Break 3 — Background Jobs and Schedulers

Work that runs independently of any request — cron, Celery beat, FastAPI's BackgroundTasks — has no parent. A common mistake here is forcibly stitching in the request context. The request already sent its response and ended, so if a 30-second child gets attached to that trace anyway, it contaminates the request latency statistics.

# Background work starts as a new root trace; the causing request is left as a link
def schedule_reindex(cart_id):
    origin = trace.get_current_span().get_span_context()

    def run():
        links = [Link(origin)] if origin.is_valid else []
        with tracer.start_as_current_span(
            "cart.reindex", kind=SpanKind.INTERNAL, links=links
        ) as span:
            span.set_attribute("cart.id", cart_id)
            reindex(cart_id)

    background.add_task(run)

Break 4 — Intermediate Layers That Strip Headers

When a proxy, WAF, API gateway, or CDN filters headers by allowlist, traceparent silently disappears. Nothing gets logged, and the symptom is "the trace starts fresh right after the gateway."

# The fastest way to see which headers actually arrive
kubectl -n commerce exec deploy/checkout-api -- \
  sh -c 'timeout 20 tcpdump -A -s0 -i any "tcp port 8000" 2>/dev/null | grep -i traceparent'

# Or add a temporary endpoint to the app that echoes back whatever headers it received
curl -s http://checkout-api.internal/__debug/headers \
  -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01' | jq .

If you're using Envoy or Istio, check that traceparent, tracestate, and baggage are on the allowlist. If legacy services using B3 headers are mixed in, configure multiple propagators.

OTEL_PROPAGATORS=tracecontext,baggage,b3multi

Stage 4 — Manual Spans Only Where Self Time Is Large

Back to the 1308ms gap. There are five candidate types of places to add manual spans.

1. Loop and batch boundaries — record the iteration count as an attribute
2. Cache lookups — record hit/miss as an attribute, and cache efficiency becomes visible directly in the trace
3. Third-party SDK calls with no instrumentation package
4. Waiting on a lock, a queue, or a connection pool
5. Stretches that spend a long time on CPU — serialization, compression, report generation

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("checkout", "2.7.1")

async def apply_promotions(cart, tenant_id):
    with tracer.start_as_current_span("checkout.apply_promotions") as span:
        span.set_attribute("cart.item_count", len(cart.items))
        span.set_attribute("tenant.id", tenant_id)
        span.set_attribute("promotion.engine", "rules-v3")
        try:
            with tracer.start_as_current_span("promotion.load_rules") as load:
                cached = await rules.from_cache(tenant_id)
                load.set_attribute("cache.hit", cached is not None)
                ruleset = cached or await rules.compile(tenant_id)
                load.set_attribute("promotion.rule_count", len(ruleset))

            with tracer.start_as_current_span("promotion.evaluate") as ev:
                result = ruleset.evaluate(cart)
                ev.set_attribute("promotion.evaluated", result.evaluated)
                ev.set_attribute("promotion.matched", len(result.matched))
            return result
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            raise

After adding this instrumentation, the trace for the same request changes to this.

SERVER    checkout-api   POST /v1/orders                       1421ms
├─ CLIENT    GET http://auth.internal/verify                      31ms
├─ CLIENT    SELECT carts WHERE id = ?                             6ms
├─ INTERNAL  checkout.apply_promotions                          1298ms
│  ├─ INTERNAL  promotion.load_rules   cache.hit=false           1241ms  <-- here
│  └─ INTERNAL  promotion.evaluate     evaluated=812               54ms
├─ CLIENT    POST http://payment.internal/charge                  74ms
└─ (self time 12ms)

There's just one span-naming rule to keep: names must be low-cardinality. It's GET /v1/orders/:id, not GET /v1/orders/A-99183 — every concrete value goes into an attribute instead. Since the backend groups by span name to build latency statistics and the service graph, an ID in the name collapses that entire aggregate view.

Stage 5 — Why You Put a Collector Between the App and the Backend

Having the SDK send directly to the backend does work. Even so, there are five reasons to put a collector in front.

1. Change policy without redeploying. Sampling rate, attribute filters, and what gets retained are values you end up tuning while running. If they live in app environment variables, you have to roll out 20 services.
2. Isolate the app from backend outages. When the backend slows down, if the SDK's export queue fills up, app memory climbs, and in bad cases it affects request handling. With a collector in front, it absorbs that pressure instead.
3. You can swap backends. Sending metrics to Prometheus, traces to ClickHouse, logs to OpenSearch — using a different destination per signal, or running two backends in parallel while migrating — all of it ends in one place: the collector's config file.
4. Strip sensitive data outside the app. An incident where a token or an email gets mixed into an attribute is bound to happen. With a defensive layer in the collector, incident response becomes a config change instead of a redeploy.
5. You can do tail sampling. To judge based on a trace's outcome, its spans have to converge in one place, and that place can't be the app.

Here's a minimal configuration that sets batch size, retries, and a memory ceiling.

# otel-collector.yaml — the agent layer, placed on the same node as the app or as a sidecar
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Must be first. Once it hits the memory limit, it refuses incoming data to keep the collector itself from dying
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

  # Attach Kubernetes metadata as resource attributes
  k8sattributes:
    auth_type: serviceAccount
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.pod.name
        - k8s.node.name

  # Strip sensitive attributes — block it here without touching the app
  attributes/redact:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: delete
      - key: db.query.text
        action: hash

  # Always last. Cuts down network round trips
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 16384

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s

service:
  telemetry:
    metrics:
      level: detailed
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, attributes/redact, batch]
      exporters: [otlp/gateway]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp/gateway]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, attributes/redact, batch]
      exporters: [otlp/gateway]

Processor order carries meaning. If memory_limiter isn't first, the collector dies to OOM under overload; if batch isn't last, a processor downstream of it re-splits the batch and the benefit disappears. The Collector documentation recommends the same order.

It's common to split the collector into two layers. The agent next to the app only collects and attaches metadata; the gateway layer does tail sampling and backend routing. If you use tail sampling, you need trace-ID-based routing in front of the gateway. If spans from the same trace scatter across different instances, each one judges based on its own fragment, and traces get cut at random.

Ways Instrumentation Breaks the App

A guide that shows only the happy path is useless, so here's a collection of failures you'll actually run into.

You can cap attribute size at the SDK level. The default is 128 attributes with no limit on value length, so specifying it explicitly is safer.

OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT=64
OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT=2048
OTEL_BSP_MAX_QUEUE_SIZE=4096
OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
OTEL_BSP_SCHEDULE_DELAY=2000

Flushing on shutdown differs by language. Python auto-instrumentation shuts down the provider on a graceful exit, but if it's killed with SIGKILL, any spans left in the queue vanish. For short batch jobs, flushing explicitly is the surer approach.

from opentelemetry import trace

def main():
    run_job()
    # A batch job must always flush explicitly
    trace.get_tracer_provider().force_flush(timeout_millis=10_000)
    trace.get_tracer_provider().shutdown()

The Bar for Saying Instrumentation Is Done

You have to clear this checklist before moving to the next stage.

• Pick any production request at random, open its trace, and the number of services involved matches the number of SERVER spans
• The root span's duration matches the response time in the gateway access log, within margin of error
• The largest self time is under 20% of the total
• Copy the trace ID from a trace, put it into log search, and that request's logs come back
• When the span-name list is sorted by cardinality, the top 20 are route templates, not IDs
• Work that crosses a message queue is linked together as one trace or via links
• Restarting the collector doesn't affect the app

The last item you only know by actually doing it. Kill the collector once and check whether the app's error rate and latency stay steady. If they wobble, the exporter is synchronous or the queue policy is wrong.

Closing — Instrumentation's Value Comes From a Trace's Completeness, Not Its Span Count

A complete trace with 12 spans is overwhelmingly more useful than a fragmented one with 500. So the order of investment follows from that too. Unbroken propagation comes first, consistent resource attributes come second, and manual spans come last.

The cheapest verification you can do right now is to open one production trace and count the SERVER spans. If a request passed through six services but there are only two SERVER spans, this week's job isn't adding manual spans — it's reviving propagation at the other four.

Further reading.

• OpenTelemetry: Instrumentation concepts documentation
• OpenTelemetry: Resource Semantic Conventions
• OpenTelemetry Collector: Configuration
• W3C Trace Context specification
• OpenTelemetry Python: Manual Instrumentation