Making Logs Searchable, and Not Going Broke Doing It — Structuring, Mapping Explosions, Retention, and Real Cost

• Introduction — The Day Log Costs Overtook Compute Costs
• Questions Logs Can Answer, and Questions They Can't
• Structured Logging — What Should Become a Field
• Mapping Explosion — The Real Path to a Dead Index
• Index Lifecycle — Rollover and Tier Migration
• Sampling — A Principled Way to Throw Logs Away
• The Real Cost of Faking Metrics With Logs
• Failure Modes and Diagnosis
• Closing — Log Cost Is Set by Field Design, Not Volume

Introduction — The Day Log Costs Overtook Compute Costs

It's not rare to be tallying up quarterly infrastructure costs and discover that log storage and indexing cost more than application compute. And most of those logs have never been searched by anyone.

The cause usually isn't compression ratios or storage unit prices. It's a path that starts with "let's just put everything in for now," where fields multiply, indexes grow, cutting retention runs into audit requirements, and eventually the cluster becomes unstable.

This post retraces that path and covers what should have been decided at each point. Verified against OpenSearch 3.5 and OpenTelemetry Collector v0.157.0. The Elasticsearch family shares the same concepts, but the feature corresponding to ISM is named ILM there instead.

Questions Logs Can Answer, and Questions They Can't

Let's draw the boundary first. Since a log is a record of an individual event, it answers "why." Only the log knows what value came in, which branch it took, and what exception occurred.

Conversely, there are questions that are expensive for logs to answer.

The first way to cut down logs isn't compression — it's moving to another signal any question that can be moved.

Structured Logging — What Should Become a Field

Structured logging uses keys and values instead of a string. The reason is searchability, not parsing cost.

# Bad — searching requires a regex, and if the format changes, that regex breaks
log.info(f"order {order_id} for tenant {tenant} failed after {ms}ms: {err}")

# Good — search and aggregate by field
log.info(
    "order.failed",
    extra={
        "order.id": order_id,
        "tenant.id": tenant,
        "duration_ms": ms,
        "error.type": type(err).__name__,
        "error.message": str(err)[:500],
        "http.route": route,
    },
)

Putting the event name where the message goes is the key move. Use a name with a finite set of values, like order.failed, and that alone gives you grouping and aggregation. Mix an ID into the message, and the same kind of event turns into a different string every time, which makes aggregation impossible.

Set the field-naming convention from the start. Use the OpenTelemetry semantic conventions as-is, and log and trace attribute names line up, which makes correlated lookups easy.

{
  "@timestamp": "2026-08-02T04:11:52.418Z",
  "severity_text": "ERROR",
  "severity_number": 17,
  "body": "order.failed",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "service.name": "checkout-api",
  "service.version": "2.7.1",
  "deployment.environment.name": "prod",
  "http.route": "/v1/orders/:id",
  "http.response.status_code": 502,
  "error.type": "UpstreamTimeout",
  "order.id": "A-99183",
  "tenant.id": "t-8871",
  "duration_ms": 1421
}

Having trace_id in there matters. With just this one field, you can jump from trace to log and from log to trace. If auto-instrumentation supports log correlation, it attaches without any code changes.

# Python auto-instrumentation injects trace_id and span_id into logs
export OTEL_PYTHON_LOG_CORRELATION=true
export OTEL_PYTHON_LOG_LEVEL=info

There are three principles for field design.

1. The name stays fixed, the value varies. Never put an ID or a date into the field name itself. This one rule alone blocks 90% of mapping explosions.
2. One field, one type. If duration is a number in one log and the string "1.4s" in another, the mapping locks to whichever type got indexed first, and the rest get dropped as indexing failures.
3. A large blob is body, not a field. Stack traces, request bodies, response payloads — don't turn these into searchable fields; put them in a body field that isn't indexed.

Mapping Explosion — The Real Path to a Dead Index

Dynamic mapping auto-registers fields it hasn't seen before. It's convenient, but the moment a field name comes from the data itself, it becomes dangerous.

{
  "message": "cache stats",
  "cache": {
    "user_8871_hits": 12,
    "user_8871_misses": 3,
    "user_9902_hits": 7
  }
}

A new field is created per user. With 100,000 users, that's 200,000 fields. Mapping goes into cluster state, cluster state is shared by every node, and it propagates on every change. The result unfolds in this order.

1. Indexing latency climbs. Every new field needs a mapping update, which goes through the master node
2. The master node's CPU and heap rise
3. Cluster-state propagation slows down and nodes start dropping out
4. Shard reallocation kicks in, and indexing and search both slow down together
5. Indexing gets rejected once it hits the field cap

The worst part is that once it reaches step 3, it's hard to reverse. A mapping can't be deleted — the only option is to recreate the index.

Build three layers of defense.

PUT _index_template/logs-app
{
  "index_patterns": ["logs-app-*"],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": {
      "index.number_of_shards": 3,
      "index.number_of_replicas": 1,
      "index.refresh_interval": "30s",
      "index.codec": "zstd_no_dict",
      "index.mapping.total_fields.limit": 1500,
      "index.mapping.depth.limit": 8,
      "index.mapping.nested_fields.limit": 20
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp":        { "type": "date" },
        "severity_text":     { "type": "keyword" },
        "body":              { "type": "keyword" },
        "trace_id":          { "type": "keyword" },
        "span_id":           { "type": "keyword" },
        "service.name":      { "type": "keyword" },
        "service.version":   { "type": "keyword" },
        "http.route":        { "type": "keyword" },
        "http.response.status_code": { "type": "short" },
        "error.type":        { "type": "keyword" },
        "error.message":     { "type": "text", "index": true, "norms": false },
        "duration_ms":       { "type": "integer" },
        "tenant.id":         { "type": "keyword" },
        "order.id":          { "type": "keyword", "doc_values": true, "index": true },
        "stack_trace":       { "type": "text", "index": false },
        "attributes":        { "type": "flat_object" }
      }
    }
  }
}

Let's look at what each of the three layers blocks.

First, dynamic: strict. Rejects a document if an undefined field shows up. Harsh, but honest. A visible indexing failure beats logs silently vanishing. If outright rejection feels too costly, you can set it to false to store without indexing instead.

Second, one flat_object field. Set aside a spot where unpredictable key-value pairs are bound to land. flat_object treats the whole object as a single field, so its sub-keys never get registered in the mapping. You can still look things up with dot notation, but there's no per-sub-key index, so search is slow. That trade-off is exactly what we want.

Third, a cap on the field count. Even when an incident happens, only that index stalls, not the whole cluster.

Diagnose the current state like this.

# Field count per index — thousands means you already have a problem
curl -s 'https://opensearch:9200/logs-app-000042/_mapping?pretty' \
  | jq '[paths(type=="object" and has("type")) | length] | length'

# Cluster state size and the mapping-update queue
curl -s 'https://opensearch:9200/_cluster/stats?pretty' \
  | jq '.indices.mappings'

curl -s 'https://opensearch:9200/_cluster/health?pretty' \
  | jq '{status, number_of_pending_tasks, task_max_waiting_in_queue_millis}'

# Which indices are writing to disk
curl -s 'https://opensearch:9200/_cat/indices/logs-*?v&s=store.size:desc&h=index,docs.count,store.size,pri,rep' \
  | head -20

If number_of_pending_tasks is consistently nonzero, that's a sign mapping updates are backing up. Alert on this value, and you can catch a mapping explosion before step 3.

Index Lifecycle — Rollover and Tier Migration

Don't keep the log index as one — split it by time or size. The reason is simple: deletion is instant at the index level, but expensive at the document level. Run a delete-by-query to wipe a day's worth of logs, and it triggers segment rewrites that shake the cluster.

PUT _plugins/_ism/policies/logs-app-lifecycle
{
  "policy": {
    "description": "30-day retention for application logs",
    "default_state": "hot",
    "ism_template": [
      {
        "index_patterns": ["logs-app-*"],
        "priority": 200
      }
    ],
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_primary_shard_size": "30gb",
              "min_index_age": "1d"
            }
          }
        ],
        "transitions": [
          { "state_name": "warm", "conditions": { "min_index_age": "3d" } }
        ]
      },
      {
        "name": "warm",
        "actions": [
          { "replica_count": { "number_of_replicas": 0 } },
          { "force_merge": { "max_num_segments": 1 } }
        ],
        "transitions": [
          { "state_name": "cold", "conditions": { "min_index_age": "10d" } }
        ]
      },
      {
        "name": "cold",
        "actions": [
          { "read_only": {} }
        ],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "30d" } }
        ]
      },
      {
        "name": "delete",
        "actions": [{ "delete": {} }]
      }
    ]
  }
}

Rolling over by min_primary_shard_size is safer than doing it by time. On a day traffic spikes and that day's index hits 300GB, a single shard blows past 100GB and search slows down sharply. With a size-based trigger, that day's index just splits into several instead.

Dropping replicas to 0 in the warm state trades cost for durability. Decide first whether you can tolerate losing older logs to a node failure, then make the call. For logs subject to audit, skip this step and separate them into their own index with their own retention policy.

Don't forget to confirm the policy is actually running. ISM fails silently.

# Indices with a policy attached, and their current state
curl -s 'https://opensearch:9200/_plugins/_ism/explain/logs-app-*?pretty' \
  | jq 'to_entries[] | select(.value.index != null)
        | {index: .value.index, state: .value."policy_id", step: .value.step.name, failed: .value.failed}'

# Only the managed indices that have failed
curl -s 'https://opensearch:9200/_plugins/_ism/explain/logs-app-*?pretty' \
  | jq '[to_entries[] | select(.value.failed == true) | .key]'

Sampling — A Principled Way to Throw Logs Away

If retaining everything doesn't fit the budget, you have to throw some away, and the question is what. There's one principle: throw away what's high in redundancy, not what's low in value.

Listed from highest impact down.

1. Health check and probe logs — commonly 20–40% of all logs. Their information content is close to zero
2. DEBUG and INFO on the success path — detailed logs for requests that had no error
3. The same event repeating — if the same error.type shows up thousands of times a second, keep only the top N plus a count
4. Requests for static assets — access logs for images, JS, CSS

Cut it in the collector. That's the spot where you can change policy without touching the app.

# otel-collector.yaml — logs pipeline
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

  # 1) Drop health checks entirely
  filter/drop_health:
    error_mode: ignore
    logs:
      log_record:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
        - 'attributes["http.route"] == "/metrics"'

  # 2) Keep only 10% of DEBUG on the normal path. Leave errors untouched
  probabilistic_sampler/debug:
    sampling_percentage: 10
    attribute_source: record
    from_attribute: trace_id

  # 3) Truncate large fields
  transform/trim:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - truncate_all(attributes, 4096)
          - delete_key(attributes, "http.request.body")
          - set(attributes["error.message"], Substring(attributes["error.message"], 0, 500))
            where attributes["error.message"] != nil

  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  opensearch:
    http:
      endpoint: https://opensearch.observability.svc:9200
    logs_index: logs-app
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      max_elapsed_time: 300s

service:
  pipelines:
    # Errors and warnings are always retained in full
    logs/critical:
      receivers: [otlp]
      processors: [memory_limiter, transform/trim, batch]
      exporters: [opensearch]
    # The normal path gets sampled
    logs/routine:
      receivers: [otlp]
      processors: [memory_limiter, filter/drop_health, probabilistic_sampler/debug, transform/trim, batch]
      exporters: [opensearch]

from_attribute: trace_id matters. Logs from the same trace get kept or dropped together, so what remains isn't fragmented. Judge each log record at random instead, and you can end up with only three lines left from one request's logs, useless for investigation.

Splitting pipelines by signal via routing is another approach. Route by severity to different indices with different retention, and a policy that keeps ERROR for 90 days while keeping INFO for only 7 becomes possible.

The bottom row gets forgotten often. If audit requirements mean you can't shorten retention, a structure where those logs sit as raw copies in object storage rather than the search cluster, pulled out only when needed, is far cheaper. Searchability and retention are different requirements.

The Real Cost of Faking Metrics With Logs

"We have all the logs, so why not just compute the error rate from them" is a natural thought, and at small scale it genuinely works. At scale, the cost structure flips.

Let's look at the numbers.

# Scale assumptions
rps            = 5_000          # requests per second
log_bytes      = 800            # size of one indexed log entry (bytes)
seconds_of_day = 86_400

daily_gb = rps * log_bytes * seconds_of_day / 1024**3
print(f"Daily indexed volume: {daily_gb:.1f} GB")
# Daily indexed volume: 321.8 GB

# The same information expressed as metrics
routes, status_classes, pods = 120, 5, 40
series = routes * status_classes * pods
samples_per_day = series * (seconds_of_day / 15)      # 15-second scrape
metric_bytes = samples_per_day * 2                    # roughly 2 bytes per sample after compression
print(f"{series:,} series, {metric_bytes/1024**3:.3f} GB per day")
# 24,000 series, 0.257 GB per day

The gap is over a thousandfold. And this is only looking at storage cost. The real gap widens further at query time.

The last row is the most dangerous. When an incident hits, logs surge, and that's exactly the moment log-based alerts and dashboards get slowest. It's an observability system that doesn't work when you need it.

The right direction isn't cutting logs — it's moving aggregation to metrics and keeping logs for investigation. The collector can do this conversion for you.

# Turn counts from logs into metrics and send them to Prometheus
connectors:
  count:
    logs:
      log.error.count:
        description: Log occurrence count by severity
        conditions:
          - 'severity_number >= 17'
        attributes:
          - key: service.name
          - key: error.type
          - key: http.route

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [opensearch, count]
    metrics/from_logs:
      receivers: [count]
      processors: [batch]
      exporters: [prometheusremotewrite]

What goes into the attributes list is everything. Whatever's there becomes a metric label as-is, so putting in order.id or tenant.id blows up the series count. A high-cardinality field that was fine in a log becomes a problem the moment it crosses over into a metric.

Failure Modes and Diagnosis

Pay special attention to the last row. If backpressure from the log pipeline propagates all the way to the app, you end up in a situation where the observability system kills the service. Let the collector drop when its queue is full, and alert on the drop counter. Losing logs is bad, but it's better than a service outage.

exporters:
  opensearch:
    sending_queue:
      enabled: true
      queue_size: 5000
      # Drop new data when the queue is full. Don't push backpressure back to the app
      block_on_overflow: false

Closing — Log Cost Is Set by Field Design, Not Volume

Between two organizations with a tenfold difference in log cost under the same traffic, the difference isn't the compression algorithm. One has fixed field names and has moved its aggregation questions to metrics; the other has dynamic mapping switched on while its dashboards scan raw logs.

There are three checks you can run right now. First, count the fields on your biggest log index. Four digits, and this quarter's project is already decided. Second, pull the list of indices that were actually searched over the last 30 days. Indices that were never searched often make up half the storage. Third, find the dashboard panels that scan logs and check whether they can be moved to metrics.

Further reading.

• OpenSearch: Mapping explosion
• OpenSearch: Flat object field type
• OpenSearch: Index State Management
• OpenTelemetry: Logs concepts documentation
• OpenTelemetry Collector: Filter Processor