[Prometheus] Service Discovery Mechanism: Complete Analysis

prometheus.yml scrape_configs
        |
        v
+-------------------+
| Discovery Manager |
|                   |
|  +-- Provider 1 (kubernetes_sd)  -- goroutine
|  +-- Provider 2 (consul_sd)     -- goroutine
|  +-- Provider 3 (file_sd)       -- goroutine
|  +-- Provider 4 (static_config) -- goroutine
|                   |
+--------+----------+
         |
    Target Groups Channel
         |
         v
+-------------------+
| Scrape Manager    |
+-------------------+

Provider detects change
    |
    v
Sends update to internal channel
    |
    v
Discovery Manager receives
    |
    v
Waits 5 seconds for additional updates (debouncing)
    |
    v
Merges latest target groups from all providers
    |
    v
Delivers complete target map to Scrape Manager

type Discoverer interface {
    Run(ctx context.Context, up chan<- []*targetgroup.Group)
}

TargetGroup:
  Source: "kubernetes/pod/default/nginx-abc123"  (unique identifier)
  Targets:
    - __address__: "10.0.1.5:8080"
    - __address__: "10.0.1.6:8080"
  Labels:
    __meta_kubernetes_namespace: "default"
    __meta_kubernetes_pod_name: "nginx-abc123"
    ...

Registration process:
1. Parse *_sd_config sections from config file
2. Call NewDiscoverer factory for the SD type
3. Register provider with Discovery Manager
4. Execute Run() in separate goroutine

1. node:
   - Kubernetes node list
   - __address__: Node Kubelet address
   - __meta_kubernetes_node_name, __meta_kubernetes_node_label_*

2. service:
   - Kubernetes Service list
   - __address__: Service ClusterIP:Port
   - __meta_kubernetes_service_name, __meta_kubernetes_service_port_name

3. pod:
   - Kubernetes Pod list
   - __address__: Pod IP:Container Port
   - __meta_kubernetes_pod_name, __meta_kubernetes_pod_container_name

4. endpoints:
   - Kubernetes Endpoints list
   - __address__: Individual endpoint address
   - __meta_kubernetes_endpoints_name

5. endpointslice:
   - Kubernetes EndpointSlice list (Endpoints extension)
   - More efficient for large clusters

6. ingress:
   - Kubernetes Ingress list
   - __address__: Ingress host
   - __meta_kubernetes_ingress_name, __meta_kubernetes_ingress_path

Kubernetes SD Watch operation:

1. Initial sync:
   a. List API to retrieve full resource list
   b. Convert all resources to TargetGroups
   c. Send full list to Discovery Manager

2. Watch start:
   a. Connect to Watch API event stream
   b. Incremental updates based on resourceVersion

3. Event processing:
   ADDED    -> Create new TargetGroup
   MODIFIED -> Update existing TargetGroup
   DELETED  -> Update TargetGroup with empty Targets

4. Error handling:
   Watch disconnect  -> Auto-reconnect
   410 Gone error    -> Full re-list
   Timeout           -> Restart Watch

Informer structure:
  Reflector
    |-- List: Initial full sync
    |-- Watch: Incremental update reception
    |-- Store: Save to local cache
    v
  Informer
    |-- EventHandler: Register event callbacks
    |-- Indexer: Indexes for efficient lookup
    v
  Prometheus SD
    |-- Convert events to TargetGroups
    |-- Deliver to Discovery Manager

Common:
  __meta_kubernetes_namespace

Pod role:
  __meta_kubernetes_pod_name
  __meta_kubernetes_pod_ip
  __meta_kubernetes_pod_container_name
  __meta_kubernetes_pod_container_port_name
  __meta_kubernetes_pod_container_port_number
  __meta_kubernetes_pod_label_*
  __meta_kubernetes_pod_annotation_*
  __meta_kubernetes_pod_node_name
  __meta_kubernetes_pod_ready
  __meta_kubernetes_pod_phase

Node role:
  __meta_kubernetes_node_name
  __meta_kubernetes_node_label_*
  __meta_kubernetes_node_annotation_*
  __meta_kubernetes_node_address_*

Service role:
  __meta_kubernetes_service_name
  __meta_kubernetes_service_port_name
  __meta_kubernetes_service_port_number
  __meta_kubernetes_service_label_*
  __meta_kubernetes_service_annotation_*

Stage 1: relabel_configs (before scraping)
  - Process __meta_* labels from discovery
  - Determine target labels (job, instance, etc.)
  - Decide target keep/drop

Stage 2: metric_relabel_configs (after scraping)
  - Transform labels of collected metrics
  - Drop unnecessary metrics
  - Transform label names/values

replace:     Match source label via regex, set target label
keep:        Drop targets not matching regex
drop:        Drop targets matching regex
hashmod:     Hash source label, modulo to target label
labelmap:    Transform matching label names
labeldrop:   Drop labels matching regex
labelkeep:   Drop labels not matching regex
lowercase:   Convert target label value to lowercase
uppercase:   Convert target label value to uppercase
keepequal:   Keep targets where source equals target label
dropequal:   Drop targets where source equals target label

__meta_* labels + __address__ + __scheme__ + __metrics_path__
                    |
                    v
          Apply relabel_configs sequentially
                    |
                    v
          __address__ -> copy to instance label
          Remove internal labels (__scheme__, __metrics_path__, etc.)
                    |
                    v
          Final target label set determined

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: (.+)
    replacement: __meta_kubernetes_pod_ip:$1
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)

relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    action: keep
    regex: production|staging

scrape_configs:
  - job_name: 'file_sd'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.json'
        refresh_interval: 5m

[
  {
    "targets": ["10.0.1.1:9090", "10.0.1.2:9090"],
    "labels": {
      "env": "production",
      "team": "backend"
    }
  }
]

- targets:
    - '10.0.1.1:9090'
    - '10.0.1.2:9090'
  labels:
    env: production
    team: backend

File SD operation:
1. Initial load: Read all files matching the pattern
2. inotify watch: Subscribe to file change events on Linux
3. Periodic polling: Full reload every refresh_interval (default 5m)
4. File change: Re-parse only changed files, update TargetGroups
5. File deletion: Remove all targets from that file

Custom SD process (external):
1. Query proprietary service registry
2. Generate results as JSON/YAML files
3. Save to directory monitored by Prometheus

Prometheus (File SD):
1. Detect file changes
2. Update target list
3. Start/stop scraping

scrape_configs:
  - job_name: 'http_sd'
    http_sd_configs:
      - url: 'http://service-registry:8080/targets'
        refresh_interval: 30s

[
  {
    "targets": ["10.0.1.1:9090"],
    "labels": {
      "__meta_datacenter": "us-east",
      "__meta_env": "production"
    }
  }
]

HTTP SD processing:
1. Send GET request to URL every refresh_interval
2. Parse response JSON
3. Convert to TargetGroups
4. Deliver to Discovery Manager
5. On HTTP error, keep previous results
6. Alert via metrics on consecutive failures

Consul SD:
- Uses Consul Service Catalog API
- Watch/Blocking Query for change detection
- Filtering by service name, tags, datacenter
- __meta_consul_service, __meta_consul_tags meta labels

EC2 SD:
- Uses AWS EC2 DescribeInstances API
- Filtering by region, availability zone, tags
- IAM role or access key authentication
- __meta_ec2_instance_id, __meta_ec2_tag_* meta labels
- Polling based on refresh_interval (no Watch support)

DNS SD:
- DNS SRV or A/AAAA record lookups
- Periodic polling approach
- Useful for simple environments
- __meta_dns_name meta label

Target lifecycle:
  Discovered
      |
      v
  Relabeled
      |
      +-- Drop decision --> Dropped
      |
      v
  Active (scraping starts)
      |
      +-- Scrape success --> up=1
      +-- Scrape failure --> up=0
      |
      v
  Disappeared
      |
      v
  Stale (stale marker added)
      |
      v
  Removed (Scrape Loop terminated)

1. drop action applied in relabel_configs
2. Not matching keep action in relabel_configs
3. __address__ label is empty
4. Duplicate target (identical label set)

Dropped targets appear in the "Dropped Targets" section of the /targets UI,
useful for debugging discovery and relabeling configuration.

Built-in metrics:
  up: 0 or 1 (scrape success)
  scrape_duration_seconds: scrape duration
  scrape_samples_scraped: samples collected
  scrape_samples_post_metric_relabeling: samples after relabeling
  scrape_series_added: newly added series

API endpoints:
  GET /api/v1/targets: full target list and status
  GET /api/v1/targets/metadata: per-target metric metadata

Kubernetes SD performance tips:
1. Namespace restriction: Narrow watch scope with namespaces field
2. Label selectors: Filter resources with selectors field
3. attach_metadata: Disable if unnecessary
4. EndpointSlice: More efficient than Endpoints for large clusters

Key metrics:
  prometheus_sd_discovered_targets: discovered targets (per SD type)
  prometheus_sd_received_updates_total: received updates
  prometheus_sd_updates_total: delivered updates
  prometheus_sd_updates_delayed_total: delayed updates

1. Targets not discovered:
   - Check SD configuration (namespace, selector, etc.)
   - Check Prometheus logs for SD errors
   - Verify discovered targets on /service-discovery UI

2. Targets in Dropped state:
   - Check relabel_configs rule order
   - Validate keep/drop action regex
   - Check Dropped Targets on /targets page

3. Target labels differ from expected:
   - Check __meta_* labels (/service-discovery page)
   - Verify relabel_configs replace rules
   - Check honor_labels setting for label conflicts

1. /service-discovery UI:
   - Shows raw target groups from each SD provider
   - Full __meta_* label visibility

2. /targets UI:
   - Active/dropped target list
   - Per-target health, last scrape time
   - Applied labels

3. Prometheus logs:
   - Enable detailed SD logs with --log.level=debug
   - Per-provider connection/error logs