Docs / dflux EdgeGuard
ContactGet started

Setting up observability

dflux EdgeGuard emits Prometheus metrics, structured logs, OTLP traces, and a deep health endpoint. This guide covers how to scrape, dashboard, alert, and trace through a typical deployment.

Prometheus metrics

Metrics are exposed at /metrics on the SBI listener and, when set, on the dedicated observability listener (observability.metrics_listen). The admin gRPC port does not serve Prometheus. Common request:

Bash

Headline series to scrape and dashboard:

  • edge_requests_total (counter) — total requests, by method, target_nf_type, status_code.
  • edge_request_duration_seconds (histogram) — end-to-end request latency, by method and target_nf_type.
  • edge_active_requests (gauge) — in-flight requests.
  • edge_errors_total (counter) — proxy-side errors by category.
  • edge_filter_decisions_total (counter) — decisions by filter and decision.
  • edge_rate_limit_exceeded_total (counter) — denials by rate-limit rule.
  • edge_producer_requests_total (counter) — forwarded requests by nf_type, producer, and result.
  • edge_circuit_breaker_open (gauge) — per-producer breaker state (1=open, 0=closed), by nf_type and producer.
  • edge_sync_lag_seconds (gauge) — seconds since the last successful policy sync from the control-plane store.
  • edge_policy_reload_errors_total (counter) — failed policy reload attempts.
  • edge_diameter_requests_total (counter) — Diameter requests by application, command, and result_code.
  • edge_diameter_forward_total (counter) — forwarded requests by result.
  • edge_diameter_forward_duration_seconds (histogram) — Diameter forward latency by result.
  • edge_diameter_active_peers (gauge) — peers currently up.

Full label list lives in Reference → Metrics.

Scrape config

YAML

Scrape every 15s. Most series are cheap; the three histograms — edge_request_duration_seconds, edge_filter_duration_seconds, and edge_diameter_forward_duration_seconds — carry the bulk of the cardinality.

Dashboards

A four-panel core dashboard covers most operational questions:

  1. Request rate by target_nf_type and decision (allowed / denied).
  2. P50 / P95 / P99 latency from edge_request_duration_seconds.
  3. Filter decisions stacked by filter (auth / policy / rate-limit / transformation).
  4. Producer health from edge_producer_requests_total + circuit-breaker state.

For Diameter:

  1. Peer count from edge_diameter_active_peers.
  2. Per-command request rate from edge_diameter_requests_total.
  3. Forward latency P95 from edge_diameter_forward_duration_seconds.
  4. Loop detection counter edge_diameter_loop_detected_total.

Alerts to set early

  • Sync lag rising. edge_sync_lag_seconds climbing past your sync interval means the proxy can't reach the control-plane store; running policy is going stale.
  • Policy reload errors > 0. edge_policy_reload_errors_total incremented — a reload was rejected and the previous policy is still in force.
  • Producer error rate > 5% over 5m. Producer-side issue or routing misconfiguration.
  • Circuit breaker open on any producer. edge_circuit_breaker_open is 1 — that producer is out of rotation. Page if it persists.
  • Diameter active peers below expected count. A peer dropped and isn't reconnecting.

Deep health

Plain liveness (GET /health on the SBI or metrics listener) tells you "the process is up". d3x-edgectl health-deep tells you "the process can actually serve traffic" — it checks producer availability, control-plane store reachability, and Diameter status:

Bash

The response carries a checks map with one entry each for producers, store, and diameter. It returns 200 only when none of those checks is fail (a 503 otherwise). Use it as a Kubernetes readiness probe; never as a liveness probe — you don't want kube to restart the proxy because a producer is briefly down.

OTLP tracing

Configure under observability.tracing:

YAML

The proxy emits spans for: inbound request, each filter, routing decision, outbound forward, and response stage. Trace context (W3C traceparent) is propagated through to the producer so downstream traces stitch into a single timeline.

Logs

Decisions are emitted as structured zerolog records — the same JSON stream you ship to a log pipeline or SIEM for the request-level trail. Choose between human-readable console output and one JSON object per line.

YAML
trace leaks bodies
level: trace includes request and response bodies. Useful for debugging, unsafe in production where bodies may contain PII.

Change the log level on a running daemon without a restart — useful when chasing an issue. The level accepts debug, info, warn, or error:

Bash

Profiling

Set observability.pprof.enabled: true to expose net/http/pprof on the SBI listener. Useful for CPU or memory profiles during load tests; never enable in production on a publicly-reachable listener.