Tracing

Summary

Tracing tracks the progression of a single user request as it is handled by Pomerium.

Each unit of work is called a Span in a trace. Spans include metadata about the work, including the time spent in the step (latency), status, time events, attributes, links. You can use tracing to debug errors and latency issues in your applications, including in downstream connections.

How to configure

Core
Enterprise

Shared Tracing Settings

Config Key	Description	Required
tracing_provider	The name of the tracing provider. (e.g. Jaeger, Zipkin)	✅
tracing_sample_rate	Percentage of requests to sample in decimal notation. Default is `0.0001`, or .01%	❌

Set tracing_sample_rate = 1 if you want to see all requests in the tracings.

Datadog

Datadog is a real-time monitoring system that supports distributed tracing and monitoring.

Config Key	Description	Required
tracing_datadog_address	`host:port` address of the Datadog Trace Agent. Defaults to `localhost:8126`	❌

Jaeger (partial)

Warning At this time, the Jaeger protocol does not capture spans inside the Proxy Service. Please use the Zipkin protocol with Jaeger for full support.

Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including:

Distributed context propagation
Distributed transaction monitoring
Root cause analysis
Service dependency analysis
Performance / latency optimization

Config Key	Description	Required
tracing_jaeger_collector_endpoint	Url to the Jaeger HTTP Thrift collector.	✅
tracing_jaeger_agent_endpoint	Send spans to jaeger-agent at this address.	✅

For quick local testing, use Jaeger all-in-one, which is an executable designed to launch the Jaeger UI, jaeger-collector, jaeger-query, and jaeger-agent, with an in-memory storage component.

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.45

Pomerium settings

tracing_provider: jaeger
tracing_jaeger_collector_endpoint: http://localhost:14268/api/traces
tracing_jaeger_agent_endpoint: http://localhost:6831

Open Jaeger UI at http://localhost:16686 in the browser to view Pomerium traces.

Zipkin

Zipkin is an open-source distributed tracing system and protocol.

Many tracing backends support Zipkin either directly or through intermediary agents, including Jaeger. For full tracing support, we recommend using the Zipkin tracing protocol.