G
GuideDevOps
Lesson 10 of 13

Distributed Tracing

Part of the Monitoring & Observability tutorial series.

The Microservices Black Box

In a simple monolithic application, if a user clicks "Checkout" and the page takes 8 seconds to load, debugging is straightforward:

  1. SSH into the single monolith web server.
  2. Read the application logs.
  3. Look at the database query execution time.

In a modern microservices architecture, a single "Checkout" click initiates a massively complex web of internal API calls.

  1. The User hits the API Gateway.
  2. The Gateway talks to the Authentication Service.
  3. It validates the session cache in Redis.
  4. The Gateway talks to the Cart Service.
  5. The Cart Service asks the Inventory Service to reserve the stock.
  6. The Cart Service asks the Payments Service to process the credit card via Stripe.
  7. The Payments Service writes the invoice to a PostgreSQL Database.

If the checkout takes 8 seconds... whose fault is it?

If you look at the logs for the Cart Service, it just says: Waiting for Payments Service. If you look at the Payments Service, it says: Waiting for Database. Metrics and Logs are completely insufficient to map this journey. You need the third pillar of observability: Distributed Tracing.

Anatomy of a Trace

Tracing maps the exact life cycle of an individual execution as it jumps across network boundaries.

A Trace consists of two core components:

1. The Trace ID

When the initial HTTP request hits the front door of your system (the API Gateway), the gateway generates a massive, universally unique identifier (e.g., TraceID: 4bf92f3577b34da6a3ce929d0e0e4736).

Every single time a microservice makes an internal HTTP/gRPC call to another microservice, it absolutely must inject this TraceID into the HTTP headers of the request. This allows the tracing backend to group 50 different internal operations together under one parent umbrella.

2. Spans

A Span represents a single, named, timed operation representing a contiguous segment of work.

A Span records:

  • The Start time and End time (duration).
  • The Name of the operation (e.g., SELECT * FROM inventory).
  • Tags (e.g., status_code=200, db.user=admin).
  • The Parent Span ID (so the system knows this span was a child of another span).

Visualizing Traces (The Waterfall)

When the Tracing System (like Jaeger, Zipkin, or AWS X-Ray) aggregates all the spans linked to a specific Trace ID, it renders them in a Waterfall UI.

Trace ID: 4bf92f... (Total Duration: 2.1s)
|
├─ API Gateway (2.1s) 
|  |
|  ├─ Auth Service (0.1s)
|  |  └─ Redis Session Check (0.05s)
|  |
|  └─ Checkout Service (1.9s)
|     |
|     ├─ Inventory Update (1.5s) 🟥 BOTTLENECK
|     |  └─ Postgres UPDATE Query (1.45s)
|     |
|     └─ Stripe Payment API (0.4s)

By glancing at the Waterfall UI, the bottleneck is instantly obvious. The Checkout Service took 1.9 seconds, but it wasn't the Checkout Service's fault. It was blocking while the Inventory Update fired a slow, un-indexed PostgreSQL query that took 1.45 seconds!

OpenTelemetry (The Modern Standard)

For a trace to work perfectly, your application code actually has to be heavily instrumented. The code has to be written to generate Spans, grab the Trace Headers, and forward them along to the next service.

Historically, this was a nightmare. If you chose Zipkin as your backend, you had to import the proprietary Zipkin library into your Node.js, Python, and Go code. If your company decided to switch to Datadog 3 years later, you had to rewrite and rip out the tracking libraries from all 20 of your microservices!

The industry realized this lock-in was unacceptable and created OpenTelemetry (OTel).

OpenTelemetry (a CNCF project formed by merging OpenTracing and OpenCensus) is a vendor-neutral, standardized framework for generating and transmitting telemetry data (Metrics, Logs, and Traces).

How it works today:

  1. You use the standardized OpenTelemetry libraries in your Python/Node/Go code.
  2. Your code sends the data to an OTel Collector generic agent running on your cluster.
  3. You configure the Collector via YAML to forward the data to any backend you want (Jaeger, Datadog, New Relic, Grafana Tempo).

If you want to switch vendors, you just change one line in the Collector YAML. No code rewrites!

The Cost of Tracing (Sampling)

Traces generate a catastrophic amount of data.

If your website handles 10,000 HTTP requests per second, and each request generates 5 internal Spans, you are asking your tracing database to ingest, index, and store 50,000 data chunks per second.

This is financially unsustainable. Furthermore, 99% of those traces represent successful, fast HTTP 200s. We don't care about the successes; we only care about tracking the failures or the slow requests.

The Solution is Sampling.

You configure your tracing system (often via the OTel Collector) to only retain a statistical subset of the data.

  • Head Sampling: Roll a metaphorical die at the API Gateway. Randomly select 1% of all new requests to be traced. Discard the other 99% immediately. (Pro: Very cheap. Con: You might miss capturing a rare failure).
  • Tail Sampling: Keep 100% of traces in temporary RAM for a few minutes. Once the request finishes, analyze it. Did it throw an HTTP 500 error? Did it take longer than 3 seconds? If yes, save it permanently to the database. If it was a fast HTTP 200, delete it from RAM. (Pro: You never miss a failure. Con: Requires massive amounts of RAM in the Collector).