Blog

Polarising Website

Enterprise Microservices Observability: Fix Trace Context First

Engineer testing trace propagation across services

Microservices observability is the ability to explain what is happening inside a distributed system from the telemetry it emits. Start by instrumenting correlated logs, metrics and distributed traces, and propagate the W3C Trace Context header on every call. Anchor your metrics on the Four Golden Signals and use OpenTelemetry so the data stays vendor-neutral.

Polarising
polarising.com
Connect Complex Systems With Confidence
Polarising integrates AI into enterprise processes, bringing engineering experience to complex systems and application development challenges.

Explore AI engineering

Key Takeaways

Effective microservices observability depends on consistent telemetry, propagated trace context, tiered sampling, and automation to enable fast, accurate diagnosis and response.

Point Details
Instrument golden signals Implement monitoring of latency, traffic, errors, and saturation for every critical service.
Propagate trace context Ensure the W3C trace headers are passed through all layers, including gateways and message queues.
Use tiered sampling Apply sampling based on service criticality, error occurrence, and request slowdowns to control costs.
Automate incident response Set SLOs and error budgets, and attach runbooks to optimize response and reduce noise.
Prioritize propagation over tools Focus on the fundamental plumbing—trace forwarding and correlation—before selecting or upgrading observability platforms.

Table of Contents

How does observability differ from monitoring?

Monitoring tells you that something is wrong. Observability lets you ask why, including questions you did not anticipate when you wrote the dashboards. The Google SRE book recommends combining white-box and black-box monitoring, alerting on symptoms worth paging a human for, and using histogram-style latency buckets to expose tail behaviour. The Four Golden Signals give you the starting set:

  • Latency: track P95 and P99 per endpoint, since averages hide the slow requests. Example: a checkout service whose P99 climbs while its mean stays flat.
  • Traffic: request rate per service and route. Example: a sudden drop in orders arriving at a payment service.
  • Errors: the rate of failed requests, including wrong results returned with a 200 status. Example: a pricing service returning empty quotes.
  • Saturation: how full the constrained resource is, such as connection pools, queue depth or CPU. Example: a database pool at its ceiling.

What do logs, metrics and traces each give you?

Logs, metrics and traces answer different questions, and together they provide complementary views of a system. Metrics are aggregated numbers that suit thresholds and alerts. Logs carry detailed textual context for one event. Traces show the path and timing of a single request across services.

  • Metrics tell you something is degrading and roughly where.
  • Logs tell you what the service believed was happening at that moment.
  • Traces tell you which hop in the chain took the time.

Correlation is what joins them. The Hackney API playbook advises using a correlationId and tracing context to link events across services, and keeping PII out of logs. A recent academic survey names correlation, topology and incident response automation as the core capabilities of an observability system for containerised microservices.

A worked example: a latency alert fires on the checkout service. A trace shows most of the time spent in a single database call from the inventory service. Logs carrying the same trace ID reveal a full connection pool and queued queries. You fix the pool, not the checkout code.

Pro Tip: Write the trace ID into every log line automatically via your logging library, so nobody has to remember to do it.

Why does trace context matter for interoperability?

Traces break wherever a component fails to pass identifiers along. W3C Trace Context standardises two HTTP headers: traceparent, which carries the trace identifiers, and tracestate, which carries vendor-specific data. In a mixed-vendor estate, that is the baseline that lets different tools stitch the same request together.

Proxies, API gateways, middleware and SaaS components must forward both headers, otherwise a single request appears as several unrelated fragments. Use this checklist:

  • Instrument inbound and outbound HTTP calls to read and write traceparent and tracestate.
  • Carry the same context in message headers for queues and event streams.
  • Sanitise tracestate so it holds only what receivers need.
  • Never place PII or secrets in trace headers, because they travel to every downstream party.

How should you sample traces?

Keeping every trace is rarely affordable at scale, so sampling is a design decision, not an afterthought. OpenTelemetry’s sampling guidance notes that representative samples can be as low as 1% for high-volume systems when traffic is stable.

Head sampling decides at the start of a trace, either deterministically or by probability. It is cheap and simple, but it cannot know whether the request will turn out slow or failed. Tail sampling decides after the trace completes, so it can keep errors and slow requests. The OpenTelemetry tail sampling write-up points out that it cuts ingest costs and prioritises interesting traces, but needs collector capacity planning and decision windows sized to your slowest requests.

  • Always keep error and slow traces.
  • Tier services by criticality and give each tier its own rate.
  • Consider a hybrid: a light head sample plus tail rules for exceptions.
  • Monitor the samplers and collectors themselves.

The OpenTelemetry demo’s criticality-based policy always samples critical services, takes high-criticality services at 50%, medium at 10% and low at 1%, and always captures errors or slow traces. It is a useful template to adapt, not a universal setting.

What does a scalable observability pipeline look like?

Most fragmented traces come from topology, not instrumentation. Spans with the same trace ID must reach the same OpenTelemetry Collector instance for tail sampling to see the whole trace. Otherwise, each collector sees a slice and decides on partial information.

  • Agent plus collector: run a lightweight agent near each workload and forward to central collectors.
  • Collector deployment: use a load-balancing exporter to route traces by trace ID to collectors that make sampling decisions.
  • Monitor the pipeline: collectors drop data silently when overloaded, so track their queue sizes and refusal rates.
  • Map dependencies: automatic topology discovery shows which services call which, so you can see blast radius during an incident.

The academic survey above lists topology, in the form of dependency graphs with automatic detection, alongside correlation as a defining capability.

How do SLOs and runbooks turn telemetry into action?

Telemetry only pays off when it changes how people respond. Define service level objectives and error budgets, then let them decide what wakes a person and what waits in a dashboard.

  1. Set an SLO for each user-facing journey, built on the golden signals.
  2. Page only on actionable symptoms, such as a fast-burning error budget, and keep causes on dashboards.
  3. Add synthetic checks for end-to-end capabilities like login or payment.
  4. Attach a runbook and an owning team to every service so an alert reaches the right people with the first steps in hand.
  5. Review noisy alerts after each incident and delete or tune them.

Pro Tip: If an alert fires and the responder does nothing, it should not page anyone.

How do you keep the observability pipeline secure and independent?

Attackers go after logs first. The NCSC secure design principles recommend that, for mission-critical systems, the observability pipeline be independent of the system it monitors, so a compromise of production cannot hide detection or tamper with logs.

  • Run collection and storage on separate infrastructure and credentials from the workloads.
  • Apply least privilege: services may write telemetry but not read or delete it.
  • Keep communication flows simple, so there are fewer paths to defend.
  • Protect log integrity so records can support forensics.
  • In regulated sectors, set retention and integrity rules deliberately rather than inheriting defaults.

Our perspective: a practical checklist for enterprise microservices

We build and integrate enterprise estates across finance, healthcare, telecommunications and logistics, and the same observability gaps keep appearing. Our checklist:

  • Instrument the Four Golden Signals for every service.
  • Enable W3C trace context end to end, including gateways and messaging.
  • Apply tiered sampling with errors always kept.
  • Centralise collectors and monitor them.
  • Set SLOs and error budgets for each journey.
  • Write runbooks with named owners.

Our senior nearshore teams in the UK and Portugal deliver cloud-native application development, enterprise integration and event-driven systems, where observability is designed in from the start. Our usual path is assessment, then pilot, then delivery, through AI Pods or an enterprise integration engagement.

What makes observability hard in microservices, and what helps?

The core difficulty is that one user action crosses many services, languages and teams, each with its own logging habits. The academic survey cited earlier frames the answer as correlation, topology and incident response automation. In practice, the challenges and the matching practices look like this.

Inconsistent telemetry. Teams name fields differently, so queries break across services. Agree a shared schema and put it in a common library.

Broken context. A single service that drops traceparent ruins every trace passing through it. Test propagation in CI, using a synthetic request that must produce a connected trace.

Noise. Health checks and redundant lines drown useful events. The Hackney playbook advises avoiding redundant logs and filtering noisy entries such as health checks.

Ownership gaps. Nobody owns the dashboard for a shared dependency. Attach owner metadata to each service and route alerts from it.

Tool sprawl. Several vendors collect overlapping data. Standardise on OpenTelemetry for instrumentation so you can change backends without re-instrumenting.

Start narrow. Pick the three journeys that matter most to the business, instrument them well, and widen coverage once the habits are established. Teams that try to instrument everything at once usually end up with data nobody trusts.

Where do AI and machine learning help with observability?

AI helps most where humans are slowest: sifting large volumes of correlated telemetry. The CNCF’s 2025 observability trends review identifies AI and automation, together with vendor-agnostic frameworks such as OpenTelemetry, as drivers of better correlation across telemetry, cost optimisation and multi-cloud visibility.

Anomaly detection learns a service’s normal pattern, including daily and weekly cycles, and flags deviations that fixed thresholds miss. It is most useful for slow drifts, such as a P99 creeping upward over a fortnight.

Correlation and root-cause suggestion groups related alerts and points to the likely upstream cause, using the dependency graph. This is where the topology work from the pipeline section pays off.

Predictive insight projects trends such as disk growth or connection pool exhaustion so you can act before an SLO is breached.

Treat these as assistance, not authority. Models are only as good as the telemetry underneath them: inconsistent labels and broken traces produce confident but wrong conclusions. Fix instrumentation first, keep a human in the loop for paging decisions, and check suggestions against traces before acting on them.

How does observability affect scalability and performance?

Observability is how you find the limit before your users do. The saturation signal shows which resource runs out first, and traces show which call dominates request time, so you scale the component that matters rather than the whole estate.

Three practical effects stand out. First, targeted scaling: if traces show that one downstream service accounts for most of the latency, scaling the frontend achieves nothing. Second, capacity planning: saturation trends, such as pool utilisation and queue depth, tell you when to add capacity ahead of demand. Third, regression detection: comparing latency histograms before and after a release catches tail latency regressions that averages hide, which is exactly why the Google SRE guidance favours histogram-style bucketing.

Be aware that instrumentation has its own cost. Heavy logging and high-cardinality metrics consume CPU, memory and network. Measure the overhead, and use sampling and field discipline to keep it small. The goal is to make performance work evidence-led: form a hypothesis from a trace, change one thing, and confirm the effect in the same signals.

How do observability tools integrate with Kubernetes and Istio?

Kubernetes and service meshes give you telemetry almost for free, provided you wire it into the same pipeline as application data. On Kubernetes, run collectors as a node-level agent and a central gateway layer, and enrich telemetry with pod, namespace and deployment metadata so signals can be filtered by team and service. That metadata is also what makes ownership routing work.

A service mesh such as Istio adds a second source of truth. Its sidecar proxies emit request metrics and can generate spans for every hop, covering traffic, errors and latency without touching application code. Two cautions apply. The mesh can only continue a trace if the application forwards trace headers on its own outbound calls, so application instrumentation is still required. And mesh telemetry should flow into the same OpenTelemetry-based pipeline, so you do not end up with two disconnected views.

In integration-heavy estates, the same principle extends to the platforms that connect systems. We work with Boomi, MuleSoft, TIBCO and Solace, and the discipline is the same on each: make sure the platform forwards traceparent and tracestate, and that messages carry correlation context, so a trace survives the trip through an integration layer or event broker.

How do observability tools integrate with Kubernetes and Istio? — overview diagram

What does effective observability look like in practice?

This section describes patterns rather than quantified results. You can read more of our delivery work on our insights pages.

Integration revamp in a regulated airline estate. When the integration layer is rebuilt, the first thing teams lose is end-to-end visibility, because old point-to-point flows had informal monitoring. Designing correlation IDs and trace propagation into the new integration layer from day one means each booking or notification can be followed across services and brokers. That turns incident calls from guesswork into reading a trace.

Event-driven logistics flows. Asynchronous systems hide latency in queues. The pattern that works is carrying trace context in message headers, then alerting on consumer lag and queue depth as saturation signals. Support teams can then answer “where is this parcel event?” from a single trace rather than interrogating several teams.

Finance and healthcare estates. Here the emphasis falls on the security principles above: a pipeline independent of production, strict access control, and deliberate retention. Engineers still get fast diagnosis, while auditors get logs they can trust.

The common thread across all three is that the fundamentals did not change: correlated telemetry, propagated context, sensible sampling and clear ownership.

What does effective observability look like in practice? — overview diagram

How can you control the cost of telemetry?

Telemetry volume grows with traffic and with the number of services, so cost needs design rather than periodic firefighting. The CNCF review links vendor-agnostic frameworks and automation to cost optimisation, and the practical levers are well established.

Sample deliberately. Use tiered tail sampling so critical services and error traces are kept and low-value traffic is thinned, as in the OpenTelemetry criticality pattern described earlier.

Filter at the source. Drop health-check noise and redundant log lines before they leave the node. This is cheaper than storing them and then ignoring them.

Control metric cardinality. Labels such as user ID or request ID on a metric multiply the number of series. Keep high-cardinality detail in traces and logs, and keep metrics for aggregates.

Tier retention. Keep recent data in fast storage, move older data to cheaper storage, and keep only what compliance requires for the full period.

Budget the pipeline. Collectors and samplers consume real compute. Include them in capacity planning, and watch their drop rates.

Review telemetry cost per service on a regular cadence. Teams that see what their own services cost tend to trim waste without being asked.

Our view: fix propagation before buying more tools

Most of the advice on this topic starts with tooling: which platform, which dashboards, how many panels. We think the order is wrong. The practical challenge is often the plumbing. Correlation, topology and propagated context are what the academic survey calls the core capabilities, and no backend can recover a trace that a proxy has already cut in half.

What is overrated is dashboard volume. A wall of charts feels like visibility, but a team that pages on twenty causes will miss the one symptom that matters. What is underrated is the unglamorous work: forwarding traceparent through every gateway and queue, keeping collectors routed by trace ID, and writing the runbook while the incident is still fresh.

Prioritise in this order. First, propagation and a shared log schema. Second, the golden signals with SLO-based paging. Third, tiered sampling that keeps errors. Only then evaluate tools, and prefer OpenTelemetry so the choice stays reversible. For integration-heavy enterprises, extend the same discipline to brokers and iPaaS platforms, because that is where traces most often disappear.

— Polarising Team

Need senior help building observability at scale?

If you are a CTO or integration manager facing fragmented traces, noisy alerts or rising telemetry bills, we can help. Our senior nearshore teams combine enterprise integration experience with cloud-native engineering, so observability is built into your estate rather than bolted on. Ask for an assessment or a pilot through AI Pods, or contact us at Polarising.

Polarising

FAQ

What are the four pillars of observability?

The term’s meaning varies across teams. It can refer to metrics, logs and traces, while some teams also include events or profiles. Definitions vary, so check which set your tooling assumes. For reliability signals specifically, the Four Golden Signals are latency, traffic, errors and saturation.

What are observability platforms?

Observability platforms collect, store and analyse telemetry such as logs, metrics and traces, and let you search and correlate it. Choosing tools that accept OpenTelemetry data keeps your instrumentation portable between them.

What are effective monitoring tools for microservices?

Look for tools that support OpenTelemetry, W3C trace context and correlation across logs, metrics and traces. The right choice depends on your platform and data volumes, so judge candidates by how they handle sampling, topology mapping and cost, not by feature lists.

How should you start with microservices observability?

Propagate trace context everywhere, add a correlation ID to logs, and instrument the golden signals for your most important journeys. Add tiered sampling and SLO-based alerting once that foundation holds.

Sources