Nobody sets out to spend more on telemetry than on the systems it observes. It happens gradually, through a series of individually reasonable decisions, and the mechanism is almost always the same: cardinality.
The pattern is familiar. An incident happens. The retrospective concludes you
couldn’t diagnose it because you lacked a dimension on a metric — you could see
that latency was bad but not for which customer. So you add customer_id as a
label. It’s one line of code. It solves a real problem.
Then it’s on the bill.
What cardinality actually costs#
A metric with no labels is one time series. Add a label with 5 values and you have 5 series. Add another with 20 and you have 100.
The multiplication is the part that catches people. Cardinality isn’t additive
across labels, it’s multiplicative. A request duration metric labelled with
endpoint (50), method (4), status_code (8), and region (3) is already
4,800 series before anyone has added anything user-specific.
Now add customer_id with 10,000 values. That’s 48 million series from one
metric.
Each series carries storage, ingestion, indexing, and — the part people underestimate — query cost. A query touching a high-cardinality metric has to scan and aggregate across all of it. This is why dashboards get slower over months with no apparent change: nothing broke, the cardinality grew.
The compounding failure is that query latency degrades exactly when you need it. Incident response depends on fast ad-hoc queries. A metrics backend struggling under cardinality is slowest under load, which is when you’re querying it most.
The dimensions that will get you#
There’s a reliable list of labels that seem useful and are cardinality catastrophes:
- User or customer identifiers — cardinality equal to your user count, growing with your business
- Request or trace IDs — unbounded and unique per request, the worst possible case
- Full URL paths — unbounded if they contain identifiers (
/orders/8a3f...) - Error messages as labels — often contain interpolated values, so effectively unbounded
- Timestamps or dates in labels — unbounded by construction
- Container, pod, or instance IDs — bounded at any moment, but churning, which produces unbounded series over time
That last category is the subtle one. Pod IDs look bounded — a few hundred at a time. But every deploy replaces them, so over a retention window you accumulate series for every pod that ever existed. Frequent deploys plus autoscaling produces cardinality growth that correlates with deployment velocity rather than traffic, which makes it genuinely hard to predict.
Where high-dimensional data belongs#
The mistake is treating “I need this dimension” as equivalent to “this must be a metric label.” Metrics are one signal type with a specific cost model, and they’re the wrong home for high-cardinality data.
Traces are built for it. A span can carry arbitrary attributes — user ID, request ID, full path, feature flags — because traces are sampled and stored as individual records rather than aggregated across a dimension space. Cardinality is nearly free here. This is the correct destination for most of what people try to put on metrics.
Logs handle it too, with structured fields. More expensive to query at scale, but no cardinality explosion — a log line with a user ID costs one log line.
Exemplars are the bridge. An exemplar attaches a sample trace ID to a metric
data point. You get a low-cardinality metric — p99 latency by endpoint — plus
a direct link to a representative slow trace. This is the highest-value
integration between metrics and traces and it’s consistently underused.
The mental model that resolves most of these decisions:
Metrics answer “is something wrong and how much.” Traces answer “for whom and why.”
If a dimension is only needed to answer why, it belongs on a trace. Adding it to a metric is paying aggregate cost for detail you only need occasionally.
Deciding what not to measure#
The uncomfortable implication is that some questions should be deliberately unanswerable from metrics. That feels wrong to engineers — more data is better — but it’s the whole discipline.
Questions worth asking before adding a label:
What decision does this dimension enable? If the answer is “we might want to slice by it someday,” don’t. Speculative dimensions are where most cardinality originates.
Is this bounded, and will it stay bounded? status_code is bounded forever.
endpoint is bounded until someone adds a path parameter. Ask what the value
looks like in two years.
Could a trace answer this? Usually yes, and usually better, since a trace carries surrounding context a metric label never will.
Would an aggregate serve? customer_tier (3 values) instead of customer_id
(10,000) answers most of the questions people actually ask, at 0.03% of the
cost.
That last substitution is worth internalizing. The genuine need is almost never “latency for customer 47,203.” It’s “are enterprise customers seeing worse latency than free-tier.” A bucketed dimension answers it at negligible cost.
Governance that works#
Individual discipline doesn’t hold across a large engineering organization. A few structural measures do:
Monitor cardinality growth as a metric. Series count per metric, tracked over time, alerting on rate of growth rather than absolute level. Growth is what tells you something changed; absolutes just tell you how big you already are.
Enforce limits at the collector. A collector tier lets you drop or aggregate high-cardinality labels centrally, as policy, without redeploying services. This is also the only mechanism that works against a label added by a team that didn’t know the rules.
Attribute cost to teams. Cardinality is a tragedy of the commons when the bill is central. Visibility per team changes behavior faster than any guideline.
Review at merge time. New metric labels in a diff are worth the same scrutiny as a new database index. Both are cheap to add and expensive to live with.
Set retention by signal type. Not everything needs the same retention. Short full-fidelity retention plus longer aggregated retention captures most of the value at a fraction of the cost — accepting that some historical questions become unanswerable at detail. State that tradeoff explicitly rather than discovering it later.
If you’ve already got a problem#
Retroactive cleanup is harder because dashboards and alerts depend on existing labels.
Start by finding the top metrics by series count. The distribution is almost always extreme — a handful of metrics account for the large majority of series. You don’t need a broad cleanup, you need to fix three or four things.
For each, determine whether the dimension is actually queried. Most backends can show which series have been read. A dimension nobody queries is pure cost, and removing it breaks nothing.
Where a dimension is used, look for the aggregate substitution — replace the high-cardinality label with a bucketed one and move the detail to traces. This usually requires updating a small number of dashboards, which is a contained piece of work with an immediately visible payoff.
The framing that gets it prioritized#
Cardinality work is invisible until it isn’t. It’s easiest to fund right after a budget review, and by then you’re making decisions under pressure rather than deliberately.
The argument that lands isn’t about cost — finance will make that argument for you eventually. It’s about incident response:
Our metrics backend is slowest when we query it hardest, which is during incidents. Cardinality is why. Reducing it makes diagnosis faster.
That reframes it from a cost-cutting exercise into a reliability improvement, which is both more accurate and considerably easier to prioritize. The savings are real, but they’re the secondary benefit.
The cost conversation arrives whether you plan for it or not. Choosing your retention and cardinality tradeoffs deliberately is strictly better than having them imposed by someone reading a bill.