[{"content":"","date":"August 1, 2026","externalUrl":null,"permalink":"/tags/ai/","section":"Tags","summary":"","title":"Ai","type":"tags"},{"content":"","date":"August 1, 2026","externalUrl":null,"permalink":"/categories/ai-enabled-software/","section":"Categories","summary":"","title":"AI-Enabled Software","type":"categories"},{"content":"","date":"August 1, 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"August 1, 2026","externalUrl":null,"permalink":"/tags/data-quality/","section":"Tags","summary":"","title":"Data-Quality","type":"tags"},{"content":"There\u0026rsquo;s a moment in most AI feature projects where someone asks whether the latest change made things better, and the room goes quiet. Not because nobody cares — because nobody can answer. The demo looks good. It looked good last week too, with different code.\nThis is the position a lot of teams are in. The feature is live, people use it, and the team is making changes based on vibes and complaints. It\u0026rsquo;s not negligence. It\u0026rsquo;s that the obvious answer — build a test set with correct answers and measure against it — doesn\u0026rsquo;t survive contact with the problem.\nWhy ground truth usually doesn\u0026rsquo;t exist # For a classifier, ground truth is tractable. Someone labels a few thousand examples, you hold some out, and you have a number.\nFor most AI features shipped into products, it isn\u0026rsquo;t:\nThere\u0026rsquo;s no single correct output. If the feature summarizes a document or answers a question from a corpus, there are many acceptable answers and many unacceptable ones, and the boundary is not crisp. You can\u0026rsquo;t diff against an expected string.\nLabelling requires the expertise you\u0026rsquo;re trying to scale. If the domain expert who\u0026rsquo;d have to write the correct answers is the bottleneck the feature exists to relieve, then building a large labelled set costs roughly what the feature was supposed to save.\nThe distribution moves. Your corpus changes, user behavior changes, and a test set built in March quietly stops representing what the system sees in September. A stale evaluation set is worse than none, because it produces confident numbers about a world that no longer exists.\nCorrectness depends on context you don\u0026rsquo;t have at label time. The same answer can be right for one user and wrong for another, depending on what they already knew and what they were trying to do.\nSo the honest starting position is: you will not get a clean accuracy number, and pretending otherwise is how teams end up optimizing a metric that doesn\u0026rsquo;t correspond to user experience.\nWhat to build instead # The goal shifts from measuring quality absolutely to detecting changes in quality reliably. That\u0026rsquo;s a weaker claim, and it\u0026rsquo;s achievable, and it\u0026rsquo;s most of what you actually need — because nearly every real question is comparative. Did this change help? Is it getting worse? Should we roll back?\nThree components, in increasing order of cost and decreasing order of how often you can run them.\n1. A small curated regression set # Build 50–200 examples by hand. Not a representative sample — a deliberately adversarial one. Include:\nThe queries that broke it before Edge cases you know are hard A few easy ones, as a canary for catastrophic regressions Cases where the correct behavior is to decline For each, record what a good response must contain and what it must not. Not a gold string — a set of assertions. \u0026ldquo;Must cite a document from the correct section.\u0026rdquo; \u0026ldquo;Must not state a figure that doesn\u0026rsquo;t appear in the source.\u0026rdquo; \u0026ldquo;Must decline.\u0026rdquo;\nThis is cheap to build, cheap to run, and catches the class of regression that matters most: the change that broke something that used to work. It will not tell you how good the system is. It will tell you, reliably, when you\u0026rsquo;ve made it worse.\nThe discipline that makes this work is adding to it every time something goes wrong in production. A regression set that grows from real failures becomes sharply targeted at your actual weaknesses within a few months.\n2. Behavioral signals from real usage # Users tell you a great deal without being asked, provided you instrumented for it before you needed it.\nCitation click-through. Did they open the source? For a retrieval-backed feature, this is a reasonable proxy for whether the answer seemed worth verifying. Reformulation rate. Did they immediately rephrase and ask again? Usually a signal the first answer missed. Abandonment. Did they leave without engaging with the result? Downstream action. Did they do the thing the feature was supposed to help them do? Each of these is weak individually, and each has a failure mode you have to say out loud:\nReformulation might mean the answer was bad — or that it was good and prompted a sharper follow-up. Citation clicks might mean the answer was credible enough to verify, or so implausible the user went to check. Abandonment might mean the answer was so complete that nothing further was needed.\nThis is why you track them as a set and watch for correlated movement. Reformulation up, citation clicks down, and abandonment up together is a real signal. Any one of them alone is noise with a story attached.\n3. Structured human review, sampled # You cannot review everything. You can review a sample, consistently, with a rubric.\nPull a stratified sample weekly — some high-confidence responses, some low-confidence, some from query types you care about. Have a reviewer score them on a small number of specific dimensions rather than a single quality score. \u0026ldquo;Was the retrieved context relevant?\u0026rdquo; and \u0026ldquo;was the answer supported by the context?\u0026rdquo; are separately answerable and separately actionable. A single 1–5 quality rating collapses them and tells you nothing about where to look.\nKeep the rubric stable. Changing it resets your time series, and the trend is the point.\nSeparate retrieval failure from generation failure # This is the single most useful structural decision, and it\u0026rsquo;s easy to miss because both failures look identical to the user: a wrong answer.\nThey have completely different remedies. If the right document was never retrieved, no amount of prompt engineering will help. If the right document was retrieved and the answer still misrepresented it, retrieval work is wasted effort.\nLog the full retrieval context with every response. Then evaluate the two stages separately:\nRetrieval: was the necessary information in the candidate set? This is much closer to a classical information retrieval problem, and it\u0026rsquo;s substantially easier to evaluate — a human can answer \u0026ldquo;is the answer present in these passages?\u0026rdquo; quickly and consistently, without composing a correct response. Generation: given this context, is the response supported by it? Splitting the problem this way converts one intractable question into two tractable ones. It\u0026rsquo;s also where most of the diagnostic value lives: in my experience the majority of bad answers are retrieval failures wearing a generation costume.\nSignals that need no labels at all # Some of the most useful metrics require no ground truth whatsoever, and they\u0026rsquo;re the ones I\u0026rsquo;d instrument first because they\u0026rsquo;re nearly free:\nAbstention rate. How often does the system decline? A rising rate means retrieval is degrading or your corpus has gaps. A falling rate might mean your threshold has drifted permissive. Either direction is informative, and it\u0026rsquo;s available immediately.\nRetrieval score distribution. The distribution of relevance scores across queries shifts before answer quality visibly degrades. It\u0026rsquo;s an early warning for corpus drift, embedding changes, and indexing problems.\nQuery pattern changes. If users start asking things they never asked before, your evaluation set is aging out from under you.\nCoverage gaps. Queries that consistently trigger abstention identify documentation that doesn\u0026rsquo;t exist. That\u0026rsquo;s arguably more valuable than any retrieval improvement, and the system produces it as a byproduct.\nNone of these tell you whether an answer was correct. All of them tell you when something changed, which is when you should look.\nOn using a model to evaluate a model # It works better than it has any right to, and the caveat is real: a model evaluating model output shares the evaluating model\u0026rsquo;s blind spots. If both systems misunderstand a domain in the same way, the evaluation will confidently confirm the error.\nThe defensible use is as an amplifier, not a replacement. Calibrate the automated judge against your human-reviewed sample. Measure agreement. If the judge agrees with human reviewers most of the time on cases where humans agree with each other, you can use it to scale — while continuing to sample for human review to detect drift in the judge itself.\nTreating it as ground truth because it\u0026rsquo;s cheap is how you get a system that optimizes for what one model thinks another model should say.\nThe part nobody wants to hear # Reviewers disagree with each other. Before concluding your system has an error rate, measure inter-reviewer agreement on the same examples. It\u0026rsquo;s routinely lower than people expect.\nThis matters for two reasons. It bounds how good your evaluation can be — you cannot measure quality more precisely than your reviewers agree on it. And it reframes some apparent model errors as genuine ambiguity in the task, which is a different problem with a different fix: usually clarifying the specification rather than changing the system.\nStart smaller than feels responsible # The failure mode I\u0026rsquo;ve seen most often isn\u0026rsquo;t building a bad evaluation. It\u0026rsquo;s planning a rigorous one, discovering it\u0026rsquo;s a quarter of work, deprioritizing it, and shipping changes uncontrolled for a year.\nFifty hand-written examples with assertions, run on every change, is worth more than a comprehensive framework that doesn\u0026rsquo;t exist yet. Add the behavioral instrumentation now, because retrofitting it means a long blind period. Everything else can come later.\nThe bar isn\u0026rsquo;t a number you can put in a slide. It\u0026rsquo;s being able to answer, honestly, whether the thing you changed yesterday made the product better — and knowing which parts of that answer you should trust.\n","date":"August 1, 2026","externalUrl":null,"permalink":"/posts/evaluating-ai-without-ground-truth/","section":"Blog","summary":"Most AI features ship without a defensible way to tell whether a change made them better. Here’s how to build an evaluation approach when ground truth doesn’t exist, and how to be honest about what it can’t tell you.","title":"Evaluating an AI Feature When There's No Ground Truth","type":"posts"},{"content":"","date":"August 1, 2026","externalUrl":null,"permalink":"/tags/evaluation/","section":"Tags","summary":"","title":"Evaluation","type":"tags"},{"content":"","date":"August 1, 2026","externalUrl":null,"permalink":"/tags/llm/","section":"Tags","summary":"","title":"Llm","type":"tags"},{"content":"Most skills pages are a list of everything the author has ever touched, which tells you nothing. This one is organised by how deep I actually go, because that\u0026rsquo;s the part that matters if you\u0026rsquo;re deciding whether to talk to me.\nWhere I go deep # These are the areas I\u0026rsquo;d be comfortable being interviewed on, designing in, or being the person a team escalates to.\nCloud and distributed systems. Designing and operating services built for cloud environments rather than lifted into them — Azure primarily. Service boundaries, asynchronous and event-driven design, failure-mode analysis, and the resilience patterns that go with them: retry budgets, backpressure, circuit breaking, graceful degradation. A lot of this is knowing which failures to design for and which to accept.\nData engineering. Batch and streaming ingestion, schema design and evolution, validation and quarantine patterns, backfill and reprocessing, late-arriving and out-of-order data. Also the operational side — the seam between applications and pipelines is where the expensive incidents live.\nObservability and reliability. Instrumentation with OpenTelemetry, tracing across service boundaries, metric design and the cardinality tradeoffs that come with it, SLI and SLO definition, and alert design that optimises for actionability rather than coverage. Incident investigation and post-incident review.\nLanguages. Java, Python, and SQL. Java is where I have the deepest and longest-running experience, including the JVM\u0026rsquo;s concurrency model and the operational behaviour of long-lived services. Python and SQL are what I reach for day to day.\nWhere I work regularly # Areas I\u0026rsquo;m effective in and use often, without claiming the same depth.\nAI-enabled software. Building product features on top of language models: retrieval design, chunking and embedding strategy, grounding and citation, and the evaluation problem when there\u0026rsquo;s no labelled ground truth. The engineering around the model is the interesting part; I\u0026rsquo;m not a research engineer.\nDelivery and automation. CI/CD pipeline design in GitHub Actions, infrastructure as code, automated testing in the deployment path, feature flags and progressive rollout. Most of my interest here is in feedback loops — slow pipelines change team behaviour, not just cycle time.\nArchitecture and technical direction. Design documents, architecture decision records, technical review, and mentoring. Decomposition and consistency-model tradeoffs.\nWhere I\u0026rsquo;m functional # Enough to be useful, not enough to lead.\nFrontend. I build internal tools, dashboards, and review interfaces — HTML, CSS, JavaScript, and consuming APIs. I care about semantic markup and keyboard accessibility. I would not take a frontend-leaning role.\nIf any of this is relevant to a conversation you\u0026rsquo;re having, the case study and the blog show the thinking behind it better than a list can. Or just get in touch.\n","date":"August 1, 2026","externalUrl":null,"permalink":"/skills/","section":"Chetan Dalal","summary":"","title":"Skills","type":"page"},{"content":"","date":"August 1, 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"I started out building applications. Not architecting platforms, not designing distributed systems — just writing features, fixing bugs, and shipping things people actually used. Most of that was Java, and I spent enough years in it to develop the habits that come from operating long-lived services: watching memory behaviour, reasoning about concurrency, and reading thread dumps at unhelpful hours. That\u0026rsquo;s still the part I enjoy most, and I think it quietly shaped everything that came after. When your first instinct is \u0026ldquo;what does this do for the person using it,\u0026rdquo; you end up asking different questions later, when the systems get bigger and the answers get less obvious.\nHow I got here # The move from application work into cloud and distributed systems wasn\u0026rsquo;t a career plan. It was the natural consequence of applications getting harder to run. One service became several. Deployments stopped being a thing you did on a Friday afternoon and became something that needed to be boring and repeatable. State moved out of the process and into somewhere you had to reason about carefully. Each of those shifts was a problem I had to understand to keep doing the job I already had, and somewhere along the way, understanding them became the job.\nData followed the same path. Applications generate data, and eventually somebody wants to do something useful with it — reporting, analytics, and now models. I got pulled into data platforms because the systems I was building were producing the inputs, and the seams between \u0026ldquo;the app\u0026rdquo; and \u0026ldquo;the pipeline\u0026rdquo; were where things broke. Working on both sides of that seam taught me something I still believe: most production incidents that look like application problems are data problems wearing a costume. A schema changed. A field that was never supposed to be null was null. A batch ran twice.\nAI-enabled software is the current version of that same story. The interesting engineering problem isn\u0026rsquo;t the model — it\u0026rsquo;s everything around it. Where does the context come from, and is it fresh? How do you know the output got worse? What happens when it fails in a way that isn\u0026rsquo;t an exception, just a quietly wrong answer? Those are data and reliability questions, which is probably why the work feels familiar rather than foreign.\nWhat I\u0026rsquo;ve come to believe # A few opinions I\u0026rsquo;ve arrived at the hard way, and hold loosely enough to argue about:\nSimple systems win on a long enough timeline. Not simple to build — simple to understand six months later, at 3am, by someone who wasn\u0026rsquo;t there when it was written. Clever code is a loan taken out against your future self\u0026rsquo;s patience.\nIf it\u0026rsquo;s hard to debug, that\u0026rsquo;s a design bug. Observability isn\u0026rsquo;t something you bolt on after an incident. Whether a system can be understood while it\u0026rsquo;s running is a property you decide at design time, and it\u0026rsquo;s much cheaper to decide it early. I\u0026rsquo;d rather ship something small and observable than something ambitious and opaque.\nMaintainability is a feature with no ticket. Nobody ever files a request for \u0026ldquo;code that\u0026rsquo;s pleasant to change,\u0026rdquo; which is exactly why it gets sacrificed first. It\u0026rsquo;s also the thing that determines whether a team is still moving quickly two years in.\nDeveloper productivity is not a soft concern. Slow feedback loops don\u0026rsquo;t just waste time — they change behavior. When a test suite takes forever, people stop running it. When deploys are scary, they get batched, which makes them scarier. A lot of what looks like a discipline problem is actually a tooling problem.\nWhat I\u0026rsquo;m working through now # I\u0026rsquo;m currently spending time on evaluation — specifically, how you tell whether an AI-enabled feature is working when there\u0026rsquo;s no clean ground truth to compare against. It\u0026rsquo;s an unsatisfying problem in a way I find genuinely interesting: the honest answer usually involves proxy metrics and human review, and every option has a failure mode you have to be upfront about.\nI\u0026rsquo;m also thinking a lot about the cost side of observability. It\u0026rsquo;s easy to instrument everything and end up with a bill and a dashboard nobody reads. Deciding what not to measure turns out to be the harder skill.\nBeyond that, I read a lot and try things. Learning new technology has never felt like a chore to me — though I\u0026rsquo;ve become more skeptical with time about adopting things quickly. The good ideas tend to still be good in two years.\nAway from the keyboard # I ride a lot. Cycling is the closest thing I have to a reset button — it\u0026rsquo;s one of the few activities where the problem in front of me is entirely physical and completely solvable by pedaling.\nThe rest of my spare time tends to end up in the garage or somewhere in the house with the trim off. I like cars, and I like working on them, which are two separate hobbies that occasionally overlap. Same with DIY and home improvement projects — furniture, repairs, the ambitious weekend plan that becomes a three-weekend plan.\nI\u0026rsquo;ve stopped pretending this is unrelated to the day job. Taking something apart to understand why it stopped working is the same instinct either way, and physical projects are usefully humbling. A brake job doesn\u0026rsquo;t care how confident you feel. There\u0026rsquo;s no rolling back a cut you\u0026rsquo;ve already made, which makes you measure properly the first time — a habit I wish transferred more cleanly to production changes than it does.\nGetting in touch # I\u0026rsquo;m always up for a conversation about interesting engineering problems, speaking opportunities, or open source. The contact page has the details, or you can find me on GitHub and LinkedIn.\n","date":"July 31, 2026","externalUrl":null,"permalink":"/about/","section":"Chetan Dalal","summary":"","title":"About","type":"page"},{"content":"The best way to reach me is LinkedIn — I check it and I do reply.\nI\u0026rsquo;m glad to hear about interesting engineering problems, roles, speaking opportunities, or collaboration on open source. If you\u0026rsquo;re getting in touch about something you read here, mention which piece — it helps me pick up the thread.\nYou can also find my code on GitHub.\n","date":"July 31, 2026","externalUrl":null,"permalink":"/contact/","section":"Chetan Dalal","summary":"","title":"Contact","type":"page"},{"content":"","date":"July 31, 2026","externalUrl":null,"permalink":"/tags/document-processing/","section":"Tags","summary":"","title":"Document-Processing","type":"tags"},{"content":"","date":"July 31, 2026","externalUrl":null,"permalink":"/tags/human-in-the-loop/","section":"Tags","summary":"","title":"Human-in-the-Loop","type":"tags"},{"content":"Some notes on a class of system I\u0026rsquo;ve spent time on: platforms that automate document-driven business workflows — intake, extraction, routing, review, approval — in settings where getting it wrong is expensive enough that you can\u0026rsquo;t take the human out.\nI want to describe the shape of the problem and the decisions that mattered, because the interesting parts weren\u0026rsquo;t where I expected them to be.\nThe obvious problem, and the real one # Document workflows fail predictably. The documents are semi-structured: the same information, arranged differently by every source. Rules-based extraction handles common layouts well and then falls apart on the long tail. The usual response is more rules, and past some threshold nobody can predict what the system does with an unfamiliar document.\nThat\u0026rsquo;s the problem everyone sets out to solve. It\u0026rsquo;s not the expensive one.\nThe expensive one showed up after extraction was working: reviewers were still re-reading documents the system had already parsed correctly. Nothing distinguished a confident extraction from a lucky one, so the safe behaviour was to check everything. We\u0026rsquo;d reduced typing but not reading, and reading was where the time went.\nReframing around that changed what we built. The goal stopped being accurate extraction and became extraction annotated well enough that a reviewer can safely skip most of it.\nConfidence as a routing input # The decision the rest of the design rests on: route documents on extraction confidence rather than document type.\nExtractions above a threshold proceed automatically. Uncertain ones escalate to a human, with the specific low-confidence fields flagged rather than the whole document handed over. This turns a binary automate-or-don\u0026rsquo;t choice into a graduated one, and makes review effort track genuine ambiguity instead of staying constant.\nTwo things I underestimated:\nThe threshold has to be configurable per workflow. The tradeoff curve isn\u0026rsquo;t symmetric — raising it sends more work to humans, dropping throughput and raising quality — and the right point depends entirely on what a wrong answer costs in that specific workflow. We initially tuned it globally and were wrong in both directions at once: too conservative for low-stakes workflows, too aggressive for high-stakes ones.\nAnd confidence is only useful if you\u0026rsquo;re honest about what it measures. A model can be confidently wrong on a document format it has never seen. Confidence routing reduces review load; it doesn\u0026rsquo;t bound the error rate. Treating it as a safety guarantee rather than a prioritisation signal is the failure mode I\u0026rsquo;d warn someone about.\nThe review interface is an architectural concern # This is the part I\u0026rsquo;d argue hardest for, and the part that\u0026rsquo;s easiest to under-resource.\nWe display each extracted value adjacent to its source region in the document, pre-focus the doubtful fields, and make confirmation a single action. That sounds like UI work. It isn\u0026rsquo;t — it requires the extraction layer to emit source coordinates alongside every value, which changes its output contract and couples extraction to presentation in a way that makes both harder to evolve.\nI\u0026rsquo;d make that trade again. Without source coordinates the reviewer has to find the field in the document themselves, and review time roughly doubles — that\u0026rsquo;s from watching reviewers work before and after, not from a controlled measurement, but the difference wasn\u0026rsquo;t subtle.\nThe general point: review sits on the critical path of the slowest stage in the system. Treating it as internal tooling and staffing it accordingly is a mistake I\u0026rsquo;ve watched teams make, including mine.\nRecording disagreement # We capture corrections as structured events — field, prior value, corrected value, document region, extractor version — rather than overwriting the extracted record.\nThis costs write-path complexity and storage for data with no immediate consumer. It\u0026rsquo;s a bet that you\u0026rsquo;ll want to evaluate and improve later, and it has to be made before you have evidence justifying it, which is why it usually doesn\u0026rsquo;t get made.\nIt\u0026rsquo;s worth it because a reviewer overriding the system is the most informative event the platform produces. When reviewers systematically disagree on a class of document, that\u0026rsquo;s a defect report arriving on a channel nobody reads unless you instrument it. Versioning the extraction logic and recording which version produced each result is what makes the stream usable — without it you can\u0026rsquo;t distinguish a genuine improvement from a redistribution of errors.\nWe captured this data for a long time before doing anything useful with it. The retraining loop is still unfinished, and it has its own problem: a model trained on corrections inherits reviewer bias, and we don\u0026rsquo;t have a good handle on how much of the apparent model error is actually disagreement between reviewers.\nWhat operating it taught me # The operational question isn\u0026rsquo;t whether the system is up. It\u0026rsquo;s whether it\u0026rsquo;s still making good decisions, and those degrade quietly.\nThe most useful signal turned out to be the distribution of extraction confidence, watched over time. A shift in it is the earliest warning that inputs have changed — a new document format, an upstream source change, a model regression — and unlike accuracy, it\u0026rsquo;s available immediately, without waiting for labels.\nCorrection rate segmented by field and extractor version is the other one. Aggregate accuracy is the number everyone asks for and the one that least informs a decision; a rising correction rate on one field is specific enough to act on.\nTwo operational things I\u0026rsquo;d tell anyone building this: separate processing latency from time-spent-waiting, because one is an engineering problem and the other is a staffing problem and averaging them hides which is degrading. And alert on the age of the oldest item awaiting review rather than queue depth — extraction scales horizontally, review does not, so without bounding this the queue grows and the oldest items quietly age out of relevance.\nThe rest is unglamorous and non-negotiable: idempotent stage handlers, because documents get reprocessed after every bug and model update; workflow state in a transactional store rather than process memory, because a deploy will always land mid-flight for some document; immutable original documents with derived data referencing them; and bounded retries with a quarantine a human can inspect, so one malformed file can\u0026rsquo;t occupy a worker forever.\nWhere I landed # Human-in-the-loop is a design stance rather than a limitation. Systems built assuming eventual full automation treat the human path as a degraded mode, and you can feel it in the ergonomics. Designing for permanent collaboration produced a better system — and better automation, because the correction data was better.\nThe thing I\u0026rsquo;d tell myself at the start: the hard problem isn\u0026rsquo;t extraction accuracy. It\u0026rsquo;s giving a reviewer enough information to trust the output without re-deriving it.\n","date":"July 31, 2026","externalUrl":null,"permalink":"/projects/intelligent-workflow-platform/","section":"Projects","summary":"Automating document-heavy workflows where a wrong answer is expensive, and why the human review step turned out to be the part that determined whether the system was worth running.","title":"Intelligent Workflow Platform","type":"projects"},{"content":"One write-up, in depth, rather than a catalogue.\nI\u0026rsquo;ve written this at a level of abstraction that stays generic — no employer systems, no internal architecture, no proprietary detail — so it focuses on the problem shape, the decisions, and what I\u0026rsquo;d do differently. Most of what I\u0026rsquo;ve learned elsewhere ends up on the blog instead, where I can make an argument without implying more than I can show.\nHappy to go deeper in a conversation — get in touch. The interesting parts are usually the decisions that didn\u0026rsquo;t make the summary.\n","date":"July 31, 2026","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"","title":"Projects","type":"projects"},{"content":"","date":"July 31, 2026","externalUrl":null,"permalink":"/tags/workflow-automation/","section":"Tags","summary":"","title":"Workflow-Automation","type":"tags"},{"content":"Every alert in your system was added by someone who had a reason. That\u0026rsquo;s the whole problem.\nAlert sets only ever grow. An incident happens, and the retrospective action is \u0026ldquo;add monitoring for this.\u0026rdquo; Nobody\u0026rsquo;s retrospective action is \u0026ldquo;remove the four alerts that fired during this incident and told us nothing,\u0026rdquo; even though that would often have helped more. There\u0026rsquo;s no natural pressure in the other direction, so the set expands until responders are filtering it mentally — which is the point at which the alerting system has stopped working, regardless of how comprehensive it looks.\nAlert fatigue is a rational adaptation # It\u0026rsquo;s tempting to frame this as a discipline problem: responders should investigate every page. They shouldn\u0026rsquo;t, and they won\u0026rsquo;t.\nIf an alert has fired forty times and thirty-eight of those were noise, the correct Bayesian response to the forty-first is to deprioritize it. Responders aren\u0026rsquo;t being careless. They\u0026rsquo;re behaving optimally given the signal quality you\u0026rsquo;ve given them.\nWhich means the fix isn\u0026rsquo;t training or process. It\u0026rsquo;s improving signal quality, and the fastest route is subtraction.\nThe compounding version of this is worse. Noisy alerts slow response to real ones. Slow response means longer incidents. Longer incidents produce retrospectives. Retrospectives produce more alerts. The standard remedy makes the problem worse, which is why teams can work hard at reliability for a year and end up with more pages and slower response.\nThe three-part test # Every page should satisfy all three of these. Not two.\nActionable. There is something a human can do about it right now. If the response is \u0026ldquo;watch it,\u0026rdquo; it\u0026rsquo;s a dashboard. If the response is \u0026ldquo;file a ticket,\u0026rdquo; it\u0026rsquo;s a ticket.\nUrgent. It cannot wait until business hours. A large fraction of alerts fail here — they describe a real problem that would be better handled on Tuesday morning by someone rested.\nRequires a human. If the remediation is deterministic, automate it. An alert whose runbook is three mechanical steps is a script that hasn\u0026rsquo;t been written yet.\nApply this to an existing alert set honestly and it typically removes a substantial fraction. That\u0026rsquo;s not a sign the alerts were badly written. Most were reasonable when added, under conditions that no longer hold.\nWhat to look at first # You don\u0026rsquo;t need to reason about every alert individually. A few queries do most of the work:\nAlerts that have never fired. They\u0026rsquo;re either unnecessary or broken, and you can\u0026rsquo;t tell which. An alert that has never fired is untested code in the most critical path you have.\nAlerts that fire constantly. If something pages daily and the system is fine, the threshold is wrong or the condition isn\u0026rsquo;t meaningful. This is the category responders have already learned to ignore, so deleting it changes nothing operationally — it only makes the ignoring official.\nAlerts with no action rate. This requires tracking outcomes, which most teams don\u0026rsquo;t. It\u0026rsquo;s worth adding: for each page, did the responder do anything? Alerts with a near-zero action rate are pure cost.\nDuplicate alerts on one failure mode. A single dependency failure that generates eleven pages across four services isn\u0026rsquo;t eleven signals. Alert on the symptom, and let the causes be diagnostic detail.\nAlerts on causes rather than symptoms. High CPU is not an incident. Users failing is an incident. High CPU while users are fine is a capacity planning input.\nSymptom-based alerting, and its honest cost # The standard advice is to alert on symptoms — user-visible impact — rather than causes. I agree with it, and I want to be straight about what it costs.\nCause-based alerts can fire before users are affected. Disk filling, memory climbing, a queue growing. Symptom-based alerts, by definition, fire once impact has begun. You are trading detection lead time for a smaller, more trustworthy alert set.\nFor most systems that\u0026rsquo;s the right trade, because the lead time was theoretical anyway — buried in noise nobody was reading. But it isn\u0026rsquo;t universal. If a failure builds slowly and takes a long time to remediate, a predictive alert genuinely earns its place. Disk exhaustion is the classic example: by the time users notice, you have a much worse problem, and the fix isn\u0026rsquo;t instant.\nThe rule I\u0026rsquo;d defend is narrower than the usual formulation: page on symptoms, plus a small deliberate set of predictive alerts where lead time is genuinely actionable. Keep that second set small and justify each member individually.\nWhat you\u0026rsquo;re actually risking # Here\u0026rsquo;s the part that makes people hesitate, and they\u0026rsquo;re right to.\nSome alert you delete will eventually have been the one that would have caught something. Not hypothetically — it will happen, and there will be a retrospective, and someone will note that this was monitored until recently.\nThat\u0026rsquo;s a real cost, and the argument for accepting it is comparative, not absolute: chronic alert fatigue causes more missed and slow-handled incidents than the deleted alert would have caught. But you should make that trade deliberately, with agreement, and preferably in writing beforehand — because after an incident is a terrible time to be litigating it.\nTwo things make it defensible:\nDelete in batches, with review. A documented decision by several people is a different thing from an individual quietly removing pages.\nKeep the underlying signal. Deleting an alert doesn\u0026rsquo;t mean deleting the metric. Move it to a dashboard. The data remains available for diagnosis; what\u0026rsquo;s removed is the interruption. Almost all the value with almost none of the cost.\nReplace the removed alerts with two things # Deletion alone leaves a gap. Fill it structurally rather than by re-adding.\nError budget burn rate alerting. Instead of alerting on many possible causes, alert on the rate at which you\u0026rsquo;re consuming your reliability budget. Multi-window — a short window for acute failures, a longer one for slow erosion. This catches novel failure modes, which cause-based alerting fundamentally cannot, because you can only alert on failures you anticipated.\nRunbooks attached to what remains. Every surviving alert should link to what to check next. This is the difference between a five-minute and a fifty-minute response for anyone who isn\u0026rsquo;t the system\u0026rsquo;s author — and it\u0026rsquo;s also how on-call stops depending on the two people who can respond from memory.\nMake the alert set an object with an owner # The reason alert sets decay is that no one owns them. Individual alerts have authors; the set has nobody.\nThe practice that works is unexciting: a recurring review of what fired, what led to action, and what didn\u0026rsquo;t. Delete accordingly. It takes under an hour and it\u0026rsquo;s the highest-return recurring meeting on an on-call team\u0026rsquo;s calendar.\nTrack alert quality as a metric in its own right — fire count, action rate, false-positive rate per alert. Without measurement you\u0026rsquo;re arguing from impressions, and the person who added the alert always has a more vivid story than the person who wants it gone.\nThe reframe that makes this land # Reliability work is usually pitched as adding capability. This is subtraction, which is a harder sell, so frame it in terms responders recognize:\nWe currently page seventeen times a week. Fourteen of those require no action. Our real detection latency is not our alerting latency — it\u0026rsquo;s alerting latency plus the time it takes someone to believe the page.\nThat last number is the one that matters, and it\u0026rsquo;s the one nobody measures. It improves through deletion, not addition.\nThe counterintuitive part isn\u0026rsquo;t really counterintuitive once stated: an alerting system\u0026rsquo;s value is set by how much responders trust it, and trust is destroyed by volume. Every unnecessary page is a small withdrawal from the account you\u0026rsquo;ll need during a real incident.\n","date":"July 8, 2026","externalUrl":null,"permalink":"/posts/deleting-alerts-improved-reliability/","section":"Blog","summary":"The instinct after an incident is to add an alert. The higher-value move is usually deleting three that have never led to action — and here’s how to make that argument without losing it.","title":"Deleting Alerts Made Us More Reliable Than Adding Them Ever Did","type":"posts"},{"content":"","date":"July 8, 2026","externalUrl":null,"permalink":"/tags/observability/","section":"Tags","summary":"","title":"Observability","type":"tags"},{"content":"","date":"July 8, 2026","externalUrl":null,"permalink":"/tags/on-call/","section":"Tags","summary":"","title":"On-Call","type":"tags"},{"content":"","date":"July 8, 2026","externalUrl":null,"permalink":"/tags/operations/","section":"Tags","summary":"","title":"Operations","type":"tags"},{"content":"","date":"July 8, 2026","externalUrl":null,"permalink":"/categories/reliability/","section":"Categories","summary":"","title":"Reliability","type":"categories"},{"content":"","date":"July 8, 2026","externalUrl":null,"permalink":"/tags/reliability/","section":"Tags","summary":"","title":"Reliability","type":"tags"},{"content":"","date":"June 10, 2026","externalUrl":null,"permalink":"/tags/distributed-systems/","section":"Tags","summary":"","title":"Distributed-Systems","type":"tags"},{"content":"","date":"June 10, 2026","externalUrl":null,"permalink":"/tags/resilience/","section":"Tags","summary":"","title":"Resilience","type":"tags"},{"content":"Retries are the first resilience pattern most people add and the one most likely to cause the outage it was meant to prevent.\nThe logic seems unimpeachable. Networks are unreliable. Some failures are transient. Trying again converts a transient failure into a success, and the user never notices. All of that is true, and it\u0026rsquo;s true right up until the failure stops being transient — at which point the same mechanism becomes an amplifier pointed at a system that is already struggling.\nThe mechanics of a retry storm # Consider a service calling a dependency that starts responding slowly — not failing, just degraded. Maybe a database is doing more work than usual, maybe an instance is being replaced.\nCallers time out. Callers retry. The dependency now receives its normal traffic plus retry traffic, which makes it slower. More calls time out. Those retry too.\nIf your retry policy is three attempts, a degradation that would have caused a partial failure now produces four times the normal request volume aimed at the component least able to handle it. The dependency was struggling at 1x. It has no chance at 4x.\nWorse, this composes. If service A calls B calls C, and each retries three times, a failure at C sees up to sixteen attempts for one user request. Retry multiplication through a call chain is how a small dependency problem becomes a platform-wide outage, and it\u0026rsquo;s essentially invisible in code review because each service\u0026rsquo;s retry policy looks reasonable in isolation.\nThen the failure clears — and every client retries simultaneously, because they all backed off by the same amount and are now synchronized. The recovering system takes a thundering herd on its first breath and falls over again.\nFour things that make retries safe # Exponential backoff with jitter # Backoff alone isn\u0026rsquo;t enough, and the reason is the synchronization above. If every client waits exactly 1s, then 2s, then 4s, clients that failed together retry together forever.\nJitter — randomizing the delay — breaks the synchronization. It\u0026rsquo;s a one-line change and it\u0026rsquo;s the difference between a recovering system getting a smooth ramp and getting a wall.\nThe variant worth knowing is decorrelated jitter, where each delay is sampled from a range based on the previous delay rather than the attempt number. It spreads retries more evenly than adding randomness to a fixed schedule.\nRetry budgets # Per-request retry limits don\u0026rsquo;t bound aggregate load. Three retries per request sounds conservative, and if every request is failing, it\u0026rsquo;s still 3x traffic.\nA retry budget caps retries as a fraction of total requests — for example, retries may not exceed 10% of successful request volume. When failures are isolated, retries proceed normally. When everything is failing, the budget exhausts and retries stop, which is exactly the behavior you want: retry when the failure is likely transient, stop when it clearly isn\u0026rsquo;t.\nThis single mechanism prevents most retry-induced outages, and it\u0026rsquo;s absent from most retry implementations.\nRetry only what\u0026rsquo;s safe to retry # Not every failed request should be retried, and the decision isn\u0026rsquo;t only about idempotency.\nA 500 might mean the request was never processed, or it was processed and the response was lost. Retrying the second case duplicates the work. If the operation isn\u0026rsquo;t idempotent, that\u0026rsquo;s a correctness bug that appears only under failure conditions — the worst kind to debug.\nSome failures should never be retried at all. A 400 will be a 400 next time. Authentication failures don\u0026rsquo;t fix themselves. Retrying deterministic failures is pure waste and it consumes budget that transient failures need.\nThe practical guidance: retry on timeouts, connection failures, 429s and 503s with a retry-after honored, and 5xx only where the operation is idempotent or you have an idempotency key.\nCircuit breakers # Retries make sense when a failure is likely transient. Once a dependency is clearly unhealthy, continued retrying is both futile and harmful.\nA circuit breaker tracks failure rate and, past a threshold, stops sending traffic entirely for a period — failing fast instead. This protects the caller (threads aren\u0026rsquo;t tied up waiting on something that won\u0026rsquo;t respond) and the dependency (it gets room to recover without load).\nIt\u0026rsquo;s a genuine tradeoff, and worth stating plainly: an open breaker fails requests that might have succeeded. Tuning is difficult, and a mistuned breaker causes outages of its own — usually by opening too eagerly under a brief blip. Half-open probing, where a small number of requests are allowed through to test recovery, is what keeps it from being a manual reset.\nTimeouts are part of the retry story # Retry behavior is meaningless without correct timeouts, and timeouts are usually wrong in a specific way: a dependency timeout longer than the caller\u0026rsquo;s own timeout.\nIf your caller gives up after 2 seconds but your call to the database has a 5-second timeout, the caller has already returned an error while the database work continues, consuming a connection and CPU for a result nobody will read. Under load this is how you exhaust a connection pool while appearing idle.\nTimeout budgets have to be reasoned about across the whole call chain: each hop gets a fraction of the remaining budget, and the sum must be less than the caller\u0026rsquo;s patience. This requires knowing your call chain, which is one of the quieter arguments for distributed tracing.\nServer-side defenses # Everything above is client-side, and client-side discipline is insufficient when you have many clients — some of which you may not control.\nLoad shedding. Under overload, rejecting a fraction of requests immediately is better than accepting everything and serving all of it slowly. Fast rejection lets clients back off; uniform slowness times everyone out and generates retries.\nExplicit backpressure. A 429 with a Retry-After tells clients precisely what to do. Well-behaved clients honor it, and that\u0026rsquo;s dramatically more effective than each client guessing.\nConcurrency limits per client. Prevents one misbehaving caller from consuming shared capacity.\nThe general principle: a server should communicate its state clearly rather than degrading silently. Silent degradation is what triggers retry storms, because clients can\u0026rsquo;t distinguish \u0026ldquo;slow\u0026rdquo; from \u0026ldquo;broken\u0026rdquo; and default to trying again.\nWhen failing fast is the right answer # The instinct that every failure should be retried is worth interrogating.\nIf a dependency is down and you retry for thirty seconds, you\u0026rsquo;ve converted a fast error into a slow error. The user waits half a minute for the same failure. Meanwhile you\u0026rsquo;re holding a connection, a thread, and memory — resources your service needs to serve requests that could succeed.\nOften the better behavior is failing immediately and degrading gracefully: return partial results, serve stale cache, or disable the non-essential feature. That requires having decided in advance what\u0026rsquo;s non-essential, which is a product conversation most teams postpone until an incident forces it at the worst possible moment.\nTest it, because assumptions rot # Retry and circuit breaker configuration is written once, under assumptions about latency and failure rate that stop being true within months. Then it sits unexercised until the incident where it matters.\nFault injection is the only reliable way to know it behaves as intended. Inject latency and failures into dependencies deliberately, in a controlled way, and watch what your retry logic does. The failures found this way are consistently surprising — timeouts that were never configured, breakers that never open, retries multiplying across a chain nobody had traced end to end.\nWhat to check tomorrow # If you want a concrete starting point:\nFind your retry multiplication. Map one critical path and multiply the retry counts. If the number is above about 5, you have an amplifier. Verify jitter is present. Backoff without jitter is extremely common and trivially fixed. Check timeout ordering. Any dependency timeout longer than its caller\u0026rsquo;s is a bug. Add a retry budget if you don\u0026rsquo;t have one. It\u0026rsquo;s the highest-value single change available here. Confirm non-idempotent operations aren\u0026rsquo;t being retried blindly. None of this is exotic, and all of it is commonly missing. Retries aren\u0026rsquo;t a bad pattern — they\u0026rsquo;re an unbounded one by default, and bounding them is the entire job.\n","date":"June 10, 2026","externalUrl":null,"permalink":"/posts/retries-are-misused/","section":"Blog","summary":"Retries are the first resilience pattern everyone adds and the one most likely to cause the outage it was meant to prevent. The mechanics of retry storms, and how to bound them.","title":"Retries Are the Most Misused Resilience Pattern","type":"posts"},{"content":"","date":"May 13, 2026","externalUrl":null,"permalink":"/categories/data-platforms/","section":"Categories","summary":"","title":"Data Platforms","type":"categories"},{"content":"","date":"May 13, 2026","externalUrl":null,"permalink":"/tags/data-engineering/","section":"Tags","summary":"","title":"Data-Engineering","type":"tags"},{"content":"The incident starts looking like an application problem. A service is returning wrong values, or a feature is behaving strangely for a subset of users, or a number on a dashboard is obviously incorrect. Engineers dig into recent deployments, read the code, and find nothing wrong — because nothing is wrong with the code.\nA schema changed upstream three weeks ago. A field that was never null is null for records from one source. A batch job ran twice during a retry and double-counted. The application is behaving perfectly with respect to inputs that are wrong.\nOnce you\u0026rsquo;ve been through this a few times, the pattern becomes hard to unsee: a large share of production incidents that present as application failures are data failures wearing a costume.\nWhy data failures are more expensive than outages # An outage is loud. Error rates spike, alerts fire, someone is paged, and the clock starts. Unpleasant, but the system is working as designed — the failure is visible and the response is immediate.\nData failures have none of those properties.\nThey fail silently. No exception, no error rate, no page. The pipeline reports success because it did what it was told with what it was given.\nThey propagate before detection. By the time someone notices, downstream systems have consumed the bad data. Reports are built on it, models trained on it, decisions made from it. The remediation surface grows with every hour.\nDetection is accidental. Usually someone notices a number looks wrong. That means detection latency is measured in days or weeks, and it\u0026rsquo;s a function of who happened to be paying attention.\nThey damage trust disproportionately. After an outage, people are annoyed and move on. After discovering a report was wrong for a month, they stop believing your numbers — and that\u0026rsquo;s much harder to repair than uptime.\nThe asymmetry is worth stating directly: silently wrong data is an outage where nobody gets paged. Most organizations treat pipeline uptime as an SLO and correctness as a softer, separate conversation. Consumers don\u0026rsquo;t experience it that way.\nThe seam is where things break # Applications produce data. Pipelines consume it. Ownership usually stops at that boundary, and the boundary is where failures concentrate.\nAn application team changes a field\u0026rsquo;s meaning — reasonable, well-tested, ships cleanly. Downstream, that field feeds a transformation with an assumption baked in from two years ago. Nothing failed. The application team had no reason to know the pipeline existed. The data team had no signal the change was coming.\nThis is not a technical failure. It\u0026rsquo;s a missing contract, and it\u0026rsquo;s why data reliability work turns out to be mostly organizational once you\u0026rsquo;re past the obvious engineering.\nFour failures that account for most of it # Schema drift. A field is added, removed, renamed, or retyped. Additions are usually benign; removals and type changes rarely are. Without explicit schema validation these arrive as nulls and coercion errors rather than as failures.\nSemantic drift. Harder, because the schema doesn\u0026rsquo;t change — the meaning does. A status field gains a new value. A currency amount switches unit. A timestamp changes from local to UTC. Every structural check passes. Every number downstream is wrong.\nDuplicate processing. A job retries after a partial failure and reprocesses records already handled. Without idempotent, keyed writes, aggregates double-count. This one is particularly nasty because it\u0026rsquo;s often correct on rerun — the bug only manifests when a specific failure interleaves with a retry.\nSilent volume changes. A source that normally delivers a million records delivers eighty thousand. Nothing errors — there just isn\u0026rsquo;t much data. Every downstream aggregate is quietly wrong, and no error-based monitoring will ever fire.\nThat last one is the clearest illustration of why standard observability practice doesn\u0026rsquo;t transfer directly. There is no error to alert on. The system is healthy. The data is wrong.\nWhat actually helps # Preserve raw input immutably # Store what you received, unmodified, before any transformation. Every downstream step becomes replayable, and — critically during an incident — \u0026ldquo;was this always wrong, or did we break it?\u0026rdquo; becomes an answerable question rather than an argument.\nIt\u0026rsquo;s a recurring storage cost with no visible consumer, which is why it gets cut. Every organization that skips it rebuilds it later, usually mid-incident, without the history that would have made the incident tractable.\nValidate in tiers, not as a gate # A single pass/fail check forces a bad choice: strict enough to be unoperable, or permissive enough to be pointless.\nTwo tiers work better. Structural failures — unparseable, schema-violating — are quarantined. Semantic failures — implausible values, referential gaps — pass through flagged. Consumers can then decide what to do with flagged records rather than having the decision made for them upstream.\nQuarantine, never drop # Rejected records go to durable storage with the rejection reason attached. Dropped records are unrecoverable and, worse, invisible — you can\u0026rsquo;t investigate what you didn\u0026rsquo;t keep, and you can\u0026rsquo;t tell the difference between \u0026ldquo;no bad records\u0026rdquo; and \u0026ldquo;no validation.\u0026rdquo;\nMonitor the things that have no error # The highest-value data monitoring produces no exceptions and requires no labels:\nFreshness per dataset — time since last successful update against expected cadence. This is what consumers actually care about and what most platforms don\u0026rsquo;t expose. Volume against recent history — catches the silent-shortfall case that error monitoring structurally cannot. Rejection rate, segmented by source and rule — the single best leading indicator available. It moves before dashboards look wrong and long before anyone files a ticket. Alert on rate of change, not absolute level. Completeness against expected partitions — so missing data is detected rather than inferred from a chart that looks low. Schema drift as an explicit alertable event, not a log line nobody reads. Make reprocessing routine # You will need to reprocess. That requires idempotent, keyed writes — natural keys and upserts rather than appends — as a platform-wide invariant rather than a per-pipeline choice.\nRetrofitting this onto append-oriented pipelines approaches a rewrite. It has to be an early decision, made before you have the evidence justifying it, which is precisely why it\u0026rsquo;s usually skipped.\nKnow what\u0026rsquo;s downstream # When a dataset is wrong, the affected consumers should be queryable, not institutional knowledge. Lineage turns \u0026ldquo;we think this feeds some reports\u0026rdquo; into a list. During an incident that distinction is the difference between a contained correction and a week of discovery.\nFor application engineers # If you build services rather than pipelines, three things make you a disproportionately good upstream neighbor:\nTreat your emitted data as an API. Someone depends on its shape and meaning. Changing a field\u0026rsquo;s semantics is a breaking change even when no code fails.\nEmit events, not just state. State tells consumers what is true now; events tell them what happened. Events are replayable and auditable, and they make downstream idempotency achievable.\nKnow who consumes your data. Not exhaustively — but if you can\u0026rsquo;t name a single downstream consumer, you can\u0026rsquo;t assess the blast radius of a change, and you will eventually make one that looks entirely safe.\nThe reframe # The useful shift is treating data correctness as an availability concern rather than a quality initiative. Not because the technical work is identical — it isn\u0026rsquo;t — but because it changes how the work gets prioritized and who considers themselves responsible.\nSilently wrong data has all the impact of downtime, none of the alerting, and a longer tail. Once it\u0026rsquo;s framed as availability, the questions become familiar ones: what\u0026rsquo;s the detection latency, what\u0026rsquo;s the blast radius, who\u0026rsquo;s on call for it. Those are questions organizations already know how to answer.\n","date":"May 13, 2026","externalUrl":null,"permalink":"/posts/data-problems-in-disguise/","section":"Blog","summary":"The incident looks like an application bug for the first hour. Why data failures propagate silently, why they cost more than outages, and what treating data quality as an availability concern changes.","title":"Most Production Incidents Are Data Problems in Disguise","type":"posts"},{"content":"","date":"April 15, 2026","externalUrl":null,"permalink":"/tags/cost/","section":"Tags","summary":"","title":"Cost","type":"tags"},{"content":"","date":"April 15, 2026","externalUrl":null,"permalink":"/tags/metrics/","section":"Tags","summary":"","title":"Metrics","type":"tags"},{"content":"","date":"April 15, 2026","externalUrl":null,"permalink":"/tags/telemetry/","section":"Tags","summary":"","title":"Telemetry","type":"tags"},{"content":"Nobody sets out to spend more on telemetry than on the systems it observes. It happens gradually, through a series of individually reasonable decisions, and the mechanism is almost always the same: cardinality.\nThe pattern is familiar. An incident happens. The retrospective concludes you couldn\u0026rsquo;t diagnose it because you lacked a dimension on a metric — you could see that latency was bad but not for which customer. So you add customer_id as a label. It\u0026rsquo;s one line of code. It solves a real problem.\nThen it\u0026rsquo;s on the bill.\nWhat cardinality actually costs # A metric with no labels is one time series. Add a label with 5 values and you have 5 series. Add another with 20 and you have 100.\nThe multiplication is the part that catches people. Cardinality isn\u0026rsquo;t additive across labels, it\u0026rsquo;s multiplicative. A request duration metric labelled with endpoint (50), method (4), status_code (8), and region (3) is already 4,800 series before anyone has added anything user-specific.\nNow add customer_id with 10,000 values. That\u0026rsquo;s 48 million series from one metric.\nEach series carries storage, ingestion, indexing, and — the part people underestimate — query cost. A query touching a high-cardinality metric has to scan and aggregate across all of it. This is why dashboards get slower over months with no apparent change: nothing broke, the cardinality grew.\nThe compounding failure is that query latency degrades exactly when you need it. Incident response depends on fast ad-hoc queries. A metrics backend struggling under cardinality is slowest under load, which is when you\u0026rsquo;re querying it most.\nThe dimensions that will get you # There\u0026rsquo;s a reliable list of labels that seem useful and are cardinality catastrophes:\nUser or customer identifiers — cardinality equal to your user count, growing with your business Request or trace IDs — unbounded and unique per request, the worst possible case Full URL paths — unbounded if they contain identifiers (/orders/8a3f...) Error messages as labels — often contain interpolated values, so effectively unbounded Timestamps or dates in labels — unbounded by construction Container, pod, or instance IDs — bounded at any moment, but churning, which produces unbounded series over time That last category is the subtle one. Pod IDs look bounded — a few hundred at a time. But every deploy replaces them, so over a retention window you accumulate series for every pod that ever existed. Frequent deploys plus autoscaling produces cardinality growth that correlates with deployment velocity rather than traffic, which makes it genuinely hard to predict.\nWhere high-dimensional data belongs # The mistake is treating \u0026ldquo;I need this dimension\u0026rdquo; as equivalent to \u0026ldquo;this must be a metric label.\u0026rdquo; Metrics are one signal type with a specific cost model, and they\u0026rsquo;re the wrong home for high-cardinality data.\nTraces are built for it. A span can carry arbitrary attributes — user ID, request ID, full path, feature flags — because traces are sampled and stored as individual records rather than aggregated across a dimension space. Cardinality is nearly free here. This is the correct destination for most of what people try to put on metrics.\nLogs handle it too, with structured fields. More expensive to query at scale, but no cardinality explosion — a log line with a user ID costs one log line.\nExemplars are the bridge. An exemplar attaches a sample trace ID to a metric data point. You get a low-cardinality metric — p99 latency by endpoint — plus a direct link to a representative slow trace. This is the highest-value integration between metrics and traces and it\u0026rsquo;s consistently underused.\nThe mental model that resolves most of these decisions:\nMetrics answer \u0026ldquo;is something wrong and how much.\u0026rdquo; Traces answer \u0026ldquo;for whom and why.\u0026rdquo;\nIf a dimension is only needed to answer why, it belongs on a trace. Adding it to a metric is paying aggregate cost for detail you only need occasionally.\nDeciding what not to measure # The uncomfortable implication is that some questions should be deliberately unanswerable from metrics. That feels wrong to engineers — more data is better — but it\u0026rsquo;s the whole discipline.\nQuestions worth asking before adding a label:\nWhat decision does this dimension enable? If the answer is \u0026ldquo;we might want to slice by it someday,\u0026rdquo; don\u0026rsquo;t. Speculative dimensions are where most cardinality originates.\nIs this bounded, and will it stay bounded? status_code is bounded forever. endpoint is bounded until someone adds a path parameter. Ask what the value looks like in two years.\nCould a trace answer this? Usually yes, and usually better, since a trace carries surrounding context a metric label never will.\nWould an aggregate serve? customer_tier (3 values) instead of customer_id (10,000) answers most of the questions people actually ask, at 0.03% of the cost.\nThat last substitution is worth internalizing. The genuine need is almost never \u0026ldquo;latency for customer 47,203.\u0026rdquo; It\u0026rsquo;s \u0026ldquo;are enterprise customers seeing worse latency than free-tier.\u0026rdquo; A bucketed dimension answers it at negligible cost.\nGovernance that works # Individual discipline doesn\u0026rsquo;t hold across a large engineering organization. A few structural measures do:\nMonitor cardinality growth as a metric. Series count per metric, tracked over time, alerting on rate of growth rather than absolute level. Growth is what tells you something changed; absolutes just tell you how big you already are.\nEnforce limits at the collector. A collector tier lets you drop or aggregate high-cardinality labels centrally, as policy, without redeploying services. This is also the only mechanism that works against a label added by a team that didn\u0026rsquo;t know the rules.\nAttribute cost to teams. Cardinality is a tragedy of the commons when the bill is central. Visibility per team changes behavior faster than any guideline.\nReview at merge time. New metric labels in a diff are worth the same scrutiny as a new database index. Both are cheap to add and expensive to live with.\nSet retention by signal type. Not everything needs the same retention. Short full-fidelity retention plus longer aggregated retention captures most of the value at a fraction of the cost — accepting that some historical questions become unanswerable at detail. State that tradeoff explicitly rather than discovering it later.\nIf you\u0026rsquo;ve already got a problem # Retroactive cleanup is harder because dashboards and alerts depend on existing labels.\nStart by finding the top metrics by series count. The distribution is almost always extreme — a handful of metrics account for the large majority of series. You don\u0026rsquo;t need a broad cleanup, you need to fix three or four things.\nFor each, determine whether the dimension is actually queried. Most backends can show which series have been read. A dimension nobody queries is pure cost, and removing it breaks nothing.\nWhere a dimension is used, look for the aggregate substitution — replace the high-cardinality label with a bucketed one and move the detail to traces. This usually requires updating a small number of dashboards, which is a contained piece of work with an immediately visible payoff.\nThe framing that gets it prioritized # Cardinality work is invisible until it isn\u0026rsquo;t. It\u0026rsquo;s easiest to fund right after a budget review, and by then you\u0026rsquo;re making decisions under pressure rather than deliberately.\nThe argument that lands isn\u0026rsquo;t about cost — finance will make that argument for you eventually. It\u0026rsquo;s about incident response:\nOur metrics backend is slowest when we query it hardest, which is during incidents. Cardinality is why. Reducing it makes diagnosis faster.\nThat reframes it from a cost-cutting exercise into a reliability improvement, which is both more accurate and considerably easier to prioritize. The savings are real, but they\u0026rsquo;re the secondary benefit.\nThe cost conversation arrives whether you plan for it or not. Choosing your retention and cardinality tradeoffs deliberately is strictly better than having them imposed by someone reading a bill.\n","date":"April 15, 2026","externalUrl":null,"permalink":"/posts/observability-cardinality-cost/","section":"Blog","summary":"Every label you add to a metric multiplies its cost. The failure mode is gradual, invisible, and ends in a budget review — here’s how to decide what not to measure.","title":"The Observability Bill Nobody Warns You About: Cardinality","type":"posts"},{"content":"This site has been rebuilt on Hugo with the Blowfish theme — dark mode, search, tags, and a proper blog section.\nMore to come on cloud engineering, data platforms, distributed systems, and the parts of building software that only show up once something is running in production.\n","date":"March 25, 2026","externalUrl":null,"permalink":"/posts/hello-world/","section":"Blog","summary":"","title":"Hello, world","type":"posts"},{"content":"","date":"March 25, 2026","externalUrl":null,"permalink":"/tags/meta/","section":"Tags","summary":"","title":"Meta","type":"tags"},{"content":"","date":"March 25, 2026","externalUrl":null,"permalink":"/categories/personal/","section":"Categories","summary":"","title":"Personal","type":"categories"},{"content":"","externalUrl":null,"permalink":"/posts/","section":"Blog","summary":"","title":"Blog","type":"posts"},{"content":"I\u0026rsquo;m a senior software engineer who builds cloud applications and the data systems behind them. My work sits where large-scale data processing meets production software: pipelines that feed real product features, services that stay observable under load, and the tooling that keeps engineering teams moving quickly. I care as much about how a system behaves at 3am as how it looks in a design review — and I write here about what holds up in practice.\nWhat I Work On # Cloud Engineering # Designing and operating cloud-native applications that are scalable, resilient, and maintainable.\nData Platforms # Building systems that collect, transform, process, and deliver data at scale.\nAI-Enabled Software # Incorporating modern AI capabilities into real-world applications and workflows.\nObservability \u0026amp; Reliability # Improving visibility into distributed systems through telemetry, monitoring, and operational excellence.\nEngineering Productivity # Helping teams deliver software more efficiently through automation, tooling, and streamlined processes.\nSelected Work # Intelligent Workflow Platform — document processing with AI assistance, and why the human review step turned out to determine whether the system was worth running. More of my thinking ends up on the blog →\nLet\u0026rsquo;s Talk # I\u0026rsquo;m always glad to hear about interesting engineering problems — whether that\u0026rsquo;s a role, a speaking opportunity, or collaboration on open source.\nGet in touch → · LinkedIn · GitHub\n","externalUrl":null,"permalink":"/","section":"Chetan Dalal","summary":"","title":"Chetan Dalal","type":"page"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]