A log is not an explanation

Every team running something automated in production reaches the same moment. Something goes wrong in a way that is not a crash, not an exception, and not a failed request. Every step returned 200, every job finished, and the outcome was still wrong. The logs are full and they explain nothing.

This is the point at which conventional observability stops helping. Traces and metrics were built for systems whose behaviour is determined by their code. Behaviour here is determined by the code, the configuration, the inputs, the state of five upstream systems at one particular second, and a decision made from all of it. Most of that is invisible to a normal tracing stack, because most of it was never an argument to a function.

The three questions a trace has to answer

When a job goes wrong, an engineer is trying to answer three questions in order, and they are not the questions a service trace was built for.

• 1.What did the system see when it acted? Not what was true in the database now — what the inputs actually were at the moment of the decision.
• 2.Which step was the first one that was wrong? Failures propagate. By the time the output is visibly broken, five reasonable-looking steps have been built on one bad premise.
• 3.Would it do the same thing again? A job that fails identically every time is a bug. A job that fails one time in twenty is a different problem with a different fix.

A stack trace answers none of these. It tells you where the process was, not what the process was working from.

Record the inputs, not just the outputs

The single highest-leverage change most teams can make is to capture the fully resolved input of every step — after defaults are applied, after config is merged, after every upstream lookup has returned — and store it next to the result. It is more data than feels comfortable. It is also the only artefact that makes a failed job reproducible.

The objection is always storage cost, and it is a real cost. It is also much smaller than the alternative, which is an engineer spending two days trying to reproduce a failure that took the system four seconds to produce.

Instrument the seams, not the steps

Things fail at seams: between a lookup and the logic that uses it, between a plan and its execution, between a result and the interpretation of that result. A span that covers 'chose this branch, for this reason, from these candidates' is worth ten spans covering the individual HTTP requests underneath it.

await vantis.span("select-target", {
  candidates: options.map(o => o.name),
  chose: decision.target,
  score: decision.score,
  inputs: resolvedInput,        // the whole thing
});

That one span is what turns 'it did something strange' into 'it scored the wrong candidate 0.61 against 0.59, here is the input that produced those numbers'. The first is a conversation. The second is a fix.

What we would tell ourselves eighteen months ago

• —Capture the resolved input on every step from day one. Retrofitting it means losing every failure that happened before you started.
• —Version configuration like code, and put the version in the trace. A config change is a deploy.
• —Sample jobs that succeed as well as jobs that fail. You cannot tell whether a failure is unusual without a picture of what usual looks like.
• —Record which version of every upstream dependency was involved. 'Which one was this?' should never be an open question.

None of this is exotic. It is the same discipline that made distributed systems debuggable, applied one layer up, to the decisions rather than the calls.