Data pipelines without trust

The pipeline runs. The dashboard loads. The numbers look plausible. Then someone asks where a figure came from, and nobody in the room can explain it.

That is the point the dashboard stops being used.

The problem is not the technology. The problem is that the pipeline was built to move data, not to prove it. Every transformation — the join, the filter, the aggregation — makes an assumption about what the source data means. If those assumptions are not written down and tested, the output is a number without a derivation.

We see this most often in executive dashboards built by copying SQL from the analyst who knew the schema. The analyst leaves. The schema changes. The query keeps running, but now it silently excludes a category of transactions that used to be NULL and are now tagged “pending”. The trend line drops. Nobody notices for three months, because the dashboard is downstream of five other pipelines and the break could be anywhere.

The fix is not better visualisation. The fix is reconciliation: a step in the pipeline that compares the output to a known control total and fails loudly if they do not match. Reconciliation does not prevent errors, but it surfaces them before the board meeting.

Most data projects we inherit do not have reconciliation. The first sprint is spent adding it, which usually means fixing the pipelines that turn out to be wrong.

When the finance director asks where a number came from, the answer should be a query and a reconciliation log, not “I’ll check and get back to you”.

---

Sample post — this reflects Sageware’s current positioning on data platform delivery, but is marked as illustrative content.