A schema change silently turns missing amounts into zero

Illustrative lab: a producer stops sending amount on one event. This is a learning case, not a reported customer incident.

Observable symptoms

Revenue drops while the pipeline still reports success.

Likely cause

The transformation uses a fallback value that makes a missing required field indistinguishable from a genuine zero.

Diagnosis steps

1. Compare raw input with the curated record.
2. Count missing and non-numeric amounts separately.
3. Check when producer and consumer versions changed.

Example

const rows = [{id:1, amount:0}, {id:2}, {id:3, amount:25}];
const valid = rows.filter(r => typeof r.amount === 'number' && Number.isFinite(r.amount));
const rejected = rows.filter(r => !valid.includes(r));
console.log(valid.map(r => r.id).join(','));
console.log(rejected.map(r => r.id).join(','));

Expected result

1,3
2

Code walkthrough

• The fixture includes a genuine zero, a missing field and a positive amount.
• The predicate accepts only finite numeric values.
• The output separates valid ids 1 and 3 from rejected id 2.

Solution and boundaries

Validate required fields before aggregation and quarantine invalid records with safe error context. Keep genuine zero values valid. Establish a schema contract and an alert threshold for rejected records.

Prevention and monitoring

Monitor missing-field rates and reconcile business totals; do not silently coerce required values.

Interview question

Why is amount || 0 not a validation policy?

Official reference · Related technical guide