A replayed batch inserts duplicate business records

Illustrative in-memory lab: the same two events are processed twice after a worker restart. This is a learning case, not a reported customer incident.

Observable symptoms

Target row counts or totals increase on a retry even though the source events did not change.

Likely cause

The sink appends each delivery without a stable event identity or an idempotent update policy.

Diagnosis steps

1. Compare business keys and event identifiers across the original run and retry.
2. Determine whether the first write committed before the timeout.
3. Check whether event identifiers are stable across redelivery.

Example

const sink = new Map();
const batch = [{id:'e1', amount:10}, {id:'e2', amount:20}];
function apply(events) { for (const event of events) sink.set(event.id, event); }
apply(batch);
apply(batch);
console.log(sink.size);
console.log([...sink.values()].reduce((sum, row) => sum + row.amount, 0));

Expected result

2
30

Code walkthrough

• The Map keys each record by event id.
• apply replaces a record with the same id instead of appending another copy.
• Running the batch twice leaves two distinct events and a total of 30.

Solution and boundaries

Use stable event identities and a durable sink contract that prevents duplicate effects. This Map demonstrates the desired invariant only: production storage needs transactional uniqueness/upsert semantics and durable state.

Prevention and monitoring

Add a replay test and an uncertain-write recovery test to every incremental load.

Interview question

Why is an in-memory deduplication set insufficient after a process restart?

Official reference · Related technical guide