Cutting Payment Turnaround From 30 Minutes to 2

When a payment takes thirty minutes, the instinct is to blame the rail. Usually that is wrong. The rail responds in seconds; the thirty minutes is everything your own system does around it.

Find out where the time actually goes

Before changing anything, instrument each stage: request accepted, validation complete, rail called, response received, state persisted, downstream notified. The distribution is more informative than the average — a p50 of two minutes with a p95 of forty means you have a queueing problem, not a latency problem.

In our case the rail call was a small fraction of wall-clock time. The rest was work happening synchronously that had no business being in the request path.

Decouple, then make retries safe

Moving processing onto a durable queue is the obvious half of the fix. The half that determines whether it works is idempotency.

Once work is queued, it will occasionally run twice — a worker dies mid-task, a broker redelivers, someone replays a batch after an incident. If executing twice can move money twice, you have traded latency for a much worse problem. Every operation needs a stable idempotency key and a check that makes the second execution a no-op.

• Accept and validate fast; hand everything else to a queue
• Give each operation a deterministic idempotency key
• Make the second execution provably a no-op, not merely unlikely
• Track state explicitly at every step so a stuck payment is visible

Batching is where the last minutes hide

After decoupling, the remaining latency was batch boundaries: work that was correct but waited for a scheduled run. Shrinking those windows — and letting some paths bypass batching entirely — took the majority of payments to roughly two minutes end to end.

The word 'majority' matters. Tail cases involving manual review or rail-side delays still take longer, and reporting a single headline number without that caveat would be dishonest.