A system sends two requests to a partner to credit loyalty points. Both end with connection reset by peer while the sender waits for a response. When the teams compare transaction references, the partner confirms that transaction A credited the points. The first search finds no record of transaction B.

Retry both and you may credit the points twice for A. Close both cases and B may go unprocessed. These two fictional transactions begin with the same error.

A lost response leaves the outcome open

Consider one request in three steps.

  1. The sender asks the partner to credit points.
  2. The partner records the credit.
  3. The sender receives the response.

If the connection drops between steps 2 and 3, the sender may log an error even though the balance changed. The sender can see the same error if the connection drops before the partner records anything. Track the missing response separately from the business outcome.

While you investigate, mark the transaction as pending confirmation. If your system retries automatically, check whether a retry is already queued or running for that transaction before starting a manual one.

Follow the logs to the transaction that changed the balance

Start with a transaction reference both sides can look up. Keep the timestamp with its time zone, the operation type, and the number of points handy for cross-checking. An attempt ID helps when reading logs. One transaction can have several attempts, and one attempt can appear in several places, such as the gateway and the application. If you see five error entries, group them by transaction before counting the impact. Keep any entries you cannot match for further review.

On the sender side, check the latest state, queued work, retries, and any callback that arrived later. On the partner side, ask for evidence that confirms whether the operation took effect, such as a status endpoint that reports the final outcome or an entry in the points ledger. If the partner says “no record found,” ask which source they checked, whether it covered the correct reference and time window, and whether the original request could still be pending. A search with incomplete coverage leaves the outcome open.

Decide transaction by transaction

Retry decision flow. Check both systems using the same transaction reference. Reconcile confirmed successes, investigate pending outcomes, and retry only once the original effect is ruled out and the retry conditions are met.

What you can confirm What to do next
The partner confirms success and the transaction details match Reconcile the sender’s records with the partner’s result, then close the retry task.
The outcome is still uncertain, or the original request may still be pending Keep it pending confirmation. Check a more complete source and assign an owner and a time for the next check.
You have confirmed there was no prior effect, resolved any pending attempts, and verified that the API supports the retry Retry as agreed, link the new attempt to the original transaction, and verify the outcome in both systems.

Transaction A goes in the first row once the reference and amount match. Transaction B stays in the middle row after the first search. Suppose the partner later checks the points ledger and queue, the sender confirms there is no competing retry, and both sides agree on how to resend the request. B can then move to the last row. After retrying, verify the final outcome, especially when the API acknowledges requests before processing finishes.

Check how the API prevents duplicate effects

A transaction reference lets both teams find the same business operation. An idempotency key lets an API that supports it recognize repeated requests so the intended effect is the same as sending the request once. If the API calls for reusing the original key, check how long the guarantee applies to that endpoint, whether the payload must match, and how the API handles a repeat while the first attempt is still running. Once the key is outside the retention window, check the transaction’s state again before deciding.

HTTP semantics advise clients against automatically retrying a non-idempotent request unless they know the operation is idempotent in practice or can determine that the original request was never applied. Follow the contract of the API you use. Stripe API v1, for example, documents how long keys may be retained and how it compares parameters when a key is reused. RFC 9110 §9.2.2 · Stripe API — Idempotent requests

Before resending, leave a short decision in the incident record. For example, “For B, we checked the points ledger and queues, confirmed nothing is pending and no points were credited, and will retry with the original key as agreed. We’ll verify the balance after processing.” The next person on the incident can see what was checked and what evidence will let them close the case.

← All notes