Question: Tell me about a time you handled a production incident.
- Situation
- At a payments company of about sixty people I was the on-call backend engineer. On a Friday evening our checkout service started returning 500s on roughly one request in five, and the error rate was climbing.
- Task
- I owned the incident until it was resolved or handed over. The immediate goal was stopping the customer-facing failures; the root cause could wait.
- Action
- I declared an incident and posted in the shared channel so support stopped guessing at what customers were seeing. The dashboards showed the failures concentrated on one database replica, so I pulled that replica out of the read pool before I understood why — the error rate dropped to near zero within about four minutes. Then I looked properly. A schema migration that afternoon had left that replica without an index, so a query taking 8 ms elsewhere took several seconds there and exhausted the connection pool. I chose to rebuild the index rather than roll the migration back, because a rollback would have re-broken the feature the release had just shipped, and I wrote that reasoning into the channel so the decision was reviewable rather than mine alone. I drafted the postmortem the same evening while the timeline was still exact.
- Result
- Customer-facing errors lasted about eleven minutes and no payments were lost. The postmortem produced two changes I pushed through the following week: migrations now verify index presence on every replica before they are marked complete, and the read pool ejects a replica automatically when its latency crosses a threshold. What I took away is that mitigating first and diagnosing second is almost always right, and I have had to argue that with engineers who want the cause before they will touch anything.


