Production alerts to support tickets in real time, exactly once
Production alerts used to reach the support desk through a spreadsheet that was polled. I replaced that with a signed webhook that creates the ticket directly, on the right account, and only once.
How it works
Each request is signed and checked before anything happens. Each alert is claimed in a table that acts as a lock, so if the same event arrives twice the second copy updates the ticket instead of opening another. The account is matched exactly, and a rule-based classifier with tests sorts the ticket.
Decisions
- Check the helpdesk API against live data before designing, then write a detailed spec and review the implementation against it.
- A rule-based classifier rather than a language model: predictable, testable and free per ticket.
Before and after
| Before | After | |
|---|---|---|
| Latency | Polling | Real time |
| Duplicate tickets on retry | Possible | Prevented |
This describes internal work at qlub. Names, internal rules and business data are left out on purpose. If you want the detail behind any of it, email mikaeel2013@gmail.com and I will walk you through it.