A latent bug in the notification pipeline caused certain classes of notifications (Pagerduty, Opsgenie and Flowdock) to be skipped or delayed at the beginning of the incident. The normal sets of pre- and post-flight checks that precede and follow deploys failed to catch the issue immediately. Our monitoring of notification traffic had quality issues, leading to a much longer-than-usual time-to-detection. As soon as we identified the issue we replayed the most recent notifications. Incident notifications are an essential part of monitoring; we are sorry for the inconvenience caused by this incident.
We attribute the cause of this incident to the conflation of several issues: One of the updates published that day contained a circular import, triggered in certain conditions, that were not triggered during our automated tests and pre-flight checks. The error required a rolling restart of a number of processes. It happened too long after the updated code was rolled out and gave us a false sense of confidence about the quality of this particular update. Our notification monitors surveyed traffic only at an aggregate level instead of that and also by class of notifications.
We have revamped and improved the quality of our notification monitoring: more granular notification monitors. We’ve also updating the deploy logic to ensure a timely rolling restart of the particular processes that failed this time. Lastly, during post-incident discussions with some of you, you asked us about the decision tree to discard/replay outdated notifications. During the incident we came up with a cut-off point prior to which we did not replay outdated notifications. In hindsight, given the duration of the incident, the better decision may have been to replay all. We are revisiting this (extending the cut-off to 4h) to have clearer rules to follow internally during an incident.