Timeline (UTC)

• 13:40 We are alerted of a critical failure in one of the database clusters; Investigation begins immediately. At this point the web application is crashing with an error message.
• 13:47 After a few minutes of investigation and attempts to restore the cluster’s ability to server requests, we trigger failover process, and disable alerts to prevent false positives. A backlog of unprocessed metric payloads has built up in the intervening minutes.
• 13:51 We create a public incident here and update it every 10-15 minutes or so until the incident is monitored.
• 14:06 The failover process completes, the web is back and additional processors are started to clear the backlog. Incoming metric payloads are processed in last-in-first-out mode to restore recent data in dashboards.
• 14:28 A configuration error resulting from the failover process causes dashboards to fail to load intermittently. That error is fixed.
• 14:38 Extra processors continue to process the backlog, exhibiting in some cases, missing data shortly after the incident began. Recent data are up-to-date.
• 15:51 All systems are now operational; alerting is enabled again.
• 16:52 The status page is updated and the incident resolved.

What happened?

We use a Postgres cluster to store user data (e.g. dashboard and monitor definitions). The cluster is composed on a master/standby pair with a number of read replicas used by most of our internal services. Read replicas are used to isolate the application from master failures, which obviously did not work as planned in this instance: an improper configuration of the bootstrap code caused most services to use connections to the master node instead of connections to one of the replicas.

Upon failure of the master, we trigger a scripted fail-over to the standby instance per our procedures. The decision to fail-over is still left to the incident responder, following a brief investigation. In this case, the fail-over was triggered by mistake on one of the replicas instead of the standby node, delaying the completion of the fail-over.

For historical reasons, this cluster also contains some small but critical data (metric names and units), required to process metric payloads. The same information is distributed to all replicas but was unused given the improper configuration. Thus the failure of the master node affected a large portion of the metric processors, leading to a backlog and to the suspending of alerts to avoid false positives.

This has been a very visible and humbling incident that highlights a number of immediate actions to avoid the same from happening again. We are sorry for the inconvenience and angst this has caused you.

How will we prevent this in the future?

1. A thorough configuration review to ensure that no extraneous connection to this cluster’s master is established, with safeguards to avoid future regressions.
2. A review of our fail-over procedures review to catch triggering on the wrong target.
3. A change in metrics processor to remove the dependency on this cluster altogether.