We have finished backfilling data across all products: all data received during the incident that had been successfully buffered but unprocessed, is now fully accessible on the platform. Due to the nature of this outage, you may see some residual gaps in the data we received within the first few hours after the start of the incident.
We truly appreciate your patience and understanding during this incident.
Posted Mar 10, 2023 - 00:25 EST
Update
We have completed backfill of data for the following products
* Real User Monitoring * Database Monitoring * Network Performance Monitoring * Network Device Monitoring
We are now in the process of validating and verifying data across all customers in those products.
For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 21:10 EST
Update
We have also completed backfilling data for the following products:
Log Management
We are now in the process of validating and verifying data across all customers in those products.
For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 18:04 EST
Update
We have completed backfill of data for APM traces and services, and CI Visibility and are now in the process of validating and verifying data across all customers in those products. For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 15:15 EST
Update
We've finished monitoring the recovery. All Datadog systems continue to receive, query, and evaluate monitors on live data as normal, while backfill operations continue for historical data.
Posted Mar 09, 2023 - 12:11 EST
Update
During the recovery operations, users might have experienced temporary elevated latency and error rates on the web application between 15:30 and 15:46 UTC , specifically for metric queries and APM. We are monitoring recovery now and we are continuing with the backfilling operations.
Posted Mar 09, 2023 - 10:55 EST
Update
All Datadog services are now available and able to receive, query, and report on live data. Monitors continue to be evaluated correctly since live data has been restored. Some customers may still observe gaps in historical data for parts of the last 24 hours.
We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 08:00 EST
Update
Network Performance Management is operational again. Logs Management are generally working, but some users may see transient errors when querying recent data. Monitors continue to be evaluated correctly since live data has been restored.
Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 07:26 EST
Update
Logs Management and Network Performance Management are generally working, but some users may see transient errors when querying recent data. Monitors continue to be evaluated correctly since live data has been restored.
Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 07:08 EST
Update
We are continuing to monitor for any further issues.
Posted Mar 09, 2023 - 05:09 EST
Update
Logs Management and Network Performance Management are generally working, but some users may see transient errors when querying recent data. Monitors continue to be evaluated correctly since live data has been restored.
Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 04:48 EST
Monitoring
Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 03:58 EST
Update
A subset of customers might experience transient errors while loading Network Performance Monitoring and Logs Management. The underlying data is still being processed and will be available query once queries are fully operational again. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 03:27 EST
Update
SLOs are operational. CI Visibility is operational. Profiling recent data is available for queries. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 02:04 EST
Update
Logs Management is operational, live data and alerting are back to normal. External Archives and Log Forwarding are still delayed. Serverless monitoring is operational. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 01:31 EST
Update
We are continuing to work on a fix for this issue.
Posted Mar 09, 2023 - 01:26 EST
Update
APM Traces is fully operational. RUM is fully operational. Security Monitoring is fully operational. Database Monitoring is fully operational. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 01:03 EST
Update
Serverless monitoring is fully operational. Synthetic Monitoring is fully operational. Network Device Monitoring is fully operational. Database Monitoring is fully operational. APM Services is fully operational. Metrics from our cloud provider integrations are available, and Metrics generated from Logs are available. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 00:19 EST
Update
Live data for Metrics is now available for all customers. We're in the process of enabling metric alerts for some customers for time windows less than 1 hour.
We're seeing partial recovery for Network Performance Monitoring. Error Tracking is seeing partial availability, and we're investigating. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 08, 2023 - 23:28 EST
Update
We're seeing partial recovery for the Profiling product, as well as metrics from our cloud provider integrations. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 08, 2023 - 22:17 EST
Update
Live data for Metrics is now available for most customers. Historical search for APM Traces is operational. Monitors for Logs and Service Checks are operational. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 08, 2023 - 21:15 EST
Update
Live data is now available for Logs. We're seeing partial recovery for Security Monitoring. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 19:54 EST
Update
Live Search on last 15 mins for APM Traces, and Live Processes is recovered. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 18:49 EST
Update
Incident Management is fully operational. We're seeing partial recovery across several products including Serverless and Network Performance Monitoring. These products may have gaps in data and partial limitations based on data available to monitors. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 18:21 EST
Update
Logs have live data available on US1 for about 33% of customers. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 18:04 EST
Update
We're seeing partial recovery across several products including SLOs, Profiling, WatchDog, Logs. These products may have gaps in data and partial limitations based on data available to monitors. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 17:49 EST
Update
Database Monitoring is operational in US1. There may be gaps in historical data. We continue progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 17:06 EST
Update
Processes Monitors are operational in US1. There may be gaps in historical metric data. We continue progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 16:21 EST
Update
At 06:00 UTC on March 8th, 2023 the Datadog platform started experiencing widespread issues across multiple products and regions . The web application was unavailable or intermittently loading, and data ingestion & monitor evaluation were delayed.
We will share a more detailed analysis post-recovery, but at a very high level: A system update on a number of hosts controlling our compute clusters caused a subset of these hosts to lose network connectivity As a result a number of the corresponding clusters entered unhealthy states and caused failures in a number of the internal services, datastores and applications hosted on these clusters.
Our current status is: We identified and mitigated the initial issue, and rebuilt our clusters We also have recovered a number of our applications and services, including our web portals We are now working on recovering and catching-up the rest of our data systems for metrics, traces and logs across the regions that are still affected (see region-specific status pages). The recovery work is currently constrained by the number and large scale of the systems involved.
What to expect next: We are focusing on bringing back live data for all customers and all products before catching-up on any historical data we may have stored during the outage We expect live data recovery in a matter of hours (not minutes, and not days) We will continue to issue regular updates as the situation unfolds
We understand how critical Datadog is to your business, we sincerely apologize for the inconvenience and we are working hard to resolve this issue.
Posted Mar 08, 2023 - 15:39 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 15:12 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 14:26 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 13:42 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 13:14 EST
Update
We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 12:29 EST
Update
We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 11:46 EST
Update
We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 11:04 EST
Update
We are still progressing towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 10:34 EST
Update
We are still progressing towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 09:49 EST
Update
Some products are recovering and we are still progressing towards a complete recovery. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 09:07 EST
Update
Some products are recovering and we are still progressing towards a complete recovery. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 08:30 EST
Update
We are still working on the identified issue and are making continued progress towards recovery. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 07:45 EST
Update
We have identified the issue, and are making continued progress towards recovery. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 07:08 EST
Identified
We are seeing reduced error rates for the web application. We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and you may observe delayed data throughout the app.
Posted Mar 08, 2023 - 06:20 EST
Update
We are seeing reduced error rates for the web application. We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and you may observe delayed data throughout the app.
Posted Mar 08, 2023 - 05:31 EST
Update
We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and you may observe delayed data throughout the app. Additionally, the web application continues to have elevated error rates.
Posted Mar 08, 2023 - 04:39 EST
Update
We are continuing to investigate this issue.
Posted Mar 08, 2023 - 03:52 EST
Update
We are still investigating issues causing delayed data ingestion across all data types. Monitor notifications may be delayed, and you may observe delayed data throughout the web app.
Posted Mar 08, 2023 - 03:40 EST
Update
We are still investigating issues causing delayed data ingestion across all data types. Monitor notifications may be delayed, and you may observe delayed data throughout the web app.
Posted Mar 08, 2023 - 03:05 EST
Update
We are investigating issues causing delayed data ingestion across all data types. As a result monitor notifications may be delayed, and you may observe delayed data throughout the web app.
Posted Mar 08, 2023 - 02:23 EST
Investigating
We are investigating loading issues on our web application. As a result, some users might be getting errors or increased latency when loading the web application.
Posted Mar 08, 2023 - 01:31 EST
This incident affected: APM, CI Visibility, Cloud Security Management, Database Monitoring, Incident Management, Log Management, Metrics and Infra Monitoring, Mobile Application, Monitors, NPM, Profiling, RUM, Serverless, Synthetics, and Web Application.