Datadog US1
All Systems Operational

About This Site

The status of Datadog https://app.datadoghq.com

If you would like to see the status of third party integrations you might have enabled with Datadog check out https://datadogintegrations.statuspage.io

If you are a customer running in another region:
- our EU region https://app.datadoghq.eu check out https://status.datadoghq.eu
- our US3 region https://app.us3.datadoghq.com check out https://status.us3.datadoghq.com
- our US5 region https://app.us5.datadoghq.com check out https://status.us5.datadoghq.com
- our GovCloud region https://app.ddog-gov.com check out https://status.ddog-gov.com

Agent Repository Operational
APM Services Operational
APM Traces Operational
CI Visibility Operational
Cloud Integrations Operational
Corporate Site (www.datadoghq.com) Operational
Database Monitoring Operational
Error Tracking Operational
Events Operational
Incident Management Operational
Log Management Operational
Metrics Operational
Mobile Application Operational
Monitors Operational
Network Performance Monitoring Operational
Processes Operational
Profiling Operational
RUM Operational
Security Monitoring Operational
Serverless Operational
SLOs Operational
Synthetics Operational
Watchdog Operational
Web Application Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Mar 24, 2023

No incidents reported today.

Mar 23, 2023

No incidents reported.

Mar 22, 2023

No incidents reported.

Mar 21, 2023

No incidents reported.

Mar 20, 2023

No incidents reported.

Mar 19, 2023

No incidents reported.

Mar 18, 2023

No incidents reported.

Mar 17, 2023

No incidents reported.

Mar 16, 2023

No incidents reported.

Mar 15, 2023

No incidents reported.

Mar 14, 2023

No incidents reported.

Mar 13, 2023

No incidents reported.

Mar 12, 2023

No incidents reported.

Mar 11, 2023

No incidents reported.

Mar 10, 2023
Resolved - We have finished backfilling data across all products: all data received during the incident that had been successfully buffered but unprocessed, is now fully accessible on the platform. Due to the nature of this outage, you may see some residual gaps in the data we received within the first few hours after the start of the incident.

We truly appreciate your patience and understanding during this incident.

Mar 10, 00:25 EST
Update - We have completed backfill of data for the following products

* Real User Monitoring
* Database Monitoring
* Network Performance Monitoring
* Network Device Monitoring

We are now in the process of validating and verifying data across all customers in those products.

For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.

Mar 9, 21:10 EST
Update - We have also completed backfilling data for the following products:

Log Management

We are now in the process of validating and verifying data across all customers in those products.

For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.

Mar 9, 18:04 EST
Update - We have completed backfill of data for APM traces and services, and CI Visibility and are now in the process of validating and verifying data across all customers in those products.
For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.

Mar 9, 15:15 EST
Update - We've finished monitoring the recovery. All Datadog systems continue to receive, query, and evaluate monitors on live data as normal, while backfill operations continue for historical data.
Mar 9, 12:11 EST
Update - During the recovery operations, users might have experienced temporary elevated latency and error rates on the web application between 15:30 and 15:46 UTC , specifically for metric queries and APM. We are monitoring recovery now and we are continuing with the backfilling operations.
Mar 9, 10:55 EST
Update - All Datadog services are now available and able to receive, query, and report on live data. Monitors continue to be evaluated correctly since live data has been restored. Some customers may still observe gaps in historical data for parts of the last 24 hours.

We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.

Mar 9, 08:00 EST
Update - Network Performance Management is operational again.
Logs Management are generally working, but some users may see transient errors when querying recent data.
Monitors continue to be evaluated correctly since live data has been restored.

Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.

Mar 9, 07:26 EST
Update - Logs Management and Network Performance Management are generally working, but some users may see transient errors when querying recent data.
Monitors continue to be evaluated correctly since live data has been restored.

Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.

Mar 9, 07:08 EST
Update - We are continuing to monitor for any further issues.
Mar 9, 05:09 EST
Update - Logs Management and Network Performance Management are generally working, but some users may see transient errors when querying recent data.
Monitors continue to be evaluated correctly since live data has been restored.

Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.

Mar 9, 04:48 EST
Monitoring - Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Mar 9, 03:58 EST
Update - A subset of customers might experience transient errors while loading Network Performance Monitoring and Logs Management. The underlying data is still being processed and will be available query once queries are fully operational again. We will continue to monitor progress towards recovering the remaining services.
Mar 9, 03:27 EST
Update - SLOs are operational. CI Visibility is operational. Profiling recent data is available for queries. We will continue to monitor progress towards recovering the remaining services.
Mar 9, 02:04 EST
Update - Logs Management is operational, live data and alerting are back to normal. External Archives and Log Forwarding are still delayed. Serverless monitoring is operational. We will continue to monitor progress towards recovering the remaining services.
Mar 9, 01:31 EST
Update - We are continuing to work on a fix for this issue.
Mar 9, 01:26 EST
Update - APM Traces is fully operational. RUM is fully operational. Security Monitoring is fully operational. Database Monitoring is fully operational. We will continue to monitor progress towards recovering the remaining services.
Mar 9, 01:03 EST
Update - Serverless monitoring is fully operational. Synthetic Monitoring is fully operational. Network Device Monitoring is fully operational. Database Monitoring is fully operational. APM Services is fully operational. Metrics from our cloud provider integrations are available, and Metrics generated from Logs are available. We will continue to monitor progress towards recovering the remaining services.
Mar 9, 00:19 EST
Update - Live data for Metrics is now available for all customers. We're in the process of enabling metric alerts for some customers for time windows less than 1 hour.

We're seeing partial recovery for Network Performance Monitoring. Error Tracking is seeing partial availability, and we're investigating. We will continue to monitor progress towards recovering the remaining services.

Mar 8, 23:28 EST
Update - We're seeing partial recovery for the Profiling product, as well as metrics from our cloud provider integrations. We will continue to monitor progress towards recovering the remaining services.
Mar 8, 22:17 EST
Update - Live data for Metrics is now available for most customers. Historical search for APM Traces is operational. Monitors for Logs and Service Checks are operational. We will continue to monitor progress towards recovering the remaining services.
Mar 8, 21:15 EST
Update - Live data is now available for Logs. We're seeing partial recovery for Security Monitoring. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Mar 8, 19:54 EST
Update - Live Search on last 15 mins for APM Traces, and Live Processes is recovered. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Mar 8, 18:49 EST
Update - Incident Management is fully operational. We're seeing partial recovery across several products including Serverless and Network Performance Monitoring. These products may have gaps in data and partial limitations based on data available to monitors. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Mar 8, 18:21 EST
Update - Logs have live data available on US1 for about 33% of customers. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Mar 8, 18:04 EST
Update - We're seeing partial recovery across several products including SLOs, Profiling, WatchDog, Logs. These products may have gaps in data and partial limitations based on data available to monitors. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Mar 8, 17:49 EST
Update - Database Monitoring is operational in US1. There may be gaps in historical data. We continue progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 17:06 EST
Update - Processes Monitors are operational in US1. There may be gaps in historical metric data. We continue progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Mar 8, 16:21 EST
Update - At 06:00 UTC on March 8th, 2023 the Datadog platform started experiencing widespread issues across multiple products and regions . The web application was unavailable or intermittently loading, and data ingestion & monitor evaluation were delayed.

We will share a more detailed analysis post-recovery, but at a very high level:
A system update on a number of hosts controlling our compute clusters caused a subset of these hosts to lose network connectivity
As a result a number of the corresponding clusters entered unhealthy states and caused failures in a number of the internal services, datastores and applications hosted on these clusters.

Our current status is:
We identified and mitigated the initial issue, and rebuilt our clusters
We also have recovered a number of our applications and services, including our web portals
We are now working on recovering and catching-up the rest of our data systems for metrics, traces and logs across the regions that are still affected (see region-specific status pages). The recovery work is currently constrained by the number and large scale of the systems involved.

What to expect next:
We are focusing on bringing back live data for all customers and all products before catching-up on any historical data we may have stored during the outage
We expect live data recovery in a matter of hours (not minutes, and not days)
We will continue to issue regular updates as the situation unfolds

We understand how critical Datadog is to your business, we sincerely apologize for the inconvenience and we are working hard to resolve this issue.

Mar 8, 15:39 EST
Update - We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 15:12 EST
Update - We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 14:26 EST
Update - We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 13:42 EST
Update - We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 13:14 EST
Update - We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 12:29 EST
Update - We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 11:46 EST
Update - We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 11:04 EST
Update - We are still progressing towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 10:34 EST
Update - We are still progressing towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 09:49 EST
Update - Some products are recovering and we are still progressing towards a complete recovery. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 09:07 EST
Update - Some products are recovering and we are still progressing towards a complete recovery. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 08:30 EST
Update - We are still working on the identified issue and are making continued progress towards recovery. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 07:45 EST
Update - We have identified the issue, and are making continued progress towards recovery. Data ingestion and monitor notifications remain delayed across all data types.
Mar 8, 07:08 EST
Identified - We are seeing reduced error rates for the web application. We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and you may observe delayed data throughout the app.
Mar 8, 06:20 EST
Update - We are seeing reduced error rates for the web application. We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and you may observe delayed data throughout the app.
Mar 8, 05:31 EST
Update - We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and you may observe delayed data throughout the app. Additionally, the web application continues to have elevated error rates.
Mar 8, 04:39 EST
Update - We are continuing to investigate this issue.
Mar 8, 03:52 EST
Update - We are still investigating issues causing delayed data ingestion across all data types. Monitor notifications may be delayed, and you may observe delayed data throughout the web app.
Mar 8, 03:40 EST
Update - We are still investigating issues causing delayed data ingestion across all data types. Monitor notifications may be delayed, and you may observe delayed data throughout the web app.
Mar 8, 03:05 EST
Update - We are investigating issues causing delayed data ingestion across all data types. As a result monitor notifications may be delayed, and you may observe delayed data throughout the web app.
Mar 8, 02:23 EST
Investigating - We are investigating loading issues on our web application. As a result, some users might be getting errors or increased latency when loading the web application.
Mar 8, 01:31 EST