Elevated Error Rates for Log Queries and Monitors
Incident Report for Datadog US1
Resolved
This incident has been resolved.
Posted Oct 04, 2023 - 11:44 EDT
Update
Fix rollout has now been completed.
Posted Oct 04, 2023 - 11:44 EDT
Update
The fix rollout is currently ongoing. Once completed we will confirm resolution.
Posted Oct 04, 2023 - 10:50 EDT
Update
The fix rollout is currently ongoing. Once completed we will confirm resolution.
Posted Oct 04, 2023 - 09:54 EDT
Update
We have successfully tested a fix for this issue and are currently deploying it to resolve this incident.
Posted Oct 04, 2023 - 09:04 EDT
Update
We're still working on a fix for historical data impacted by this incident.
Posted Oct 04, 2023 - 08:18 EDT
Update
We're still working on a fix for historical data impacted by this incident.
Posted Oct 04, 2023 - 07:41 EDT
Update
We're still working on a fix for historical data impacted by this incident.
Posted Oct 04, 2023 - 07:07 EDT
Update
We're still working on a fix for historical data impacted by this incident.
Posted Oct 04, 2023 - 06:26 EDT
Update
We're still working on a fix for historical data impacted by this incident.
Posted Oct 04, 2023 - 05:19 EDT
Monitoring
We have deployed a fix and we are monitoring the results. We will provide another update once the issue is fully resolved.

At this time, newly ingested data is properly queryable, and monitors targeting Logs sent from 2023-10-03 20:40 UTC onwards are valid.

Queries targeting logs between 2023-10-02 11:40 UTC and 2023-10-03 20:40 UTC may return erroneous data. We are evaluating a fix that will restore query correctness for this time-window.
Posted Oct 03, 2023 - 16:47 EDT
Identified
We have identified the underlying issue and are working on a fix.
Posted Oct 03, 2023 - 15:32 EDT
Update
We are continuing to investigate these issues, and will provide an update as soon as possible.
Posted Oct 03, 2023 - 14:49 EDT
Investigating
We are actively investigating issues with Log Queries returning unexpected results.
As a result of this issue, some users may experience issues querying logs on the web application or API, and with Logs based Monitors and Log-Based Metrics.
Posted Oct 03, 2023 - 13:30 EDT
This incident affected: Log Management and Monitors.