Subset of integration metrics delayed

Incident Report for Datadog US1

Resolved

This incident has been resolved.
Posted Jul 29, 2019 - 17:39 EDT

Monitoring

AWS metrics are current, we have identified and fixed the issues with monitor evaluation delays.
Posted Jul 29, 2019 - 17:37 EDT

Update

AWS metrics are current, we are actively working on resolving the remaining issues we identified with the monitor evaluation delays. Next update in 20m
Posted Jul 29, 2019 - 16:46 EDT

Update

AWS metrics are current, we are actively working on resolving the issues we identified with the monitor evaluation delays. Next update in 20m
Posted Jul 29, 2019 - 16:20 EDT

Update

AWS metrics are current, we are still working on resolving the issues with the monitor evaluation delays. Next update in 20m
Posted Jul 29, 2019 - 15:58 EDT

Update

AWS metrics are current, we are still working on resolving the issues with the monitor evaluation delays. Next update in 20m
Posted Jul 29, 2019 - 15:29 EDT

Update

AWS metrics are now current, we are working on resolving the issues with the monitor evaluation delays. Next update in 20m
Posted Jul 29, 2019 - 15:05 EDT

Update

More AWS metrics should be current now and we are still working through the backlog of AWS metrics. Next update in 20min.
Posted Jul 29, 2019 - 14:37 EDT

Update

A fix is being implemented and as a result a subset AWS metrics should be current now.
We are working through the backlog of AWS metrics. Next update in 20min.
Posted Jul 29, 2019 - 14:15 EDT

Update

We're still addressing an issue with the ingestion of metrics from AWS. Other integration metrics are current. Next status page update in 20 minutes.
Posted Jul 29, 2019 - 13:53 EDT

Identified

Some jobs to fetch metrics from integrations were failing to schedule, causing delays in the metrics ingestion. The team is working on mitigating this capacity issue and most integrations metrics are now current.
At this point, only AWS metrics are affected.
Posted Jul 29, 2019 - 13:24 EDT

Investigating

We’re actively investigating an issue that is affecting a subset of integration metrics. As a result users may see delays on graphs as well as get delayed monitor evaluations for monitors using those metrics.

This may result in “no data” alert conditions for Metric Monitors, to avoid spurious alerts we’ve temporarily disabled alerts on those metrics.

It’s important to note metrics are delayed but will be backfilled once all services are operational.
Posted Jul 29, 2019 - 12:58 EDT