Some customers may see gaps in Browser Test results between Friday, 13 November 23:00 UTC and Saturday, 14 November 20:30 UTC. Historical Data is being reprocessed and gaps will be filled over the next several hours. Impacted customers will see notifications of gaps in-app.
Posted Nov 14, 2020 - 17:10 EST
Monitoring
Live Synthetics Browser tests are restored.
As a result of the incident, Browser Test results may have gaps since Friday, 13 November at 23:00 UTC. Gaps in data will be filled over the next several hours as historical data continues to be processed. It is important to note that no data has been permanently lost.
We will be posting an update on Saturday, 14 November at 00:00 UTC.
Posted Nov 14, 2020 - 16:37 EST
Update
Live Synthetics Browser tests are now restored. Synthetics Browser Tests are running from all 15 datacenters (Asia Pacific - Tokyo/Mumbai/Singapore/Seoul/Sydney, EU - Frankfurt/London/Paris/Stockholm/Ireland, South America - São Paulo, Canada - Central, US - Ohio, Oregon, N. California).
As a result of the incident, Browser Test results may have gaps since Friday, 13 November at 23:00 UTC. We are beginning to backfill missing data. It is important to note that no data has been permanently lost.
We will be posting a new update at 21:30 UTC.
Posted Nov 14, 2020 - 15:35 EST
Update
Synthetics Browser Tests are now running from 10 datacenters (Asia Pacific - Tokyo/Mumbai/Singapore/Seoul/Sydney, EU - Frankfurt/Paris/Stockholm/Ireland, South America - São Paulo) as we are continuing the work on mitigations and backfilling. We will be posting a new update at 21:00 UTC.
Posted Nov 14, 2020 - 15:05 EST
Update
Synthetics Browser Tests are now running from 8 datacenters (Asia Pacific - Mumbai/Singapore/Seoul/Sydney, EU - Paris/Stockholm/Ireland, South America - São Paulo) as we are continuing the work on mitigations and backfilling. We will be posting a new update at 20:00 UTC.
Posted Nov 14, 2020 - 14:00 EST
Update
Synthetics Browser Tests are now running from 8 datacenters (Asia Pacific - Mumbai/Singapore/Seoul/Sydney, EU - Paris/Stockholm/Ireland, South America - São Paulo) as we are continuing the work on mitigations and backfilling. We will be posting a new update at 19:00 UTC.
Posted Nov 14, 2020 - 13:09 EST
Update
Synthetics Browser Tests are still running only from 7 datacenters as we are continuing the work on mitigations and backfilling. We will be posting a new update at 18:00 UTC.
Posted Nov 14, 2020 - 12:11 EST
Update
Synthetics Browser Tests now have live results coming in from 7 different datacenter locations (Asia Pacific - Mumbai, Asia Pacific - Singapore, Asia Pacific - Seoul, Asia Pacific - Sydney, EU - Paris, EU - Stockholm, EU - Ireland, South America - São Paulo), we are working on mitigating the issue for the remaining datacenters. At the same time, we are continuing working on backfilling test results which may cause gaps in Synthetics pages and metrics. We will be posting a new update at 17:00 UTC.
Posted Nov 14, 2020 - 11:13 EST
Update
Synthetics Browser Tests now have live results coming in from 7 different datacenter locations (Asia Pacific - Mumbai, Asia Pacific - Singapore, Asia Pacific - Seoul, EU - Paris, EU - Stockholm, EU - Ireland, South America - São Paulo), we are working on mitigating the issue for the remaining datacenters. At the same time, we are continuing working on backfilling test results which may cause gaps in Synthetics pages and metrics. We will be posting a new update at 16:00 UTC.
Posted Nov 14, 2020 - 10:08 EST
Update
At this point, all Synthetics Tests outside of Synthetics Browser Tests are working as expected but customers might see gaps in historical data and we are continuing to work on backfilling. For Synthetics Browser Tests, we are continuing to work on mitigating the issue and results are still partial. We will be posting a new update at 15:00 UTC.
Posted Nov 14, 2020 - 09:07 EST
Identified
We have identified an issue with synthetics which results in delayed test results and possible gaps in synthetics based metrics. We are actively mitigating the issue and are working on backfilling delayed test results.