We are very sorry for the inconvenience caused by this incident. We understand that our Delivery API is the backbone of your website and any full or partial outages are not up to the standard we hold ourselves to.

We are still working through the details of what exactly happened, but we want to let you know what we know so far. Two aspects of our process failed in this situation.

The first issue is of course that we deployed a release with a serious defect. The problem was not discovered during our normal review and testing process. All changes are peer-reviewed and go through multiple layers of automated testing. While we are not yet fully aware of the scale of the issue, it is clear that a subset of Delivery API requests was affected by this issue. The issue went undetected in our automated testing due to the pattern of the affected URLs was not included in our testing. The fact this incident only affected a subset of URLs caused our automated testing to falsely approve the deployment.

We are yet to identify the scale and exact pattern that was affected.

UPDATE 14th of October 2022

Upon further investigation we have conclude that URLs matching either / or https://www.eaxmple.net/ was affected, all other URLs (i. e. /about, https://www.eaxmple.net/about etc.) was not affected by this incident.

Secondly, our internal monitoring didn't pick up on the problem and notified us of the issue. We suspect that the root cause of the issue is a bug in how we validate URLs. The problem manifested itself as returning the wrong "meta HTTP status code", but from our internal monitoring system's perspective, all Delivery API requests were successfully delivered (HTTP status code 200). But from the requesting frontend's perspective, they received a 404. So from our perspective, everything seemed to be working as intended, but the clients didn't get the data they expected.

We will work on improving both our automated tests and internal monitoring to make sure a similar problem can't be released at all and furthermore that we will get alerted in case of a similar issue.

We are very sorry for the inconvenience caused by this incident and if you have any questions please reach out on support@enterspeed.com

Emil Rasmussen
CTO

The rollback was success and all Delivery API requests now returns the correct meta status code.

Around 12.45 PM CET we were notified that some Delivery API requests returned a 404 error. The error was related to a deployment of that was done at 12.35 PM CET, within minutes of us being notified we rolled back and service was fully restored at 13.03 PM CET.