We have investigated the cause of the errors earlier today.

The cause of the problems was due to a new JSON serialiser that was part of today's deployment. As a side note, we can share that we process a lot of JSON, and we can gain a significant performance improvement with the new JSON serialiser, but we digress.

In one specific use case, we had not tested the new code, which resulted in 0.83% of Ingest API requests returning an error response in a scenario where the requests were supposed to be valid.

We will update our automated tests to include this scenario, so this will not happen again.

We apologise for the inconvenience. If you have any questions, then please reach out to support@enterspeed.com or via your Slack support channel.

The rollback procedure was completed a few minutes ago, and we have verified that we are back to normal operations.

Our initial investigation shows that the v1 endpoint was affected, while the v2 endpoint was not.

We will get back to you with more details once we have investigated the scope and cause of the problem.

The errors appears to be related to a recent deployment and we have initiated the rollback procedure.

We will report back with when the normal operations of the Ingest API is restored.

We have seen an increase of failed ingests on the Ingest API. We are currently investigating.