Elavated number of API errors and timeouts
Incident Report for Reepay
Postmortem

Our service provider has finished an investigation. Their response:

It can be concluded that one of the underlying database storage volumes was observing hardware issue and was unable to perform at the expected rate. The internal monitoring system was able to identify this issue and under take corrective measure by replacing it with a healthy volume. We do monitor for problems with the underlying hardware so we can schedule maintenance windows and alert our customers in advance. However, failures cannot always be predicted and I deeply regret the inconvenience caused. We make every possible effort to ensure we are highly available and resilient, but as you know, hardware and networking equipment are subject to failures, maintenance, and may experience issues.

Having said that, please note that hardware failures do occur very rarely, and when they do, the agility and durability of the automation takes appropriate measures to quickly make the database instance available. In this case the issue was resolved in approximately two minutes.

Posted Dec 28, 2023 - 14:55 CET

Resolved
From approximately 00:56 CET to 00:59 CET we experienced latency issues with a database causing slower than normal response times from our services, and timeouts. We are looking into the incident with our service provider which seems to be caused by a hardware failure of the underlying storage subsystems.
Posted Dec 28, 2023 - 01:00 CET