On September 8, an outage at a third-party infrastructure provider we rely on for message processing disrupted our platform for approximately 40 minutes, delaying asynchronous processing and returning errors on synchronous requests. Additionally, this slowed down our processing of transaction artifacts resulting in delayed visibility in our dashboard. Our team executed an emergency failover to backup infrastructure and restored normal processing the same afternoon. No data was lost.
In the early afternoon of September 8, a third-party infrastructure provider that Redox relies on for message processing experienced an outage. This caused approximately 40 minutes where our platform was not processing traffic. Our team immediately began an emergency failover, redirecting message processing to backup infrastructure to restore service while the third-party issue was ongoing.
This failover was successful. Processing was restored, and by 4:05 PM Central our team confirmed the platform had caught up and declared the incident resolved.
Customers experienced approximately 40 minutes of delayed asynchronous processing and errors on synchronous requests while the third-party outage was active. Additionally, our dashboard and log visibility was delayed for approximately 2 and a half hours.
Our team redirected message processing to backup infrastructure within minutes of confirming the outage. Processing was fully restored and the dashboard/log visibility was caught up by 4:05 PM Central the same day.
We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team.
The emergency failover used to resolve this incident led to a second, separate issue affecting dashboard and log visibility. See [Incident 2: Dashboard & Log Visibility Delay] for details.