On February 11th at 11:30 AM CT, a small number(<20) email notifications to users were delayed. These notifications were not sent until February 13th at 12:35 PM CT.
What Happened
On February 11th, a misconfigured API key in a new notification service prevented email notifications from being sent from a subset of systems.
As a result, customers experienced delays in receiving:
Notifications about traffic errors and resolutions.
Notifications about upcoming certificate expirations (30 days out).
The API key was updated and normal processing resumed on February 13th at 12:35 PM CT.
What we are doing about this:
Additional monitoring and alerting were added to prevent similar potential failures in the future.
Posted Feb 20, 2025 - 15:30 CST
Resolved
This incident has been resolved.
Posted Feb 13, 2025 - 11:38 CST
Monitoring
A fix has been implemented and customers will now start receiving the queued emails. We are continuing to monitor the results.
Posted Feb 13, 2025 - 11:15 CST
Identified
Two email notification services that Redox uses are down. These services affect emails for notifying customers of expiring certificates and for message traffic or VPN related issues. We have identified the issue and are implementing a fix.
When the issue is resolved, customers will receive all backlogged emails that would have been sent while the services were down.