Logs observability is delayed, some processing impacted

Incident Report for Redox

Postmortem

Summary

At approximately 10:39AM CT, September 29 2025, we observed system-wide degradation creating latency for transaction processing and observability. Latency was resolved for most customers by about 13:30PM CT, Sep 29, 2025.

At 4:43AM CT, September 30 2025, we observed a similar system degradation. Latency was resolved for customers by 13:23PM CT, September 30 2025.

What Happened

A confluence of several issues generated a number of errors in our system. These factors included a change in the way we manage database partition rotations, an upgrade of a core library, and the interaction of some services with kubernetes.
These errors caused some services to crash, and although our infrastructure is fault-tolerant and can recover from such crashes, the degree of crashing put pressure on our infrastructure in a way that manifested as latency in transaction processing, and a delay in transaction observability.

What we are doing about this:

We rolled back the upgrade made to the affected third-party library.
We are adding and improving alerting on affected services to provide clearer visibility to the teams that are responsible for those services.

Posted Oct 22, 2025 - 19:43 CDT

Resolved

Logs should now be visible across the entire Redox platform. If you have any addition questions or are still experiencing issues, please reach out to support@redoxengine.com

Posted Sep 30, 2025 - 13:23 CDT

Monitoring

A fix has been implemented for the log visibility issues and most logs should now be visible. We are actively monitoring the situation and ensuring that all affected customers' logs are visible again. The previously mentioned processing delays have been resolved.

If you have questions or are still experiencing issues, please reach out to support@redoxengine.com

Posted Sep 30, 2025 - 13:01 CDT

Identified

We are currently aware of an issue in which the the last one hour of logs are not properly displaying on the dashboard. Some processing for synchronous traffic may also be delayed.

What this means: While asynchronous logs are still processing properly, the dashboard is not displaying them as expected. Search within the dashboard will not work as expected until a fix has been implemented. Additionally, synchronous logs may take longer to process than normal.

If you have questions about specific logs or your connection and whether it has been affected, please contact us at support@redoxengine.com

Posted Sep 30, 2025 - 11:54 CDT

This incident affected: Traffic Processing and Logs (view/search).