Message Processing Delay for Subset of Customers
Incident Report for Redox Engine

Root cause

Extreme depth in several individual message processing queues caused a performance issue impacting a select group of customers.

Impact on customers

All customers exchanging data leveraging the affected processing queues within the Redox engine were affected, experiencing delays between 5 to 20 minutes.

What Happened?

At approximately 11:58 AM CT Redox sent notification to our customers that a message processing delay was impacting several queues. A queue with extreme depth was determined to have caused a backup, and after scaling up our system resources, Redox was able to parse through the messages to return the engine to a less overtaxed state.

Learnings / Follow-ups

Redox will be reviewing and implementing additional alerts such that we can be proactively notified of queue growth in the future.

Additionally, Redox will be evaluating our backfill process to ensure that any large parsing of messages is carefully handled in a location that will not affect daily message traffic.

Posted Nov 29, 2021 - 13:17 CST

This incident has been resolved.
Posted Nov 19, 2021 - 14:03 CST
A fix has been implemented and we are monitoring the results.
Posted Nov 19, 2021 - 13:44 CST
The issue has been identified and a fix is being implemented.
Posted Nov 19, 2021 - 12:55 CST
We are currently investigating a processing delay that is affecting a subset of customers. We will update with more information as it becomes available.
Posted Nov 19, 2021 - 11:59 CST
This incident affected: Engine Core.