Extreme depth in several individual message processing queues caused a performance issue impacting a select group of customers.
All customers exchanging data leveraging the affected processing queues within the Redox engine were affected, experiencing delays between 5 to 20 minutes.
At approximately 11:58 AM CT Redox sent notification to our customers that a message processing delay was impacting several queues. A queue with extreme depth was determined to have caused a backup, and after scaling up our system resources, Redox was able to parse through the messages to return the engine to a less overtaxed state.
Redox will be reviewing and implementing additional alerts such that we can be proactively notified of queue growth in the future.
Additionally, Redox will be evaluating our backfill process to ensure that any large parsing of messages is carefully handled in a location that will not affect daily message traffic.