Some production connections across Redox that were not yet live had been paused for an extended period of time; when these connections were unpaused the Redox engine tried and was unable to ingest a very old transmission, which caused a backup of some transmissions that affected a small subset of customers.
A handful of customers on a particular partition were affected by message processing delays.
At approximately 4:56PM CT observed an increase in message processing time and began investigating the issue. By 5:41PM CT a fix of increasing the partition size was implemented and we observed the decrease of queue depth to resolve the issue.
Redox has committed to creating and implementing alerting based on this specific scenario and areas affected.