MLLP Inbound Message Processing Degradation
Incident Report for Redox Engine

Root cause

On Monday 10/4, ongoing infrastructure scalability tasks inadvertently caused increased DNS lookups by services between 2:41PM CT and 7:00PM CT.

What Happened?

DNS performance degradation led to application services degrading as a result. We reverted partial changes on Monday (10/4) evening, which immediately improved performance. As volume increased we experienced a more severe degradation on Tuesday 10/5. Reverting all changes deployed between 2:41PM CT and 7:00PM CT on Monday 10/4 resolved DNS lookup performance, and returned message processing to normal speed.

Impact on customers

Customers with high-volume MLLP feeds experienced messages queueing up on the HCO side due to a sporadic increased latency for all HTTP requests requiring DNS resolution on the order of seconds (5-10).

Learnings / Follow-ups

Internal teams are evaluating technology which allows us to improve our testing and alerting on network degradation.

Posted Oct 15, 2021 - 15:05 CDT

This incident has been resolved.
Posted Oct 05, 2021 - 16:45 CDT
A fix has been implemented and we are monitoring the results.
Posted Oct 05, 2021 - 12:13 CDT
The issue has been identified and a fix is being implemented.
Posted Oct 05, 2021 - 11:50 CDT
We are continuing to investigate this issue.
Posted Oct 05, 2021 - 11:50 CDT
In addition to MLLP Routing we are experiencing a delay within our dashboard.
Posted Oct 05, 2021 - 10:40 CDT
We are continuing to investigate this issue.
Posted Oct 05, 2021 - 10:37 CDT
We are currently experiencing an issue with our inbound MLLP processing. We are actively investigating and should have an update shortly. If you have any questions please contact
Posted Oct 05, 2021 - 10:34 CDT
This incident affected: Dashboard and Engine Core.