Inbound SFTP Processing Degradation
Incident Report for Redox Engine
Postmortem

Executive Summary

Root cause

An issue on 10/19 with Redox’s key management services resulted in our SFTP Polling service being unable to communicate with the Redox Engine, and as a result was unable to support polling SFTP connections.

Impact on customers

All customers using SFTP Polling with Redox were unable to poll messages from Redox beginning at 11:22 AM CT on 10/19. The issue was discovered at approximately 08:00 AM CT on 10/20 and was resolved at approximately 09:54 AM CT.

What Happened?

At approximately 08:00 AM CT we discovered that our SFTP Polling was unable to communicate with the Engine, and was subsequently unable to poll SFTP connections. The issue originated as a result of another issue incurred on 10/19, which caused a large backup for many customers.

Learnings / Follow-ups

Redox is looking into adjusting our SFTP Processing alert times to ensure that we are alerted to these issues as quickly as possible so that such a backup does not occur again.

Posted Nov 03, 2021 - 08:00 CDT

Resolved
This incident has been resolved.
Posted Oct 20, 2021 - 10:32 CDT
Update
A fix has been implemented and we are monitoring the results.
Posted Oct 20, 2021 - 09:38 CDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 20, 2021 - 09:37 CDT
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 20, 2021 - 08:52 CDT
Investigating
We are currently investigating an issue that is impacting Redox's ability to poll files from external SFTP sources.

At this time, files writing to Redox's SFTP server are not impacted and are processing successfully. Outbound transmission processing is also not impacted at this time.
Posted Oct 20, 2021 - 08:23 CDT
This incident affected: Polling.