PhishER - Message Queue Delays and Intermittent Access Issues
Incident Report for KnowBe4
Postmortem

At approximately 3:40 p.m. (ET) on 06/29/20, the PhishER platform experienced an influx of message traffic within a short period of time, causing a delay in how quickly the messages could be processed. This delay inadvertently caused stress on some backend services, which caused the intermittent access issues some users experienced. PhishER experienced degraded console performance until around 5:04 p.m. (ET). Message queues were back to normal around 11:47 p.m. (ET). No data loss occurred as a result of this issue.

Posted Jun 30, 2020 - 14:16 UTC

Resolved
This issue has been resolved.
Posted Jun 30, 2020 - 11:22 UTC
Update
We are continuing to monitor for any further issues.
Posted Jun 30, 2020 - 02:27 UTC
Monitoring
A fix has been implemented and we are monitoring for any further issues.
Posted Jun 30, 2020 - 02:25 UTC
Identified
The cause of the issue has been identified and a fix is being implemented.
Posted Jun 29, 2020 - 21:53 UTC
Investigating
We are currently investigating reports that some message queues have been delayed and some users are experiencing intermittent console access issues.
Posted Jun 29, 2020 - 21:07 UTC
This incident affected: PhishER (Console).