PhishER - Message Delays
Incident Report for KnowBe4
Postmortem

On March 5, 2021 from approximately 1:45 to 5:30 p.m. (ET), processing of new messages, actions, and rule evaluations in PhishER experienced delays. These delays occurred because a batch of messages with a large quantity of links was sent to PhishER in a short period of time. The number of links caused a spike in queued tasks, temporarily blocking unrelated queue inquiries from processing.

These messages were moved into a low priority queue, allowing the query to process new inquiries at its normal rate. Moving the messages resolved the delays for all other users in PhishER. We are working on an enhancement to prevent this type of issue in the future.

No data loss occurred as a result of this issue.

Posted Mar 09, 2021 - 17:18 EST

Resolved
This issue has been resolved. Message queues have returned to normal.
Posted Mar 05, 2021 - 17:18 EST
Monitoring
We’ve implemented a fix for this issue and are monitoring the results. Some message delays might still be occurring but we are expecting queues to return to normal within two hours.
Posted Mar 05, 2021 - 16:42 EST
Identified
We’ve identified the cause of this issue and are implementing a fix.
Posted Mar 05, 2021 - 15:35 EST
Investigating
Some PhishER users may be experiencing message delays. We will update this page when we have more information.
Posted Mar 05, 2021 - 14:07 EST
This incident affected: PhishER (Notification Service).