KMSAT and PhishER - Message Delays
Incident Report for KnowBe4
Postmortem

From approximately 9:09 a.m. (ET) to 1:07 p.m. (ET), our service for sending specific types of emails from KMSAT or PhishER experienced downtime. During this period, message delivery was delayed. Affected email types included: KnowBe4 training notifications, PhishER emails, account emails (such as password resets and account confirmations), and report emails (such as phishing test reports, user import confirmations, and EEC Pro results).

The delay began to clear at approximately 12:55 p.m. (ET) after the mail was rerouted through a different IP address which was not experiencing an outage. No data loss occurred as a result of this issue.

To prevent this issue in the future, the affected email services will now send from the following IP range: 147.160.167.0/26. Review our whitelisting guide for additional whitelisting assistance.

Posted Jul 30, 2020 - 16:45 UTC

Resolved
This issue has been resolved. To prevent this issue in the future, the affected email services will send from the following IP range: 147.160.167.0/26. See our main whitelisting article on support.knowbe4.com for more information on our IP addresses and how to whitelist.
Posted Jul 29, 2020 - 19:45 UTC
Monitoring
We have implemented a fix for this issue and all previously delayed messages have now been sent. We are monitoring for any further issues.
Posted Jul 29, 2020 - 17:10 UTC
Identified
We have identified the issue and are working to implement a fix.
Posted Jul 29, 2020 - 15:08 UTC
Investigating
Since 9:05 a.m. (ET), our KMSAT and PhishER products have experienced message queue delays. Some messages that were scheduled to go to users may be delayed. We are currently investigating the cause of this issue.
Posted Jul 29, 2020 - 14:37 UTC
This incident affected: PhishER (Notification Service) and Kevin Mitnick Security Awareness Training (KMSAT) (Email Delivery).