KMSAT, KCM, and PhishER - Increased Error Rates and Latencies (US Only)
Incident
Report for KnowBe4
Postmortem
On Tuesday, 6/13/2023, from approximately 6:58 p.m. to 9:45 p.m. (UTC), all of KnowBe4's products on the US instance experienced job processing delays.
This incident was caused by a third-party service outage that affected multiple organizations in the eastern region of the United States. Once the third-party service was restored, KnowBe4's products were able to resume job processing by 9:45 p.m. (UTC) and return to normal performance by 10:40 p.m. (UTC).
To help prevent similar issues, we will continue to improve our software and reduce recovery times after third-party outages.
No data loss occurred as a result of this issue.
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented and most workloads are returning to normal performance. We are continuing to monitor all systems and we will continue to post on our status page with any new information or updates.
Identified
We’ve identified the cause of the increased error rates and latencies across all KnowBe4 products. We are working on implementing a fix, and we’ll continue to post on our status page with any new information or updates.
This incident affected: KCM GRC (Console), PhishER (Console, Notification Service, Inbox, Integrations, PhishRIP, PhishML, Rooms, Webhooks), KnowBe4 Security Awareness Training (KSAT) (Console, Phishing, Training, Learner Experience (LX), Email Delivery, User Provisioning, Reporting, API), and SecurityCoach (Console, Security Vendor Integration Service, User Mapping Engine, Real-Time Coaching Campaigns, Coaching Delivery).