KSAT - Login Issues
Incident
Report for KnowBe4
Postmortem
On Tuesday, 07/30/2024, from approximately 9:56 p.m. to Wednesday, 07/31/2024 at 2:20 a.m. (UTC), all of KnowBe4's products on the US instance experienced elevated error rates and intermittent service disruption.
This incident was caused by an unexpected technical issue that impacted many of our systems. The disruption began to recover around Wednesday, 07/31/2024, at 1:47 a.m. (UTC), KnowBe4's products were able to return to normal performance by 2:20 a.m. (UTC)
No data loss occurred as a result of this issue.
Resolved
This incident has been resolved.
Monitoring
At 2:20am UTC, our error rate returned to normal. It has remained low for 30 minutes, and we have seen signs of all impacted features and services reaching or nearing a full recovery. We will leave this incident in Monitoring status for the next few hours as we continue to ensure the health of our products and features.
Update
We are seeing good signs of recovery and a significant reduction in our error rate. Most delayed background jobs were able to process successfully around 2:00 UTC. We are still working towards a full recovery.
Update
We are continuing to work on resolving this incident. Some services are showing early signs of recovery, though we are still experiencing elevated error rate. Most background jobs are still processing successfully with some elevated delays or latency.
Identified
We have identified the cause of this incident, and we are continuing to investigate the scope of this issue.
Investigating
We are continuing to investigate this issue.
Identified
We are currently investigating an issue with the KSAT Console in the US region.
This incident affected: KCM GRC (Console), PhishER (Console, Notification Service, Inbox, Integrations, PhishRIP, PhishML, Rooms, Webhooks), KnowBe4 Security Awareness Training (KSAT) (Console, Phishing, Training, Learner Experience (LX), Email Delivery, User Provisioning, Reporting, API, PIQ), and SecurityCoach (Console, Security Vendor Integration Service, User Mapping Engine, Real-Time Coaching Campaigns, Coaching Delivery).