Increased Delays and Error Rates (US and CA)

Incident Report for KnowBe4

Postmortem

On Wednesday, October 22, 2024, from approximately 10:15 a.m. to 12:20 p.m. (UTC), some customers experienced increased delays and errors for US and CA instances of the KSAT console.

These issues were caused by unusually high traffic in developer testing environments. These testing environments shared certain cloud computing limits with KSAT console instances, which caused slower loading times for both environments. To resolve this issue, the testing environments were adjusted to use fewer resources. The KSAT console loading times returned to normal performance by October 22, 2024, at 12:20 p.m. (UTC), and all queued tasks were completed by 2:11 p.m. (UTC).

To prevent this type of issue in the future, we have an ongoing project to separate these limits for our testing environment resources and KSAT console resources. This project is due to be completed in November 2024.

No data loss occurred as a result of this issue.

Posted Oct 28, 2024 - 16:32 UTC

Resolved

This incident has been resolved.

Posted Oct 22, 2024 - 15:25 UTC

Monitoring

We’ve implemented a fix for the delay and error issues and we’re monitoring the results to make sure no further issues occur.

Posted Oct 22, 2024 - 13:47 UTC

Identified

We've identified the cause of these issues and we are working on a fix.

Posted Oct 22, 2024 - 12:54 UTC

Investigating

We are investigating increased delays and elevated error rates in the US and CA regions of KSAT and PhishER. We will update this page when we have more information.

Posted Oct 22, 2024 - 10:28 UTC

This incident affected: PhishER (Console, Notification Service, Inbox, Integrations, PhishRIP, PhishML, Rooms, Webhooks), KnowBe4 Security Awareness Training (KSAT) (Console, Phishing, Training, Learner Experience (LX), Email Delivery, User Provisioning, Reporting, API, PIQ, Account Settings, ModStore), and SecurityCoach (Console, Security Vendor Integration Service, User Mapping Engine, Real-Time Coaching Campaigns, Coaching Delivery).