KSAT - Campaign Delays
Incident Report for KnowBe4
Postmortem

On Monday, 08/05/2024, from approximately 3:45 p.m. until 5:50 p.m. (UTC), all instances of the KSAT console experienced delays with training and phishing campaigns.
This incident was caused when we deployed an update to the database access layer for one of our services. The update unexpectedly prevented some background jobs from completing successfully. While we manually implemented some mitigation steps, the US instance of the KSAT console also experienced approximately 15 minutes of downtime. Once the update finished deploying, KSAT was able to return to normal performance by 5:50 p.m. (UTC). Any jobs that failed to process during the deployment were able to be completed successfully after the deployment was completed.
To prevent this type of issue in the future, we made adjustments to our updating and testing processes. We also completed additional, after-hours maintenance on the remaining environments on Thursday, 08/08/2024, at 1 a.m. (UTC).
No data loss occurred as a result of this issue.

Posted Aug 12, 2024 - 15:40 UTC

Resolved
This incident has been resolved.
Posted Aug 05, 2024 - 20:05 UTC
Update
The US instance of the KSAT web console experienced approximately 15 minutes of downtime while we implemented the fix for the previous issue. This issue has been mitigated as well. No additional issues are expected from this incident. Some after-hours maintenance will be performed this Wednesday at 9pm EDT to reconcile some other components of this release.
Posted Aug 05, 2024 - 18:05 UTC
Monitoring
We’ve implemented a fix for the issue causing delays in the KSAT console, and we’re monitoring the results to make sure no further issues occur.
Posted Aug 05, 2024 - 17:11 UTC
Identified
We've identified the cause of an issue causing delays with training and phishing campaigns in the KSAT console. We are implementing a fix, and we'll continue to post on our status page with any new information or updates.
Posted Aug 05, 2024 - 16:30 UTC
This incident affected: KnowBe4 Security Awareness Training (KSAT) (Console, Phishing, Training).