On Monday, 08/05/2024, from approximately 3:45 p.m. until 5:50 p.m. (UTC), all instances of the KSAT console experienced delays with training and phishing campaigns.
This incident was caused when we deployed an update to the database access layer for one of our services. The update unexpectedly prevented some background jobs from completing successfully. While we manually implemented some mitigation steps, the US instance of the KSAT console also experienced approximately 15 minutes of downtime. Once the update finished deploying, KSAT was able to return to normal performance by 5:50 p.m. (UTC). Any jobs that failed to process during the deployment were able to be completed successfully after the deployment was completed.
To prevent this type of issue in the future, we made adjustments to our updating and testing processes. We also completed additional, after-hours maintenance on the remaining environments on Thursday, 08/08/2024, at 1 a.m. (UTC).
No data loss occurred as a result of this issue.