On Tuesday, October 1, 2024, from approximately 12:00 p.m. until 10:00 p.m. (UTC), some users experienced issues with phishing and training campaigns not starting as expected, training modules not being marked as complete, and some reports not downloading on the US instance of their KSAT console.

This incident was caused when a system update to improve user enrollments did not update our scaling system and caused our queues to become backed up.

To resolve this issue, we adjusted the system settings to allow the console to scale for increased traffic.
Once the settings were adjusted, KSAT was able to return to normal performance by 10:00 p.m. (UTC).

To prevent this type of issue in the future, we updated the default settings in our code to prevent similar slowdowns. We also implemented additional monitoring for the scaling system.

No data loss occurred as a result of this issue.