On Thursday, September 12th, 2024, from approximately 8:20 a.m. to 12:48 p.m. (UTC), some customers on the CA and EU instances of the KSAT console experienced loading issues with training campaigns and the Learner Experience.

This incident was caused by code changes during a database upgrade. The new code worked initially, but over a longer period of time, the code needed additional configurations. We implemented those additional configurations at approximately 12:48 p.m. (UTC), and the console returned to regular performance.

To prevent this type of issue in the future, we have implemented additional monitoring for the affected code and additional testing when we update it. We also contacted the database’s third-party vendor and validated that the new configurations were accurately applied.

No data loss occurred as a result of this issue.