On Monday, 07/08/2024, from approximately 8:50 a.m. until 12:05 p.m. (UTC), some users experienced issues in the US instance of the KSAT Learner Experience (LX). Videos did not play properly, and training sessions expired unexpectedly or did not record completion.
This incident was caused by a software component in one of our libraries that left behind unnecessary data over time, which led to our system's memory storage running out of space.
Once we updated the software component and removed unnecessary data, KSAT was able to return to normal performance by 12:05 p.m. (UTC).
To prevent this type of issue in the future, we increased the system’s memory storage space, upgraded the system after updating the software component, and set up regular cleanup tasks to remove old data.
During this incident, some training progress was not tracked. Users who were completing training may have needed to redo the training that was in progress during this incident.