On Tuesday, October 15, 2024, from approximately 6:33 p.m. until 6:54 p.m. (UTC), the US instance of the KSAT console experienced elevated error rates.
This incident was caused by a database query enhancement that took longer than expected to apply and prevented some new connection requests from being processed. This issue blocked certain database operations and led to increased error rates. Upon evaluating the impact, the change was canceled and the KSAT console returned to normal performance by 6:54 p.m. (UTC).
To prevent this issue in the future, we altered our enhancement strategy and implemented a new procedure for these types of updates in the future, including expanded usage of our maintenance window where required.
No data loss occurred as a result of this issue.