KSAT Intermittent Login Issues - US

Executive Summary

On July 1, 2026, the KnowBe4 Security Awareness Training (KSAT) platform experienced a period of degraded performance resulting in intermittent login errors and high latency for users on our United States (US) instance. The issue was initiated following a routine platform deployment that introduced an unoptimized database query. This query placed an excessive operational load on our primary database reader cluster, causing database sessions to saturate and subsequent login requests to queue up.

Engineering teams promptly identified the degradation, reverted the deployment, and systematically cleared the backlogged database sessions to restore optimal performance. The issue did not affect data integrity or security, and service was fully stabilized.

Technical Root Cause

The root cause was determined to be a newly introduced query within a standard application update. Upon deployment, this specific query pattern bypassed optimal indexing strategies, resulting in full table scans and highly extended execution times on the database reader infrastructure.

As a high volume of authentication and application requests arrived concurrently, the database reader quickly exhausted its available connection pool due to these long-running, unmitigated database sessions. This resource starvation immediately manifested as severe application latency and intermittent timeouts during the user authentication process.

Timeline of Events

The incident began in the mid-morning hours and progressed through identification, remediation, and verification stages over a period of approximately 152 minutes.

An internal high-severity incident response group was established immediately following automated monitoring alerts indicating that health checks targeting our application programming interface (API) routing layer were failing from regional cloud monitoring nodes. This behavior was confirmed by concurrent engineering analysis of user HTTP Archive files, which demonstrated severe latency spikes specifically isolated to the authentication endpoints.

Within three minutes of establishing the response team, cross-referencing recent system changes pointed to a recent application deployment as the primary catalyst. Engineers immediately initiated the rollback process, drafting and approving a revert modification to extract the problematic code from the deployment pipeline.

The deployment of the reverted codebase to the US production cluster commenced shortly thereafter. While the deployment processed over the subsequent twenty-five minutes, technical personnel prepared direct database interventions to clear the residual system strain. Once the stable code version was completely active across the fleet, engineers began systematically terminating the lingering, long-running database sessions that had been spawned by the unoptimized query. To ensure a pristine state, the database reader infrastructure was cycled twice.

Following these administrative infrastructure restarts, operational telemetry showed the database reader load dropping significantly to a healthy baseline of approximately thirty-six percent. System performance normalized, and administrative logging verified that authentication requests were processing within standard latency thresholds. After monitoring the environment to confirm sustained stability, engineers officially marked the incident as mitigated, later shifting the status to fully resolved following an extended window of zero performance spikes and complete passes on all automated sanity test suites.

Mitigation

To alleviate the immediate infrastructure distress and restore user access, the engineering team executed a multi-phased mitigation strategy:

• Codebase Rollback: The changes introduced in the recent deployment were immediately isolated, reverted, and redeployed to production to prevent any further generation of the unoptimized query.
• Database Session Termination: Internal engineering tools were utilized to explicitly terminate active, long-running database queries that were blocking the connection pools.
• Infrastructure Cycling: The database reader instances were restarted twice in succession to flush out stale memory allocations and guarantee that all orphaned database sessions were permanently cleared.

Preventative Measures

To prevent a recurrence of this specific issue and mitigate the impact of similar query-based database bottlenecks in the future, KnowBe4 is implementing the following actions:

• Enhanced Query Linting and Analysis: Integrate automated query execution plan analysis into our continuous integration and continuous deployment pipelines to flag unindexed or high-cost queries before they reach production environments.
• Database Connection Pooling Guardrails: Adjust database timeouts and implement aggressive circuit-breaker thresholds for user authentication paths to prevent single, long-running query patterns from exhausting the entire connection pool.
• Load Shedding Policies: Implement strict application-level timeouts on read-heavy database calls to ensure they fail gracefully rather than degrading the overall availability of the core login workflows.

Conclusion

We sincerely apologize for the inconvenience and friction this performance degradation caused our customers and partners. KnowBe4 is dedicated to maintaining high availability and reliability across our product suites. By refining our pre-deployment automated query validation and bolstering our database connection resiliency, we are actively working to ensure the continuous, seamless operation of the KSAT platform.

Glossary of Technical Terms

• API (Application Programming Interface): A set of protocols that allows different software applications to communicate with one another. In this context, it routes authentication requests from the user interface to the backend servers.
• Database Reader: A dedicated database instance or cluster responsible for handling read-only queries (such as fetching user profiles or validating login configurations), separating this traffic from write operations to optimize performance.
• HAR (HTTP Archive) File: A JSON-formatted log file that records a web browser's interaction with a website, used by engineers to diagnose performance and network latency issues.
• Latency: The time delay or duration it takes for a data packet or request to travel from its source to its destination and return a response.
• Sanity Suite: A collection of automated tests executed against a deployment environment to quickly verify that the core functionality of an application is working correctly.