Phisher Outage During Maintenance

Incident Report for KnowBe4

Postmortem

On Thursday, April 17, 2025, from approximately 1:10 a.m. to 6:46 a.m. (UTC), some customers were unable to access their PhishER console.

This issue was caused by a planned maintenance update to PhishER. During the maintenance, an unexpected configuration issue caused errors for our database clusters, and PhishER data could not be loaded. To resolve this issue, we identified the affected clusters and initiated restorative processes. All database clusters were restored to full operation by 5:10 a.m., and the PhishER console returned to normal performance by 6:46 a.m. (UTC).

To prevent this type of issue in the future, we have added additional safeguards and testing protocols for future updates. We have also improved configuration controls for similar database updates.

No data loss occurred as a result of this issue.

Posted May 01, 2025 - 15:19 UTC

Resolved

We have been closely monitoring PhishER in all regions after last night's maintenance, and we have not identified any lingering effects. This is resolved.

Posted Apr 17, 2025 - 14:57 UTC

Monitoring

All services appear to have returned to normal. We will continue to monitor PhishER to ensure complete functionality.

Posted Apr 17, 2025 - 06:23 UTC

Update

Impacted regions are beginning to recover from the unanticipated downtime. The inbox is available in our US region. The inbox in other regions and other services in all regions - such as email ingestion - are beginning to recover at this time.

Posted Apr 17, 2025 - 05:30 UTC

Update

Progress is being made on addressing the unanticipated downtime that resulted from this planned maintenance and we are working on bringing affected regions back online one-by-one.

Posted Apr 17, 2025 - 04:57 UTC

Identified

During our maintenance window this evening, PhishER is experiencing unanticipated downtime. The engineering team has identified the cause and is currently working to address the issue.

This is affecting PhishER in all regions.

Posted Apr 17, 2025 - 01:56 UTC

This incident affected: PhishER (Console).