On Thursday, November 8, 2024, from approximately 4:48 p.m. to 5:55 p.m. (UTC), the PhishER console experienced delays in ingesting reported messages.

This issue was caused by a third-party vendor’s software update that changed how our PhishER software package updates are deployed. The deployment changes caused the PhishER message ingester service to look for a prior archived package instead of the latest package. As a result, the service was unable to scale and experienced delays. To resolve this issue, we added the new update requirements to our code and redeployed the update at 5:31 p.m. (UTC). The ingester service resumed, and the PhishER console returned to normal performance by 5:55 p.m. (UTC).

To prevent this type of issue in the future, we have improved our deployment code to accommodate the third-party vendor’s new update requirements.

No data loss occurred as a result of this issue.