Summary

During a routine deployment on August 26, 2026, an infrastructure management framework unexpectedly removed both the primary message-processing queues and the secondary queues that hold messages that could not be processed. This resulted in a temporary interruption of incoming email delivery to PhishER customers' inboxes.

No data loss occurred as a result of this issue.

Root Cause

On August 24, 2026, an internal configuration change left live configuration records out of sync with our code. A later code update did not revert those records.

When a routine production deployment ran on August 26, 2026, the active configuration code did not contain the relocated resource records. As a result, the deployment treated the active queues as unmanaged resources and removed them.

Mitigation and Remediation

Automated monitoring alerted us to a drop in message processing, and an incident response team was assembled.

We found that the queues had been deleted. This was mitigated in the short term by rerunning the affected production jobs to recreate the missing queues.

We then verified that real-time email processing had resumed. Finally, we replayed email payloads held in cloud storage during the outage into customers' inboxes.

Timeline

All times are in UTC.

• August 24, 2026, 18:41: An internal configuration change wrote updated resource records into live production configuration.
• August 26, 2026, 15:15: A production deployment removed the inbound queues and the queues that hold messages that could not be processed in production, halting email delivery into customers' inboxes.
• August 26, 2026, 15:59: Our automated monitoring alerted the engineering team to zero messages being processed, and an incident response team was paged.
• August 26, 2026, 16:01: We confirmed the queues had been deleted and verified that incoming email payloads were still stored safely.
• August 26, 2026, 16:15: We identified the configuration mismatch as the primary root cause.
• August 26, 2026, 16:25: Our team reran the affected production jobs, and the queues were recreated.
• August 26, 2026, 18:44: Our automated replay finished delivering all backlogged emails to customers' inboxes.

Preventative Measures

To prevent this type of issue in the future, we've removed a live-configuration step from pre-merge checks and separated deployment planning from production changes. We've also added automated checks to confirm queue health after releases are complete.

Customer Impact

From 15:15 to 16:25 (UTC) on August 26, 2026, incoming emails submitted to PhishER were not processed into customer inboxes.

Inbound email payloads were stored safely in cloud storage throughout the disruption. After the queues were restored, an automated replay delivered held emails to customers' inboxes. All backlogged messages were delivered by 18:44 (UTC).

No customer email data was lost as a result of this issue.