On July 28, 2026, an update to the PhishRIP service caused some PhishRIP search queries to match more broadly than they should have. Benign emails that didn't meet the configured sender criteria were quarantined across some customer accounts.
Automated monitoring and customer support tickets surfaced the problem quickly. Engineering reverted the change within approximately 25 minutes of declaring the incident and stopped the unintended quarantining. Engineering then manually restored the incorrectly quarantined emails to customers’ inboxes.
On July 28, 2026, we deployed a maintenance update to the query service behind PhishRIP. This update was designed to improve the handling of invalid sender addresses by allowing the system to fall back to searching only the domain portion of the address.
However, existing validation logic used a regular expression that required an "@" symbol in the sender field. When a query used a bare domain, such as example.com with no username or "@" symbol, the regex flagged the domain as invalid.
This conflict between the existing validation and the domain-search update caused the system to drop sender criteria entirely. The query ran against inboxes using only its remaining parameters, matching and quarantining emails well outside the intended sender.
All times are in UTC.
| Time | Event |
|---|---|
| 19:36 | Our automated monitoring detected anomalous query execution patterns and alerted our engineers, who immediately began investigating. |
| 19:51 | The issue was escalated to a high-severity production incident, and a response team was assembled. |
| 19:52 | Our support team escalated the first customer tickets that reported unexpected quarantines, corroborating the monitoring alert. |
| 19:54 | The root cause was traced to a recent queries API deployment, and engineering began reverting the change. |
| 20:00 | Incident communications were posted to our status page, with PhishRIP marked as Degraded. |
| 20:16 | The deployment was fully reverted. |
| 20:35 | Production checks confirmed that subject and domain queries were again matching only their intended targets, and that no further messages were being quarantined in error. Our status page was updated accordingly. |
| 20:39 | Engineering began restoring the incorrectly quarantined emails. |
| 03:49 (next day) | Restorations were completed for all customer query sets logged through our support team. |
Code revert. We weighed halting active PhishRIP operations mid-run against reverting the code. Halting the background tasks would have disrupted legitimate security workflows for every tenant, so reverting was the better option. The code revert was deployed and verified in production within approximately 40 minutes of escalation.
Email restoration. To avoid releasing genuinely malicious emails back into customer environments, we manually restored them rather than in bulk. Engineering used internal admin tools to restore emails in batches, and worked with Support to verify each account ID, query ID, and receive customer approval first. All reported accounts were restored overnight.
A single validation rule caused this: a regex that assumed every sender field contained an @. When queries used a bare domain, that assumption dropped the sender filter and the queries matched far more mail than intended. The fix was straightforward once we found it, and the revert stopped the damage within about 25 minutes of declaring the incident.
Restoring quarantined mail was a harder process: we couldn't restore emails in bulk without risking genuinely malicious emails going back into inboxes. Our team chose to proceed carefully, working account by account with Support throughout the night.
Two preventative measures would have caught this earlier: a test covering domain-only sender queries, and a guardrail that refuses to run a query when its sender filter has been stripped. Both are now planned for. The immediate issue is resolved, and all reported accounts have been restored.