KnowBe4 Incident History

Upon further investigation we have determined that the errors affecting Google PhishRIP queries were related to individual issues on Google accounts.

Based on our investigation into the recent errors affecting Google PhishRIP queries, it appears these may be related to suspended Gmail accounts. We are continuing to monitor the issue.

We are currently investigating an issue affecting Google PhishRIP queries, which may cause them to fail.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified an issue preventing account settings from loading in the SAT console.

From Tuesday, June 23, 2026, at approximately 10:03 \(UTC\) to Thursday, June 25, 2026, at approximately 11:54 \(UTC\), some US and EU customers experienced intermittent 502 errors when logging in to KCM GRC. This issue was caused by network traffic attempting to connect to invalid multilevel subdomains, which overwhelmed the cache serving KCM GRC and resulted in login errors. Our team initially updated the configuration of our content delivery network, which temporarily resolved the errors, but they returned later that day. After further investigation, we identified the caching issue as the root cause and deployed an infrastructure-level fix to prevent multi-level subdomain traffic from affecting the cache. KCM GRC returned to normal performance by 11:54 \(UTC\) on June 25, 2026. To prevent this type of issue in the future, we are evaluating additional protections to guard against similar traffic that could affect the cache. No data loss occurred as a result of this issue.

We have identified an issue where users may receive a 500 error upon logging into KCM GRC. A fix has been implemented, and users should be able to login without error.

We have identified an issue where users may receive a 500 error upon logging into KCM GRC. A fix has been implemented, and we will continue to monitor this issue.

We have identified an issue where users may receive a 500 error upon logging into KCM GRC, and our engineering team is looking into a fix.

## **Summary** On July 28, 2026, an update to the PhishRIP service caused some PhishRIP search queries to match more broadly than they should have. Benign emails that didn't meet the configured sender criteria were quarantined across some customer accounts. Automated monitoring and customer support tickets surfaced the problem quickly. Engineering reverted the change within approximately 25 minutes of declaring the incident and stopped the unintended quarantining. Engineering then manually restored the incorrectly quarantined emails to customers’ inboxes. ## **Root cause** On July 28, 2026, we deployed a maintenance update to the query service behind PhishRIP. This update was designed to improve the handling of invalid sender addresses by allowing the system to fall back to searching only the domain portion of the address. However, existing validation logic used a regular expression that required an "@" symbol in the sender field. When a query used a bare domain, such as [example.com](http://example.com) with no username or "@" symbol, the regex flagged the domain as invalid. This conflict between the existing validation and the domain-search update caused the system to drop sender criteria entirely. The query ran against inboxes using only its remaining parameters, matching and quarantining emails well outside the intended sender. ## **Timeline** All times are in UTC. | **Time** | **Event** | | --- | --- | | 19:36 | Our automated monitoring detected anomalous query execution patterns and alerted our engineers, who immediately began investigating. | | 19:51 | The issue was escalated to a high-severity production incident, and a response team was assembled. | | 19:52 | Our support team escalated the first customer tickets that reported unexpected quarantines, corroborating the monitoring alert. | | 19:54 | The root cause was traced to a recent queries API deployment, and engineering began reverting the change. | | 20:00 | Incident communications were posted to our status page, with PhishRIP marked as Degraded. | | 20:16 | The deployment was fully reverted. | | 20:35 | Production checks confirmed that subject and domain queries were again matching only their intended targets, and that no further messages were being quarantined in error. Our status page was updated accordingly. | | 20:39 | Engineering began restoring the incorrectly quarantined emails. | | 03:49 \(next day\) | Restorations were completed for all customer query sets logged through our support team. | ## **Mitigation and remediation** **Code revert.** We weighed halting active PhishRIP operations mid-run against reverting the code. Halting the background tasks would have disrupted legitimate security workflows for every tenant, so reverting was the better option. The code revert was deployed and verified in production within approximately 40 minutes of escalation. **Email restoration.** To avoid releasing genuinely malicious emails back into customer environments, we manually restored them rather than in bulk. Engineering used internal admin tools to restore emails in batches, and worked with Support to verify each account ID, query ID, and receive customer approval first. All reported accounts were restored overnight. ## **Preventative measures** * **Regression coverage**: We are adding unit and integration tests for domain-only, non-standard, and bare-string sender queries to catch these input cases before deployment. * **Query guardrails**: We are adding a check in the query engine that blocks queries from running if their core filtering parameters are dropped during evaluation. ## **Conclusion** A single validation rule caused this: a regex that assumed every sender field contained an `@`. When queries used a bare domain, that assumption dropped the sender filter and the queries matched far more mail than intended. The fix was straightforward once we found it, and the revert stopped the damage within about 25 minutes of declaring the incident. Restoring quarantined mail was a harder process: we couldn't restore emails in bulk without risking genuinely malicious emails going back into inboxes. Our team chose to proceed carefully, working account by account with Support throughout the night. Two preventative measures would have caught this earlier: a test covering domain-only sender queries, and a guardrail that refuses to run a query when its sender filter has been stripped. Both are now planned for. The immediate issue is resolved, and all reported accounts have been restored. ## **Glossary** * **PhishRIP:** A PhishER feature that lets administrators search for and quarantine matching phishing emails across every user inbox in an organization. * **Regex \(regular expression\):** A pattern used in software to search text and validate input, such as checking that an email address is formatted correctly. * **Code revert:** Backing out a software update to return to the previous stable version.

This incident has been resolved.

A fix has been implemented for this issue and PhishRIP queries are no longer quarantining messages unexpectedly. We are still investigating queries that may have been impacted

We are investigating an issue related to emails being quarantined by PhishRIP unexpectedly

This incident has been resolved.

A fix has been implemented for this issue and PhishRIP queries are no longer quarantining messages unexpectedly. We are still investigating queries that may have been impacted

We are investigating an issue related to emails being quarantined by PhishRIP unexpectedly

We’ve implemented a fix for the online store and we’re monitoring the results to make sure no further issues occur.

We have received reports that customers are unable to purchase Protect licenses on store.knowbe4.com. We are investigating this issue and will update this page when we have more information.

We’ve implemented a fix and we’re monitoring the results to make sure no further issues occur.

We are investigating users being unable to purchase Protect licenses on store.knowbe4.com. We will update this page when we have more information.

We’ve implemented a fix for the online store and we’re monitoring the results to make sure no further issues occur.

We have received reports that customers are unable to purchase Protect licenses on store.knowbe4.com. We are investigating this issue and will update this page when we have more information.

We’ve implemented a fix and we’re monitoring the results to make sure no further issues occur.

We are investigating users being unable to purchase Protect licenses on store.knowbe4.com. We will update this page when we have more information.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

We will be performing maintenance on the US region instance of Prevent on July 29, 2026 from 10:00 a.m. to 11:30 a.m. (UTC). During maintenance, Prevent users may experience up to 60 minutes of service disruption.