KnowBe4 Incident History

1. CompletedOct 28 · 20:00 UTC

   The scheduled maintenance has been completed.

Reported by KnowBe4 on their status page.

1. resolvedOct 23 · 14:55 UTC

   This incident has been resolved.

Reported by KnowBe4 on their status page.

1. ResolvedOct 23 · 14:55 UTC

   This incident has been resolved.

Reported by KnowBe4 on their status page.

1. CompletedOct 28 · 21:00 UTC

   The scheduled maintenance has been completed.

Reported by KnowBe4 on their status page.

1. postmortemOct 14 · 14:29 UTC

   Summary

   On October 13, 2026, from approximately 14:19 to 15:05 (UTC), customers in all regions were unable to log in to the KnowBe4 Security Awareness Training (SAT) console, the KnowBe4 Learner Experience, and the PhishER console through KnowBe4 SAT. SAML and password logins returned errors, and the KnowBe4 SAT console didn't appear in the KnowBe4 platform. Some background processes, including user synchronization and training emails, also failed during this time.

   The incident was caused by a production release that conflicted with another change in our code. We reverted the release, and logins recovered across regions between 14:59 and 15:05 (UTC).

   No data loss occurred as a result of this issue.

   Root Cause

   A code deployment removed a startup configuration value to speed up application startup time. A separate component, added to our code after the original change was tested, didn’t account for that changed value. As a result, every attempt to save a user record raised an error, including the update that records sign-in attempts.

   Customer Impact

   Customers across all regions couldn't log in to the KSAT console via SAML or password. They were also unable to access the Learner Experience, and couldn’t access the PhishER console through the KnowBe4 SAT console. New user creation, user synchronization, and training emails failed during this time but were automatically retried after the deployment was reverted.

   Mitigation and Remediation

   We found that the deployment removed a configuration value that another component still required. This was mitigated by reverting the release, and we confirmed recovery across all regions through direct checks and automated tests. We also reverted the original change in our source code and prepared a corrected version that includes a compatibility fix, so the original improvement can be reintroduced safely.

   Preventative Measures

   We're adding additional code verifications and testing for deployments.

Reported by KnowBe4 on their status page.

1. postmortemOct 20 · 12:48 UTC

   From Monday, October 12, 2026, at approximately 14:25 \(UTC\), to Tuesday, October 13, 2026, at around 15:07 \(UTC\), some customers in EU instances of Prevent and Protect had difficulty sending emails from Microsoft Outlook. Affected messages weren't sent, and the KnowBe4 Email Security add-in displayed the following error: "The add-in KnowBe4 Email Security is preventing this message from being sent." This issue was caused by an error in how an add-in version was prepared for release, which produced two slightly different copies of the same version. When Outlook loads the add-in, it downloads a set of files that must match. In EU instances, Outlook sometimes received files from both copies, so the add-in couldn't finish loading, and the message wasn't sent. Some devices also saved the failed download, so the error continued on those devices until a fix was applied. To resolve this issue, our team corrected the EU add-in files so that every customer receives a matching set. We then released an updated version of the add-in to all regions, which cleared any saved errors. The KnowBe4 Email Security add-in returned to normal performance by about 15:07 \(UTC\) on October 13, 2026. To prevent this type of issue in the future, we're changing how add-in versions are prepared and released, so that each version always produces the same set of files and every customer receives the same version at the same time. We're also changing the add-in so that failed downloads of add-in files aren't saved on devices. Lastly, we're adding alerts for add-in loading errors to detect them sooner. No data loss occurred as a result of this issue.

Reported by KnowBe4 on their status page.

1. postmortemOct 6 · 15:51 UTC

   Summary

   From Monday, September 28, 2026, at approximately 16:00 (UTC), to Wednesday, September 30, 2026, at around 12:35 (UTC), some customers experienced delayed mail flow and intermittent problems accessing US instances of Prevent and Protect.

   On September 28, our monitoring showed heavy network traffic, with spikes at the top of each hour. Some customers began experiencing intermittent problems accessing US instances of the Prevent and Protect consoles. Internal lookups that mail processing depends on started timing out, and messages were queued as a result. Customers reported the issue to our support team, and we posted a status page notice. Performance recovered later that day as US traffic decreased.

   On September 29, our engineers reviewed the previous day's data across the mail gateways, the internal services they depend on, and the network layer. Mail flow delays and console access issues returned briefly, then eased as network traffic declined, following the same pattern as September 28.

   On September 30, from approximately 12:19 to 12:35 (UTC), our team increased the capacity of the US network services while service continued. Their load dropped immediately after the change, and Prevent and Protect returned to normal performance by around 12:35 (UTC) on September 30, 2026. We monitored peak traffic over September 30 and October 1 and saw no recurrences.

   No data loss occurred as a result of this issue.

   Root Cause

   This issue was caused by a capacity limit in the network services that handle traffic for our US mail-processing services. During peak traffic, and especially during the hourly spikes, the network services couldn't process connections fast enough. This added delay to traffic passing through them and caused some connections to drop.

   That traffic included internal requests our mail gateways send to look up distribution list members, so the right policies are applied. These requests are subject to a strict time limit so that a slow lookup can't hold up mail processing indefinitely. When the network services delayed requests beyond that limit, they failed. As a result, mail processing slowed, and messages were queued.

   Two factors made the issue worse and harder to identify. First, some internal requests also used these network services and were affected by the capacity limit. Second, the network services began to degrade before their compute use reached levels that would normally signal a problem. Standard compute monitoring didn't catch the network saturation, so the cause took longer to identify. Finding a safe way to increase capacity without interrupting service also took time, which is why the fix was applied on September 30.

   Mitigation and Remediation

   We found that the US network services didn't have enough capacity to handle peak traffic. We've mitigated this by increasing their capacity in a way that kept service running throughout the change. After the capacity increase, the network services handled peak traffic without further issues.

   We also found that compute use alone wasn't a reliable sign of network saturation. We've expanded our monitoring to track network health indicators in addition to compute use.

   Finally, we found it was difficult to find where failing internal requests broke down. We've added logging that shows where internal requests fail and time out.

   Customer Impact

   During peak US traffic on September 28 and 29, some customers experienced delays in processing and delivering inbound and outbound mail through Prevent and Protect. Messages addressed to distribution lists may have been processed more slowly while lookups were retried. Some customers also had intermittent difficulty logging in to or using the US console.

   The impact eased each day as US traffic decreased, and queued mail was processed once capacity recovered. After the change on September 30, we saw no further degradation during peak traffic.

   Preventative Measures

   To prevent this type of issue in the future, we've increased the capacity of the US network services and expanded our monitoring to track network health indicators in addition to compute use. We've also added logging to improve visibility into failed internal requests.

Reported by KnowBe4 on their status page.

1. ResolvedOct 2 · 10:32 UTC

   This incident has been resolved.

Reported by KnowBe4 on their status page.

1. postmortemOct 6 · 15:51 UTC

   Summary

   From Thursday, September 24, 2026, at approximately 17:07 (UTC), to Friday, September 25, 2026, at around 09:16 (UTC), some customers experienced intermittent slowness, errors, and delayed mail flow for US instances of Protect and Prevent.

   On September 24, an unusually large spike in email traffic placed sustained load on the network services that handle traffic for our US mail processing. Our monitoring alerted us at approximately 17:07 (UTC). Some customers saw slow or intermittent access to Protect and occasional errors when loading pages or advanced options. Access was inconsistent, so some customers could use Protect while others saw errors. Mail processing through US instances slowed, and some messages were delayed or didn't appear in the moderation inbox when expected.

   Performance appeared to improve, so we moved our status page notice to Monitoring at 18:39 (UTC). The issue later recurred, and customers reported continued slowness to our support team. At 19:57 (UTC), we moved the notice back to Investigating and expanded it to also include Prevent as the impact became clearer.

   Our team monitored the affected systems while the elevated mail volume subsided. As the load on the network services returned to normal, mail processing recovered, and mail queues returned to normal overnight. We confirmed that Protect was loading normally and that test messages were being delivered again, and then continued monitoring. Protect and Prevent returned to normal performance by around 09:16 (UTC) on September 25, 2026.

   No data loss occurred as a result of this issue.

   Root Cause

   This issue was caused by an unusually large spike in email traffic. The increased volume placed a sustained load on the network services that handle traffic for our US mail processing.

   Those network services operate within fixed capacity limits, and their capacity couldn't be scaled up during the spike. Because they couldn't keep up with the surge, they slowed down, and some processing failed intermittently. As a result, a large volume of mail was queued in US mail processing, some pages and actions in Protect returned errors, and some messages were delayed or didn't appear in the moderation inbox when expected.

   Two factors made the issue harder to manage. First, because access failed intermittently, some checks succeeded while customers continued to see errors, making recovery harder to confirm. Second, our monitoring showed peak load on the network services more clearly than the current load, which made it harder to judge in real time whether the load was still elevated.

   Mitigation and Remediation

   We found that the network services handling US mail processing had reached their fixed capacity limits during the traffic spike, and their capacity couldn't be increased during the event. Our team monitored the affected systems while the elevated mail volume subsided and network service load returned to normal. Mail processing then recovered, and queued mail was processed.

   Because the issue had previously appeared to improve before recurring, we confirmed recovery in several ways before resolving the incident. We confirmed that Protect was loading normally, that test messages were being delivered through US mail processing, and that support requests were progressing normally. We also monitored overnight and received no new reports from customers on US instances.

   Customer Impact

   Some customers had slow or intermittent access to Protect, and occasionally saw errors when loading pages or advanced options. Mail processing through US instances was delayed for Protect and Prevent. In some cases, messages were delayed or didn't appear in the moderation inbox when expected.

   Customers whose mail is encrypted, decrypted, or moderated as it passes through our US mail processing were more likely to be affected. Most Protect customers using the add-in weren't affected by the mail processing delays. The Protect and Prevent consoles remained available, although some console actions were affected by the slower mail processing.

   Preventative Measures

   To prevent this type of issue in the future, we're improving our monitoring of unusual increases in email volume. We're also improving our monitoring to show the current load on our network services, not only peak values. In addition, we're evaluating ways to scale our network capacity more dynamically during sudden traffic spikes, and assessing longer-term improvements to the network infrastructure that handles US mail processing.

   We've also since increased the capacity of US network services in response to a later incident.

Reported by KnowBe4 on their status page.

1. postmortemOct 20 · 15:53 UTC

   On Wednesday, September 23, 2026, from approximately 08:15 to 11:18 \(UTC\), some customers experienced slowness and failed form submissions in UK instances of Secure Webforms. This issue was caused by database performance limits in the UK production environment following a recent change. A database setting also caused some queries to run inefficiently. As a result, response times increased and some form submissions failed. To resolve this issue, our team temporarily paused a background synchronization task to reduce load, adjusted the database settings to improve query efficiency, and increased storage throughput. Secure Webforms returned to normal performance by 11:18 \(UTC\). To prevent this type of issue in the future, we’re adding additional monitoring for database bandwidth. We’re also improving the efficiency of the background synchronization task. No data loss occurred as a result of this issue.

Reported by KnowBe4 on their status page.

1. postmortemOct 8 · 13:45 UTC

   On Monday, September 21, 2026, from approximately 08:55 until Tuesday, September 22, 2026, at approximately 14:30 \(UTC\), some users experienced errors and were unable to send emails via Web Access \(Secure Message Portal\) in the UK region.  This incident was caused by a configuration error during a database update, which led to intermittent service disruption. To resolve this issue, we reverted the configuration, and Web Access returned to normal performance by 14:30 \(UTC\) on September 22, 2026. To prevent this type of issue in the future, we are reviewing and updating our database configuration.  No data loss occurred as a result of this issue.

Reported by KnowBe4 on their status page.

1. postmortemOct 8 · 14:07 UTC

   Summary

   On September 17, 2026, a background job archived a large user group and ran as a single unbounded database transaction. This transaction locked a large number of user rows, causing US region logins to time out because they write to the same table. After hitting a platform execution limit, the job was terminated and automatically retried five times in 15-minute intervals, resulting in a 75-minute outage. To resolve the issue, we deployed a fix that decoupled logins from the lock. We then deployed a follow-up fix for a regression introduced by the login change, as well as for the archival job.

   What Happened

   A prior change moved the "archive all users" action from a synchronous, time-out web request into an unbounded background job. When a large user group was archived, this triggered the new background job. The job locked all affected user rows in a single transaction, performing per-user writes and a synchronous external service call before committing. Login attempts, which write to the same table, were queued behind the locked job and timed out for US users.

   When the job reached a 15-minute platform execution limit, it was terminated mid-transaction, the transaction rolled back, and the job was redelivered to the queue. This cycle occurred five times, resulting in a 75-minute outage. When failures were detected internally at 22:35 (UTC), an incident was declared, and an engineer was paged to respond. The mitigation to decouple the write action from the login flow was deployed at approximately 23:44 (UTC).

   The mitigation itself caused a regression: any user in any region who had previously mistyped a password could no longer log in because resets for failed login attempts tried to write to a read-only database. By 09:17 (UTC) on September 18, 2026, the fix for this regression was deployed across regions. The original US-only archive job issue recurred twice more between 13:45 and 14:05 (UTC) before a permanent fix was deployed at approximately 15:00 (UTC).

   The incident was fully resolved by 19:12 (UTC) on September 18, 2026.

   Root Cause

   An unbounded, single-transaction background job for large-group user archival held exclusive locks on many affected rows, blocking the login process's own row write. A prior change to move this operation off a web request timeout and into a background job removed the only remaining limit on its duration. The platform's 15-minute execution limit then triggered automatic kill-and-redeliver cycles, repeating the lock contention five times and resulting in a 75-minute outage window.

   Findings and Mitigations

   We found that the unbounded archive transaction locked a large number of affected user rows, blocking other table writes. This was mitigated in the short term by decoupling the login process's write dependency from the affected table, and then fixed in the long term by correcting the archive job. We also found that uncoordinated worker concurrency amplifies lock contention. This was mitigated in the short term by manually reducing worker concurrency. The long-term fix to stagger and serialize these jobs is planned. Finally, we found that the login-fix regression caused all-region login failures for users with prior failed attempts because writes were rejected by a read-only database. This was fixed via a targeted code change that was deployed and verified across all regions.

   Preventive Measures

   We’re improving our large-group archival operations by bounding them, processing them in small, committed batches, and keeping external calls outside the database transaction. We’re also adding a change-check to a specific background worker to skip writes that don't change anything. Finally, we’re adding direct alerts for database transaction age and purging delays, in addition to periodic indirect checks.

   Customer Impact and Recovery

   Login failures lasted approximately 75 minutes in the US region. Two brief, US-only recurrences occurred the next morning and were cleared manually within minutes. Some admin actions, such as user management, were also affected. Partner logins were unaffected. The secondary regression lasted around 10 hours and affected all regions for users who had previously failed a login attempt. By the afternoon of September 18, full recovery was confirmed across all regions, and the status page was marked as resolved.

   No data loss occurred as a result of this issue.

   Conclusion

   A single unbounded background transaction, combined with an automatic retry mechanism, caused a login outage in the US region. The specific triggering job and its retry behavior have been fixed, and login no longer depends on the contended write path. A same-incident regression and two minor recurrences were each resolved within hours. Remaining preventive work, including worker coordination and database transaction monitoring, is tracked above.

Reported by KnowBe4 on their status page.

1. postmortemOct 5 · 21:16 UTC

   Summary

   On September 17, 2026, a background job archived a large user group and ran as a single unbounded database transaction. This transaction locked a large number of user rows, causing US region logins to time out because they write to the same table. After hitting a platform execution limit, the job was terminated and automatically retried five times in 15-minute intervals, resulting in a 75-minute outage. To resolve the issue, we deployed a fix that decoupled logins from the lock. We then deployed a follow-up fix for a regression introduced by the login change, as well as for the archival job.

   What Happened

   A prior change moved the "archive all users" action from a synchronous, time-out web request into an unbounded background job. When a large user group was archived, this triggered the new background job. The job locked all affected user rows in a single transaction, performing per-user writes and a synchronous external service call before committing. Login attempts, which write to the same table, were queued behind the locked job and timed out for US users.

   When the job reached a 15-minute platform execution limit, it was terminated mid-transaction, the transaction rolled back, and the job was redelivered to the queue. This cycle occurred five times, resulting in a 75-minute outage. When failures were detected internally at 22:35 (UTC), an incident was declared, and an engineer was paged to respond. The mitigation to decouple the write action from the login flow was deployed at approximately 23:44 (UTC).

   The mitigation itself caused a regression: any user in any region who had previously mistyped a password could no longer log in because resets for failed login attempts tried to write to a read-only database. By 09:17 (UTC) on September 18, 2026, the fix for this regression was deployed across regions. The original US-only archive job issue recurred twice more between 13:45 and 14:05 (UTC) before a permanent fix was deployed at approximately 15:00 (UTC).

   The incident was fully resolved by 19:12 (UTC) on September 18, 2026.

   Root Cause

   An unbounded, single-transaction background job for large-group user archival held exclusive locks on many affected rows, blocking the login process's own row write. A prior change to move this operation off a web request timeout and into a background job removed the only remaining limit on its duration. The platform's 15-minute execution limit then triggered automatic kill-and-redeliver cycles, repeating the lock contention five times and resulting in a 75-minute outage window.

   Findings and Mitigations

   We found that the unbounded archive transaction locked a large number of affected user rows, blocking other table writes. This was mitigated in the short term by decoupling the login process's write dependency from the affected table, and then fixed in the long term by correcting the archive job. We also found that uncoordinated worker concurrency amplifies lock contention. This was mitigated in the short term by manually reducing worker concurrency. The long-term fix to stagger and serialize these jobs is planned. Finally, we found that the login-fix regression caused all-region login failures for users with prior failed attempts because writes were rejected by a read-only database. This was fixed via a targeted code change that was deployed and verified across all regions.

   Preventive Measures

   We’re improving our large-group archival operations by bounding them, processing them in small, committed batches, and keeping external calls outside the database transaction. We’re also adding a change-check to a specific background worker to skip writes that don't change anything. Finally, we’re adding direct alerts for database transaction age and purging delays, in addition to periodic indirect checks.

   Customer Impact and Recovery

   Login failures lasted approximately 75 minutes in the US region. Two brief, US-only recurrences occurred the next morning and were cleared manually within minutes. Some admin actions, such as user management, were also affected. Partner logins were unaffected. The secondary regression lasted around 10 hours and affected all regions for users who had previously failed a login attempt. By the afternoon of September 18, full recovery was confirmed across all regions, and the status page was marked as resolved.

   No data loss occurred as a result of this issue.

   Conclusion

   A single unbounded background transaction, combined with an automatic retry mechanism, caused a login outage in the US region. The specific triggering job and its retry behavior have been fixed, and login no longer depends on the contended write path. A same-incident regression and two minor recurrences were each resolved within hours. Remaining preventive work, including worker coordination and database transaction monitoring, is tracked above.

Reported by KnowBe4 on their status page.

1. postmortemOct 16 · 20:36 UTC

   Summary

   During a routine deployment on August 26, 2026, an infrastructure management framework unexpectedly removed both the primary message-processing queues and the secondary queues that hold messages that could not be processed. This resulted in a temporary interruption of incoming email delivery to PhishER customers' inboxes.

   No data loss occurred as a result of this issue.

   Root Cause

   On August 24, 2026, an internal configuration change left live configuration records out of sync with our code. A later code update did not revert those records.

   When a routine production deployment ran on August 26, 2026, the active configuration code did not contain the relocated resource records. As a result, the deployment treated the active queues as unmanaged resources and removed them.

   Mitigation and Remediation

   Automated monitoring alerted us to a drop in message processing, and an incident response team was assembled.

   We found that the queues had been deleted. This was mitigated in the short term by rerunning the affected production jobs to recreate the missing queues.

   We then verified that real-time email processing had resumed. Finally, we replayed email payloads held in cloud storage during the outage into customers' inboxes.

   Timeline

   All times are in UTC.

   • August 24, 2026, 18:41: An internal configuration change wrote updated resource records into live production configuration.
   • August 26, 2026, 15:15: A production deployment removed the inbound queues and the queues that hold messages that could not be processed in production, halting email delivery into customers' inboxes.
   • August 26, 2026, 15:59: Our automated monitoring alerted the engineering team to zero messages being processed, and an incident response team was paged.
   • August 26, 2026, 16:01: We confirmed the queues had been deleted and verified that incoming email payloads were still stored safely.
   • August 26, 2026, 16:15: We identified the configuration mismatch as the primary root cause.
   • August 26, 2026, 16:25: Our team reran the affected production jobs, and the queues were recreated.
   • August 26, 2026, 18:44: Our automated replay finished delivering all backlogged emails to customers' inboxes.

   Preventative Measures

   To prevent this type of issue in the future, we've removed a live-configuration step from pre-merge checks and separated deployment planning from production changes. We've also added automated checks to confirm queue health after releases are complete.

   Customer Impact

   From 15:15 to 16:25 (UTC) on August 26, 2026, incoming emails submitted to PhishER were not processed into customer inboxes.

   Inbound email payloads were stored safely in cloud storage throughout the disruption. After the queues were restored, an automated replay delivered held emails to customers' inboxes. All backlogged messages were delivered by 18:44 (UTC).

   No customer email data was lost as a result of this issue.

Reported by KnowBe4 on their status page.

1. postmortemSep 30 · 20:06 UTC

   On Tuesday, August 18, 2026, from approximately 14:10 to 14:55 (UTC), customers using the Phish Alert Button (PAB) with the Send Us a Copy feature enabled received non-delivery reports for the Send Us a Copy mailbox.

   This issue was caused by a configuration change to the shared email routing infrastructure. This change renamed a messaging component used by both our staging and production environments. Because a production rule depended on this shared component, the rename caused delivery failures to the Send Us a Copy mailbox and triggered non-delivery reports. To resolve this issue, our team updated the affected configuration to reference the correct component, and the Send Us a Copy feature returned to normal performance by 14:55 (UTC).

   To prevent this type of issue in the future, we are separating email routing configurations for our staging and production environments. We are also reconciling legacy configurations into our current infrastructure management standards.

   No data loss occurred as a result of this issue.

Reported by KnowBe4 on their status page.