Kustomer Incident History

# **Summary** On July 30, 2026, the draft text editor on Kustomer’s Timeline product experienced degraded functionality in certain user workflows. Impact included sporadic cursor behavior, text erasure, and issues with copying and pasting text and shortcuts. # **Root Cause** This issue was introduced by a change to the editor that was not fully caught before release. Due to misalignment in our QA process our pre-release validation did not adequately cover the real-world editing patterns affected by this change. Furthermore, this misalignment contributed to a delay in Kustomer’s understanding of the incident's resolution status. # **Timeline** ## **Jul 30, 2026** **11:12 AM ET:** The editor change was deployed **12:27 PM ET:** Kustomer Technical Support escalated the incident to Kustomer’s OnCall process, notifying Engineering immediately. **12:34 PM ET:** Kustomer Engineering identified the issue and completed a rollback **12:51 PM ET:** Internal testing incorrectly confirms that the issue has been fully resolved, due to some customers reporting the issue no longer presented itself **1:29 PM ET:** Continued reports of issue are received from customers who did not receive the full resolution rollout **1:34 PM ET:** Discovery made that  the rollback did not fully deploy to resolve the issue. Kustomer Engineering makes additional change to ensure resolution **2:01 PM ET:** Corrective change deployed, and fix is confirmed # **Lessons/Improvements** Kustomer Engineering maintains a robust CI/CD process, and a multi-environment release process designed to prevent issues of this nature. As part of our continual investment in these areas, we have identified the following action items: * Ensure that our internal pre-production environments align with our customer-facing experience, including but not limited to the editor experience, so that issues of this nature are caught earlier in development * Complete an audit of our automated CI/CD process and test coverage of major features, removing any gaps that are identified

Kustomer has resolved an event affecting text editor that may cause issues when typing and adding content such as Shortcuts. To resolve this issue, our team has rolled back to a previous version. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.

Kustomer has implemented an update to address an event affecting text editor that may cause issues when typing and adding content such as Shortcuts. Refreshing your browser will fully resolve the issue. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.

Kustomer is aware of an event affecting text editor that may cause issues when typing and adding content such as Shortcuts. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support via support@kustomer.com for any further questions or updates.

## **Summary** On June 25, 2026, some customers were unable to send outbound WhatsApp replies from within Kustomer. The issue affected reply sending for a subset of WhatsApp channel configurations, which disrupted agent workflows and prevented some automated outbound messages from being sent through the same path. The issue was identified and resolved the same day. Service was fully restored, and the platform is operating normally. ## **Root cause** A change released earlier that day introduced stricter validation in the outbound WhatsApp reply flow. For a subset of supported channel configurations, valid sender values were incorrectly rejected before messages were sent. This caused reply attempts in those configurations to fail. The issue was limited to specific WhatsApp channel setups and did not affect all WhatsApp traffic equally. ## **Timeline** * **June 25, 2026, approximately 22:12 UTC** — Reports began coming in that outbound WhatsApp replies were failing for some customers. * **Shortly after detection** — Investigation confirmed the issue was tied to a recently released validation change in the outbound reply flow. * **June 26, 2026, approximately 01:07 UTC** — A fix was deployed and reply sending was restored. ## **Lessons and improvements** * Validation changes for messaging flows now require broader test coverage across supported channel configuration variants before release. * Additional safeguards are being added to reduce the risk of valid outbound requests being rejected. * Monitoring and regression checks around outbound messaging paths are being strengthened to detect similar issues more quickly.

Kustomer has resolved the issue affecting PROD 1 that impacted the ability to send outbound messages through MessageBird. Our team has verified that the fix has been successfully deployed and service has been restored. If you continue to experience issues sending outbound MessageBird messages, please contact Kustomer Support at support@kustomer.com We apologize for the disruption and appreciate your patience while we worked to resolve the issue.

Kustomer has identified and implemented a fix for the issue affecting PROD 1 that impacted the ability to send outbound messages through MessageBird. Our team is actively monitoring the platform to ensure the fix remains effective and that service has fully recovered. If you continue to experience issues sending outbound MessageBird messages, please contact Kustomer Support at support@kustomer.com. We will provide another update once monitoring is complete or if there are any significant developments. Thank you for your patience.

Kustomer has identified and implemented a fix for the issue affecting PROD 1 that impacted the ability to send outbound messages through MessageBird. Our team is actively monitoring the platform to ensure the fix remains effective and that service has fully recovered. If you continue to experience issues sending outbound MessageBird messages, please contact Kustomer Support at support@kustomer.com. We will provide another update once monitoring is complete or if there are any significant developments. Thank you for your patience.

Kustomer has identified the cause affecting PROD 1 that is impacting the ability to send outbound messages through MessageBird. Please expect another update within the next 30 minutes, as we reach a resolution. If you have any additional questions or concerns, please reply to this conversation or contact Kustomer Support at support@kustomer.com. Thank you for your patience while we work to resolve this issue.

Kustomer is aware of an event affecting PROD 1 that may affect the ability to send outbound messages for MessageBird. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support at support@kustomer.com for any further questions or updates.

## **Summary** Between May 13 and May 14, 2026, Kustomer Chat experienced a service event that affected chat availability and performance for a subset of customers. During this period, some customers may have seen intermittent failures, elevated error rates, or degraded behavior in chat-related settings and runtime flows. The event was mitigated through service rollback, additional capacity, and targeted configuration changes. Service health returned to normal after these actions were completed. ## **Root cause** The event was caused by a combination of reduced service capacity during a deployment rollback and higher-than-expected traffic through chat-related request paths. Under those conditions, the affected chat service became unstable and restarted repeatedly, which reduced available capacity further and increased customer-facing errors. Our investigation also identified specific high-volume request patterns that increased memory pressure during the event. We addressed those patterns with caching, traffic protections, and capacity changes. ## **Timeline** * May 13, 2026, early afternoon ET: We detected elevated instability in the chat service and began incident response. * May 13, 2026, afternoon ET: We rolled back affected changes, increased infrastructure capacity, and stabilized dependent services. * May 13, 2026, evening ET: We continued monitoring after initial mitigation and investigated recurring memory pressure. * May 14, 2026: We deployed additional mitigations, including higher minimum service capacity and request-path protections. * Following the mitigation deployments, service health returned to normal and remained stable. ## **Lessons and improvements** We completed several improvements to reduce the likelihood of recurrence: * Increased minimum service capacity to provide more headroom during deployments and recovery. * Added caching for high-volume chat settings requests. * Hardened URL processing behavior with stricter filtering, failure caching, and concurrency limits. * Added improved memory telemetry to speed up detection and diagnosis of similar issues. * Continued follow-up work on deployment recovery procedures and service safeguards for cross-service rollbacks. ## **Current status** The mitigations above have been deployed, and the affected chat service is operating normally.

Kustomer has resolved an event affecting Chat on Prod 1 that caused messages to fail and assistants to not follow the configured flow. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support if you have additional questions or concerns.

Kustomer has implemented an update to address an event affecting Chats on Prod 1 that caused messages to fail and assistants to not follow the configured flow. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support if you have additional questions or concerns.

Kustomer is aware of an event affecting Chats that may cause messages to fail and assistants to not follow the configured flow. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, please reach out to Kustomer Support for any further questions or updates.

## Summary On July 25, 2026, some customers experienced elevated platform latency affecting messaging, conversation updates, routing, voice, and other customer-service workflows. Customers may have seen messages remain in a sending state, delayed inbound or outbound messages, slower page and API responses, delayed routing or assignment, and intermittent voice-call delays. The incident was caused by an unusually large burst of background data-processing activity. This created a sudden increase in demand on shared platform infrastructure. Automatic scaling added capacity, but it could not absorb the burst quickly enough to prevent degradation in dependent workflows. We restored service by increasing available capacity, reducing the rate of the initiating workload, and carefully processing delayed work while monitoring platform health. ## Impact * **Customer effect:** Intermittent platform latency; delayed inbound and outbound messaging; slower conversation and API updates; delayed routing or assignment; and intermittent voice-call delays * **Duration:** Customer-visible degradation began at approximately 5:22 PM ET. Service initially recovered at approximately 7:31 PM ET, but degradation later recurred. Broad platform performance was restored and the incident was resolved at 10:30 PM ET. * **Scope:** The incident affected a subset of customers within one production environment. We did not identify corresponding customer-facing degradation in other production environments. ## Timeline * **Approximately 5:22 PM ET:** Processing latency and message backlogs began increasing. * **6:51 PM ET:** The incident response team began coordinated investigation and recovery. * **Approximately 7:00–7:23 PM ET:** We identified affected processing paths and began carefully recovering delayed work at controlled rates. * **7:31 PM ET:** Additional capacity had come online, customer-facing performance had materially improved, and the incident was initially resolved while monitoring continued. * **8:28 PM ET:** The incident was reopened after renewed reports of intermittent degradation. * **Approximately 8:38 PM ET:** Investigation confirmed that messaging, routing, channel, and voice symptoms shared the same underlying platform dependency. * **Approximately 9:52 PM ET:** We reduced the rate of the initiating background workload to protect customer-facing traffic. * **10:08 PM ET:** Monitoring confirmed that workload pressure had fallen substantially and platform health continued improving. * **10:30 PM ET:** After continued monitoring confirmed recovery, the incident was resolved. * **After resolution:** Recovery of remaining delayed work continued at controlled rates under monitoring. ## Root cause A burst of high-volume background data-processing activity generated more downstream work than shared platform infrastructure could safely absorb over a short period. Automatic scaling responded and added capacity, but capacity was added incrementally and did not come online quickly enough for the size and speed of the burst. While the platform was catching up, increased request latency and timeouts affected customer-facing workflows that depended on the same infrastructure. ## Resolution We restored platform performance by: * Increasing available capacity and improving the rate at which additional capacity could be added. * Reducing the rate of the initiating background workload. * Recovering delayed work at controlled rates to avoid creating another traffic spike. * Continuing monitoring after customer-facing performance returned to expected levels. ## Preventative actions We are taking the following actions to reduce the likelihood and impact of recurrence: * Add stronger limits and backpressure controls for high-volume background operations. * Improve the speed at which shared infrastructure scales during sudden traffic increases. * Improve isolation between background processing and latency-sensitive customer workflows. * Expand alerting for rapid backlog growth, resource pressure, and unusual workload patterns. * Strengthen controlled recovery procedures for delayed work. * Expand burst-load and failure-recovery testing for shared platform services. ## Current status Platform performance returned to expected levels, and the initiating workload remained controlled. We continued monitoring service health and the recovery of delayed work after resolution.

Kustomer has resolved an event affecting platform events in PROD1 orgs that caused delays on event-based data. After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.

Kustomer has implemented an update to address an event affecting platform events in PROD1 orgs that caused delays on event-based data. Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer Support at support@kustomer.com if you have additional questions or concerns.

Kustomer continues to work on the issue affecting platform events in PROD1 orgs that may cause delays on event-based data. Our team is actively working to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at support@kustomer.com for any further questions or updates.

Kustomer has identified an event affecting platform events in PROD1 orgs that may cause delays on event-based data. Our team is currently actively working to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at support@kustomer.com for any further questions or updates.

Kustomer has identified an event affecting platform events in PROD1 orgs that may cause delays on event-based data. Our team is currently working to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer Support at support@kustomer.com for any further questions or updates.