Historical record of incidents for Landbot
Report: "Technical issues affecting our Services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring results.
We are currently experiencing technical issues that are impacting our bots and overall platform functionality.Our engineering team is investigating the root cause and working on a fix with the highest priority. We’re committed to restoring full service as quickly as possible and minimizing the impact on our users.Thank you for your patience and understanding.
Report: "Technical issues affecting our Services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring results.
We are currently experiencing technical issues that are impacting our bots and overall platform functionality. Our engineering team is investigating the root cause and working on a fix with the highest priority. We’re committed to restoring full service as quickly as possible and minimizing the impact on our users. Thank you for your patience and understanding.
Report: "Technical Incident Affecting Chatbots"
Last updateWe experienced an intermittent issue affecting our chatbots between 7.30 AM CET and 12.30 PM CET . While not all conversations were impacted, some users may have encountered disruptions during interactions. The issue has been identified and resolved. We apologize for any inconvenience and appreciate your patience.
Report: "Incident affecting App and Chatbots"
Last updateThis incident has been resolved. Thank you for your patience while we were working on this.
A fix has been implemented and we are monitoring results.
The issue has been identified and a fix is being implemented
We are continuing to investigate this issue.
We are currently experiencing an issue causing downtime for our App and Chatbot services. Our team is actively working to identify the root cause and restore normal functionality as soon as possible.
Report: "Google Sheets and Airtable GET Requests Service Disruption"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing an issue affecting GET requests for both Airtable and Google Sheets integrations. Users may encounter difficulties when attempting to retrieve data from these services.
Report: "WhatsApp Message Template Sending Issue"
Last updateThe issue affecting the sending of WhatsApp message templates with buttons has been resolved. We appreciate your patience and apologize for any inconvenience caused.
A fix has been implemented and we are monitoring results.
The issue has been identified and a fix is being implemented. We appreciate your patience and will provide further updates as we make progress.
We are currently experiencing an issue affecting the sending of WhatsApp message templates that include buttons. Our team is investigating the root cause and working to resolve it as quickly as possible. We apologize for any inconvenience and will provide updates as soon as we have more information. Thank you for your patience.
Report: "Technical incident affecting our app"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently experiencing issues with our app where users may notice slow loading times or occasional failures to load. Our team is actively investigating the cause of this problem.
Report: "Incident affecting our Services"
Last updateThis incident has been resolved.
A fix has been implemented and we're monitoring it.
The issue has been identified and is being worked on.
Our service is being affected at the moment, we're investigating this issue.
Report: "Degraded performance in Web bots"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating a degraded performance in web bots
Report: "Incident affecting WhatsApp integration"
Last updateThis incident has been resolved. Thank you for your patience and understanding during this time.
A fix has been implemented and we are monitoring results.
The issue has been identified and a fix is being implemented.
We are currently experiencing an incident affecting WhatsApp integration, and WhatsApp bots are unresponsive at this moment. Our technical team is actively investigating the root cause of the issue, and we will provide updates as soon as more information becomes available. Thank you for your understanding.
Report: "Incident affecting sending of WhatsApp templates"
Last updateThe incident affecting WhatsApp templates has now been fully resolved. We have completed the rollback process for template ID changes that impacted API-based sending. This means that previous template IDs have been restored, and the key is now back to "template_id", as it was before (not "template_uuid"). Thank you for your cooperation and support throughout the process.
Our team has implemented solutions to address the issues affecting WhatsApp templates in the app (sending, requesting...). Also, we initiated the rollback process for template ID changes that impacted API-based sending. While this rollback is underway, we'll notify you as soon as the process is complete and ensure everything is fully restored. We deeply appreciate your patience and understanding throughout this process.
We've made progress in fixing some of the issues affecting WhatsApp templates. While some solutions have been implemented, additional fixes are still required for complete resolution. We will continue to provide updates as we make further progress. Thank you for your continued support.
Our team is actively working on a fix to address the ongoing technical issue affecting the sending of templates in our WhatsApp integration. We understand the inconvenience this is causing and appreciate your patience as we work to restore normal functionality. Resolving this issue remains our top priority, and we will continue to provide updates as soon as more information becomes available. Thank you for your understanding and continued support.
We're currently experiencing a technical issue impacting the sending of templates in our WhatsApp integration. Our team is actively investigating the issue and working towards a resolution. During this time, you may encounter errors when attempting to send templates via WhatsApp both in-app and via API. We apologize for any inconvenience this may cause and appreciate your patience as we work to restore normal functionality. Rest assured, we are prioritizing this issue and will provide updates as soon as more information becomes available. Thank you for your understanding.
Report: "Incident with variable functionality"
Last updateWe have experienced an incident that affected variable functionality across all channels. A fix was implemented and the incident was resolved at 23.00 CET on 20/01/2024. Our teams are actively investigating the root cause to prevent any recurrence. We sincerely apologize for any inconvenience this incident may have caused and appreciate your understanding as we work to improve our systems.
Report: "Incident affecting our Services"
Last updateWe experienced an incident during migration that affected all of our Services. Fix has been implemented to resolve the issue, but some components have experienced up to a 2-hour downtime. We apologize for the inconvenience this has caused.
Report: "Outage affecting our Services"
Last updateThis incident has been resolved
A fix has been implemented and we are monitoring results
The issue has been identified and fix is being implemented
We are currently investigating the issue.
Report: "Technical issues affecting our Services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring results
The issue has been identified and a fix is being implemented
We are facing technical issues affecting the performance of our bots, and app. We are investigating it and working to minimize the effects with maximum priority.
Report: "Incident affecting Web bots on iOS and macOS devices"
Last updateWe have experienced an incident that affected the visualization of Web bots on iOS and macOS devices. A fix has been implemented for this incident and we are monitoring the results.
Report: "Outage affecting our Services"
Last updateWe have experienced an outage affecting our Services until 1.15 AM (November 8). The incident has been resolved and we are investigating the cause of the incident.
Report: "Technical issue affecting our Services"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Builders failed to load with 500 error message, and the performance of the bots was affected by delays. A solution was implemented and we are monitoring the results.
Report: "Global WhatsApp Outage"
Last updateThis incident has been resolved.
The global WhatsApp outage affected our WhatsApp Business API integration. We are monitoring the issue to make sure our services are restored.
Report: "Degraded performance in WhatsApp bots (360 Dialog)"
Last updateThis incident has been resolved.
360dialog has already fixed the issue and is monitoring the situation: https://status.360dialog.com/
Our official WhatsApp provider (360 Dialog) is experiencing degraded performance in their messaging services, which is also impacting our Whatsapp bots. Please refer to https://status.360dialog.com/ for updates on the issue's status. We apologize for the inconvenience and thank you for your patience.
Report: "Builder failing to load existing bots"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Downtime during component migration"
Last update# Incident summary Between the hour of 16:23 and 16:46 CEST on the 20th of June, a cascading failure during a Redis migration to version 6 affected some deployments, resulting in a service outage. # Impact This incident affected some customers, who experienced a downtime in the application for 23 minutes. # Root Causes During a Redis migration, a bug in the GitLab CI pipeline was identified. This bug truncates the trailing 0 of the container image tag, so it deployed a release version `1.2` instead of `1.20`, causing a CrashLoopBackOff error for one of our Deployment containers. The environment variables for the Deployments were moved to the new Redis 6 instance when the Deployments were still in CrashLoopBackOff. # Trigger One of our Deployment’s ConfigMap was modified, changing the Redis server environment variable to the new Redis 6 instance. The restart of these Deployments updated their ConfigMaps, when one of them were still in a CrashLoopBackOff status, causing a cascading failure that affected other Deployments and triggering the incident. # Detection The Redis 6 queue started to fill up with tasks ahead of time, and the application stopped working. # Timeline **2022-06-20 \(all times are CEST\)** * **15:29** - Created new Redis 6 instance * **15:44** - Created ConfigMap and Secret for auxiliary Deployments, containing environment variables pointing to Redis 6 instance * **15:59** - _INCIDENT BEGINS_ Created auxiliary Deployments, that were deployed with release version `1.2` instead of `1.20` * **16:17** - Updated environment variables, pointing to Redis 6 instance * **16:19** - Redis 6 queue began to fill up progressively * **16:23** - _OUTAGE BEGINS_ The application stopped working * **16:39** - Deleted auxiliary Deployments * **16:46** - Redis 6 queue reached 309K length * **16:46** - _OUTAGE MITIGATED_ Environment variables restored to previous Redis 5 instance * **16:46** - _OUTAGE ENDS_ All services restored and working correctly * **17:08** - Followed procedure to continue with the Redis 6 migration * **17:10** - After running GitLab CI pipeline to create auxiliary Deployments, container images were manually set to release `1.20` * **17:11** - Redis 6 queue started to decrease * **17:15** - _INCIDENT ENDS_ Redis 6 queue emptied * **17:18** - Original Deployments moved to Redis 6 instance * **17:31** - Deleted auxiliary Deployments * **17:44** - Redis 6 migration completed # Action Items as result of Postmortem * Investigate GitLab CI bug * Update Redis migration playbook * Recompose internal metrics during the incident
This incident has been resolved
A fix has been implemented and we are monitoring results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
There is an ongoing incident affecting the Landbot app. The bots are working correctly. We are investigating and working on the issue with maximum priority.
Report: "Incident in variables storage"
Last updateThe incident has been resolved, all bots should be fully operational again.
While we work on a fix, all Web bots (landbots) are currently disabled. We apologize once again for the inconvenience.
We are currently facing an issue storing data in variables from some bots. The issue happens across all channels, and even though the bots are active, they are not storing any data inside the variables. Our team has already tracked the issue and is working on a solution to be deployed asap. We apologize for the inconvenience and thank you for your understanding.
Report: "Landbot Domain Suspended"
Last updateOn 05/05 [Landbot.io](http://Landbot.io) suffered a major incident with an impact on all features and customers. The root cause of this incident was a suspended domain of [landbot.io](http://landbot.io) due to the misuse of our bots in phishing activities. ### **Main activities** held and their timelines: ``` 04/05 (01:34) CEST - Our provider send us an email informing us that some of our bots were being used in phishing activities and that if we did nothing our domain was going to be suspended. 04/05 (15:50) CEST - By this time, we told our provider that we had already stopped those bots. 05/05 (00:50) CEST - Our provider informed us again of more bots being used for phishing attacks. 05/05 (01:40) CEST - Our Domain was suspended. 05/05 (07:30) CEST - We contacted our provider for support but we did not get a prompt action because their department was in another timezone and they did not had 24x7 support. 05/05 (08:40) CEST - We were able to reestablish some of the bots in a new existing domain (with some limitations) 05/05 (11:00) CEST - We bought a temporary domain and started moving all the bots to the new domain. (...) We continued to try to talk with our provider on multiple channels (LinkedIn, Phone) to see if we could get a prompt response. 05/05 (13:00) CEST - We finished moving all bots to the new domain. 05/05 (15:00) CEST - We were finally able to get a call to our provider in Az to see if they could unblock our domain. 05/05 (16:10) CEST - After 1 hour on a call, we were finally notified that the issue has been fixed. 05/05 (16:40) CEST - We had restored full service. ``` ### Main Actions to Take: * Review our On-Call program. There were a couple of hours between the incident and us acknowledging it. This was due to us relying on email for alarmistic and we did not receive emails due to the domain suspension. * Review the Incident Management process. There are some improvements we can do to speed up our process, mostly regarding the organization of the incident. * Proactive identification of abuse use cases and finding protection measures for bots not being used for malicious purposes. * Find a domain provider with better worldwide support \(24x7\) * Review our abuse policies & terms of use We really want to thank our customers for their empathy and support during this incident.
Landbot.io suffered a major incident with an impact on all features and customers. The root cause of this incident was a suspended domain of landbot.io due to the misuse of our bots in phishing activities.
Report: "Incident causing a few seconds of delay when starting bot service (web bots)"
Last updateThis incident has been resolved.
We have an incident in progress that is affecting the "welcome" service of web bots. Due to this incident, the first interaction of web bots is taking longer than usual to initiate the conversation (a couple of seconds). This means that the web bots take longer to start, but besides this small lag, the app and all bots are functional and working normally. The incident started on Sat 7th at 17:18 (CET) with lags about 20 or 30 seconds in the initial interaction. A fix was implemented, and since then the lag should be about just a couple of seconds, as we are currently experiencing. Our team has already tracked the cause of the issue, and we are working on a solution.
Report: "Issue in Calendly Integration"
Last updateThis incident has been resolved.
We have tracked an issue with the Calendly block, which is not allowing users to book meetings. We are currently investigating this issue.
Report: "Technical issues affecting the performance of bots"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are facing technical issues, affecting the performance of our bots and builder. Our team is working to minimise the effects with maximum priority.
Report: "Technical issues affecting the performance of bots"
Last updateThe incident has been resolved.
We are facing technical issues with our bots due to an operation in our database that is taking longer than expected to be completed. If there are recent changes in new bots, it may take around 50 minutes to be displayed. New bots may not be started until that period is completed. Our team is working to minimize the effects with maximum priority. We apologise for the inconvenience and thank you for your comprehension.
Report: "Technical issues affecting the performance of bots"
Last updateThis incident has been resolved.
We are facing technical issues with our bots due to an operation in our database that is taking longer than expected to be completed. If there are recent changes in new bots, it may take around 50 minutes to be displayed. New bots may not be started until that period is completed. Our team is working to minimize the effects with maximum priority. We apologise for the inconvenience and thank you for your comprehension.
Report: "Technical issues affecting the performance of bots"
Last updateThe incident has been resolved.
There is an ongoing incident affecting the performance of our bots. We are investigating and working on the issue with maximum priority.
Report: "Metrics and Logs issue - August 2nd (10h40) to August 3rd (13h30)"
Last updateThis incident has been resolved.
During the performance of a server migration, there was an issue that erased the metrics and logs from August 2nd 10:40 to August 3rd 13:30 (CEST). Only the metrics and integration logs were affected, NOT any customer data. That means, all the data collected by the bots will still be available. We apologize for any inconvenience.
Report: "Provider Outage - Timestamp Workaround affected"
Last updateWe have developed a possibility of capturing the Timestamp in our Formulas block. Please check this article to find more information on how to use it: https://help.landbot.io/article/9p4hxupax5-formulas-date#now
Unfortunately due to yesterday’s internet providers outage, a free API (worldtime API) that was recommended to use as a workaround, to extract a custom timestamp, is down. If you are using such API, please check your bot, and if needed reconnect the flow to avoid the brick timestamp. Thank you for your comprehension.
Report: "Outage affecting our Services"
Last updateThis incident has been resolved.
We are continuing to investigate the issue. All channels seem to be affected.
We are experiencing an outage affecting our Services. We are currently investigating the issue.
Report: "Outage at the Landbot Home (www.landbot.io) - Services and bots not affected"
Last updateFor more information about the incident, please check: https://status.fastly.com/incidents/vpk0ssybt3bj
Due to the outage in some of our providers' services, some elements might not work as expected. So far, we identified issues with Giphys, Viveo and our home page.
To log into Landbot without visiting our Home page, please access app.landbot.io directly.
Due to a global issue with our provider, our Home (www.landbot.io) is currently not available. Our Provider is already working on the issue, and we will update this incident as their investigation progresses. Landbot Services and bots are not affected by this issue.
Report: "Outage affecting our Services"
Last updateWe have experienced an outage affecting our Services for 10 minutes approximately. The issue has been resolved.
Report: "Major outage affecting our Services"
Last updateA production change had an unexpected result affecting our services. The issue has been resolved and we’re analyzing it.
We have experienced an outage affecting our Services during 30 minutes approximately.
Report: "Outage affecting Whatsapp Services"
Last updateThere was an incident today in Smooch's services, with WhatsApp message delivery issues from 06:11 to 09:16 CET. The issue has already been solved. You will be able to find the root cause analysis here: https://support.zendesk.com/hc/en-us/articles/1260804654070-Service-Incident-April-27th-2021 We’re sorry for the inconvenience this may have caused.
Report: "Outage affecting our Services"
Last updateThe incident has been resolved.
We are experiencing an outage affecting our Services. We are currently investigating the issue.
Report: "Whatsapp Services Affected"
Last updateThis incident has been resolved.
Please check updates on https://status.fb.com/whatsapp-business-api
Facebook is facing issues that are affecting our Whatsapp Services.
Report: "Outage affecting our Services"
Last updateThis incident has been resolved.
We are experiencing an outage affecting our Services. We are currently investigating the issue.
Report: "Outage affecting our Services"
Last updateThe incident has been resolved.
We are experiencing an outage affecting our Services. We are currently investigating the issue.
Report: "Outage affecting our Services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue. Only the Platform is affected, the bots are operational.
We are experiencing an outage affecting our Services. We are currently investigating the issue.
Report: "Technical issues affecting the performance of Web bots"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are experiencing technical issues that are affecting the performance of some Web bots. We're currently investigating the issue.
Report: "Technical issues affecting the performance of bots"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue was been identifies and a fix will be implemented.
We're experiencing a partial outage that's affecting the performance of some bots. We're currently investigating the issue.
Report: "Technical issues affecting the performance of bots"
Last updateThis incident has been resolved.
The issue has been identified.
We're experiencing a partial outage that's affecting the performance of our bots. We're currently investigating the issue.
Report: "Outage in Chats section"
Last updateThe incident has been resolved.
The issue has been identified and a fix is being implemented.
We're experiencing a partial outage that's affecting the performance of the Chats section for some brands. We're currently investigating the issue.
Report: "Outage affecting our Services"
Last updateThe incident has been resolved.
We are continuing to investigate this issue.
We're experiencing a partial outage that's affecting the performance of our Services. We're currently investigating the issue.
Report: "Outage affecting our Services"
Last updateThis incident has been resolved.
We're experiencing a partial outage that's affecting the performance of our Services. We're currently investigating the issue.
Report: "Outage affecting our Services"
Last updateWe were suffering a partial outage in our systems for approximately one hour. A fix has been implemented by the technical team, and this incident has been resolved.
Report: "Outage affecting our Services"
Last updateThis incident has been resolved.
We are suffering a partial outage in our systems. Our technical team is currently investigating this issue.
Report: "Partial outage affecting the performance of the chatbots."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're experiencing a partial outage that's affecting the performance of the website, Messenger and WhatsApp chatbots, causing significant levels of latency depending on the specific region. We're currently investigating the issue.