Historical record of incidents for BombBomb
Report: "Email Sending Issue"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented for sending, and we are currently monitoring the situation.
Our teams have identified an issue affecting the ability to send emails from BombBomb servers. We are actively working on this and deploying a fix to resolve the problem.
Report: "Advanced Composer Issue"
Last updateThis incident has been resolved.
We are currently investigating an issue within our Advanced Email Composer.
Report: "Reduced ability to record videos"
Last updateThis incident has been resolved.
Service is restored, we are still monitoring.
We are currently investigating this issue.
Report: "Email Delivery Issue with Yahoo"
Last updateThis incident has been resolved.
Our teams have implemented a fix for this issue, and we will be closely monitoring our delivery.
Our team is actively investigating a mail delivery issue with Yahoo.
Report: "Slow or Unresponsive"
Last updateThis incident has been resolved.
We are investigating an issue that may be causing BombBomb to respond slowly or unresponsively.
Report: "BombBomb recorder is not loading."
Last updateThis incident has been resolved, and our teams will monitor the functionality closely.
We have released a temporary fix to allow the BombBomb Recorder to work without using Virtual Backgrounds. Our teams are investigating an intervention regarding the use of the BombBomb Screen recorder which appears to be working intermittently.
We are continuing to work on a fix for this issue.
We have identified the issue causing the BombBomb Recorder not to load is due to an outage with Effects SDK. Our engineering teams are actively working on interventions. As a workaround, our BombBomb mobile app is recording successfully.
We are currently investigating an issue preventing our BombBomb recorder from loading due to an outage.
Report: "Auth0 login outage"
Last updateThis incident has been resolved.
Service has been restored, we are monitoring the situation.
Our login provider, Auth0 is having an outage, and as a result new logins cannot be performed on BombBomb at this time.
Report: "System outage"
Last updateThis incident has been resolved.
We are currently investigating an issue impacting our Web Application and causing an outage.
Report: "Google Warning Flag Issue"
Last updateThis incident has been resolved.
We have implemented the final fixes to this issue and are closely monitoring the situation.
Multiple fixes have been deployed additional fixes are still being implemented.
The issue has been identified and a fix is being implemented.
We are currently investigating the cause of a Google Warning Flag impacting the functionality within the BombBomb platform.
Report: "Google Chrome is showing an error when loading BombBomb application"
Last updateThis issue has been resolved for all users.
Fix has been pushed to all users experiencing error messages in Google Chrome. We will continue to closely monitor this situation.
Fix for BombBomb Prompt functionality has been pushed to all users.
Currently working on a fix for users getting error messages connected to BombBomb Prompt functionality.
Teams functionality fix has been pushed out to impacted Users.
A fix has been pushed out which will resolve the majority of impacted users logging in. Teams are aware of the error message is still impacting Teams functionality and Prompt Functionality, actively investigating a fix.
Our team has identified the cause of this issue and is pushing out a fix that will resolve the majority of impacted users. Continuing to actively investigate the remaining impacted users.
Our team has identified the cause of this issue and is pushing out a fix that will resolve the majority of impacted users. Continuing to actively investigate the remaining impacted users.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are actively investigating an issue affecting some users where Chrome is showing an error when loading our BombBomb application.
Report: "Login issues"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
AWS is seeing improvements in their lambda service. On our end we are seeing restored ability to log in and access the app. We will continue testing and monitoring.
We are continuing to investigate this issue.
AWS has indicated that they have identified the cause and are actively working to fix it.
Video processing will have degraded performance. We are continuing to investigate this issue.
We are investigating issues with our login service. Initial indicators point to a more widespread issue with AWS. Services appear functional if you are already logged in.
Report: "Livefire feed temporarily down"
Last updateThe tracking feed was temporarily showing empty as we performed a configuration change. This change has been completed and the tracking feed is showing normal again.
Report: "Degraded Performance"
Last updateThis incident has been resolved.
Our engineers have identified the issue and fully recovered the performance of the web application.
The system reboot has improved the web applications performance for users. Our engineers are currently working to identify the source of the issue.
The system was rebooted to improve the web application performance. Users may continue to see slowness.
You may be experiencing temporary slowness and delay in the BombBomb web application. Our engineers are working to resolve this for impacted users.
Report: "Team functionality is temporarily unavailable for some users"
Last updatePrompt sends are turned back on for impacted customers. Automations will be slightly delayed.
Our engineers are working towards sending out the automated and schedule emails, and Prompt sends from earlier today.
Our engineers have turned off Prompt and Automation sends for impacted users temporarily to help with the resolution of this issue.
Our engineers have successfully re-associated 90% of shared content to members of teams. We will continue this work and monitor what's been completed.
Our engineers are implementing the second step in resolving this issue, which will re-associate shared video content back to members. We estimate one hour for completion of this step.
We have completed the first step in resolving this issue. Members have been re-associated with their teams.
Our engineers are releasing a fix to re-associate members back to teams in the next few minutes. We will then release a fix to re-associate shared video content back to members, we estimate this portion to be complete by 2:30 MST.
We have a full solution in progress to fix this issue. We estimate a full recovery by 2:30 MST.
We have identified 227 teams that have been dismantled. The individuals associated with these teams still have access to their BombBomb accounts, however, they do not have access to their team's members, content, and analytics. Our engineers have identified the issue and expect a full recovery in rebuilding these teams in the next few hours.
Report: "Performance Issues"
Last updateDetermined cause of performance issues and applied resolutions.
We have identified an issue and deployed a performance fix. The application appears to be recovering, and we will continue monitoring performance.
We are currently experiencing degraded performance.
Report: "Degraded Performance"
Last updateApp performance has fully recovered.
We are beginning to see latency normalize as AWS continues to recover.
AWS latency and error rates are still higher than normal, but appear to be gradually improving. We will continue to monitor and update this page.
The cloud provider our application utilizes is experiencing degraded performance. Our instrumentation indicates a small increase in error rates and some slower load times for the web dashboard, api, and background processes. We're monitoring the situation and will post here as we learn more.
Report: "Increased error rates"
Last updateFunctionality has been restored
A fix has been implemented and we are seeing signs of recovery.
We are seeing errors for users accessing the web application. This is caused by a failure in one of our caching databases. We are working to replace the failed cache.
Report: "BombBomb's AWS (Amazon web server) is down"
Last updateWe will continue to monitor the AWS incident. All of BombBomb's systems are green and operational.
We are continuing to see latency in the navigation of our application and with the use of our video/screen recorder.
We received an update from Amazon regarding the AWS outage. They have executed a mitigation which is showing significant recovery in the US-EAST-1 Region. They're continuing to closely monitor the health of the network devices and expect it to continue to make progress towards full recovery. They still do not have an ETA for full recovery at this time.
Our engineers have restored the ability for you to log in to your BombBomb account. The functionality of the web application has also improved, however, you may still experience latency as you navigate in it.
Our engineers continue to work to restore the ability to log in to our application. If you were logged into the web application prior to the log in page going down, the functionality of the BombBomb application remains limited and inconsistent. You may experience issues using our BombBomb extension and recording and saving videos.
Amazon provided an update on this outage: We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. Services impacted include: EC2, Connect, DynamoDB, Glue, Athena, Timestream, and Chime and other AWS Services in US-EAST-1. The root cause of this issue is an impairment of several network devices in the US-EAST-1 Region. We are pursuing multiple mitigation paths in parallel, and have seen some signs of recovery, but we do not have an ETA for full recovery at this time.
Our log in page continues to be unresponsive. If you've been logged in prior to this page being unresponsive, all parts of the application continue to work with degraded performance with latency issues.
Our account login page is down due to the Amazon API errors rates. We're focusing our efforts on the login service to see if we can get it working in a different way. It's hosted on AWS Lambda which is unresponsive.
The BombBomb application continues to function, however, latency continues.
Our LiveFire notifications are not sending, which is currently not allowing us to collect "Video Watched" events.
We are continuing to work on a fix for this issue.
As the incident with Amazon continues, we are focusing our efforts on how to minimize the impact to us and how to keep our application up throughout this.
Amazon has identified the root cause of the issues in the US-EAST-1 Region. This is a network issue in this Region which is impacting multiple services. They're actively working towards recovery.
Amazon web services is down at the moment. Our application is still running, however, you may experience latency.
Report: "Email delivery is temporarily delayed"
Last updateWe have enabled our email tracking pages and metrics. Our system is running as normal. This incident is resolved.
The database has recovered and our application performance is green. Our email tracking pages and metrics are still disabled.
Performance is still degraded, however, the database is showing signs of improvement.
We've disabled our email tracking pages temporarily while we work through database performance issues.
The database is still experiencing performance issues. We are investigating what else is causing the issues.
The web application's performance is running slowly. Our engineers are continuing to monitor the system.
The web application and email delivery are operating well. We will leave this incident open as we monitor performance.
Email delivery and web application performance are improving. We have identified the issue and will monitor the performance of the application.
We are still experiencing delays in email delivery and web application performance. We are investigating the continued slow down.
We have rebooted our system. Email delivery is starting up again but is still delayed.
Email delivery is slightly delayed as we investigate an issue with our systems.
Report: "BombBomb Web Application Temporarily Unavailable"
Last updateThis incident has been resolved.
We continue to investigate the cause of the issue but our application is 100% back online
Email delivery queues are working to catch up now that the web app and task runners are both back online.
Background task runners are coming back online. We are continuing to investigate the root cause of the issue.
We've rebooted our servers and the web application is available again. We are still working to identify the issue.
The BombBomb web application is temporarily unavailable. We are investigating what is causing the issue.
Report: "Increased Error Rate"
Last updateAfter monitoring, our application metrics are healthy. We will continue to keep an eye on the health of our cloud hosting provider and provide any updates to this page if our application experiences interruptions.
Our main application experienced a few minutes of interruptions and increased error rates between 9:14 and 9:28PM (MDT). This was caused by underlying failures happening in our cloud hosting provider. Errors lasted while our databases moved out of the failing availability zone into a healthy one. Our application is back up and running, and metrics are back to normal. We will continue to monitor application health.
Report: "Degraded application performance"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We have increased our server load to help with traffic. We’re seeing improvements in overall latency and will continue to monitor the situation.
We have identified the issue.
The BombBomb application is currently experiencing higher than normal traffic. We are working on increasing our server capacity and should have normal operations soon.
Report: "Degraded application performance"
Last updateThis incident has been resolved.
We have implemented a fix and are seeing improved response times. We will continue monitoring system performance.
We have identified a problem causing the main application to be unresponsive and are working towards a solution.
Report: "Outage"
Last updateThis incident has been resolved.
We are seeing signs of recovery and monitoring the application performance
We are continuing to investigate this issue.
We are currently investigating an outage on our main application.
Report: "WWW Website Outage"
Last updateCloudFlare is rolling out a fix to resolve the DNS issues affecting our www.bombbomb.com website. We will continue to keep a close eye on our systems.
It appears that CloudFlare and Google experienced major DNS server outages. Service appears to be restored for the www.bombbomb.com website. Other parts of the bombbomb ecosystem appear to be functioning normally.
A world-wide DNS issue is affecting our www.bombbomb.com website. We are investigating to see if any other services are affected. As of now, our main app (app.bombbomb.com) and core services are still functioning.
Report: "Email Delivery Temporarily Unavailable"
Last updateWe have successfully resolved this issue.
All of our email delivery is caught up.
Sending batches are slowly recovering
We are successfully connecting to one of our two mail servers. We are turning our queued sending back on now.
We have identified the connection issue to our mail servers. We are now working to resolve it.
We are investigating an issue with our SMTP connections not working.
Report: "Application and APIs Temporarily Unavailable"
Last updateRecovery has completed, and functionality has been restored.
Our engineers have identified and deployed a solution. We are monitoring the recovery now. Emails, tracking information, and other changes are beginning to catch up.
Our engineers have identified a problem affecting the BombBomb application which is causing the site to be temporarily unavailable. We are working to identify a solution.
Report: "Application Temporarily Unavailable"
Last updateThis incident has been resolved.
The servers are up and running. We are identifying the root cause
Our engineers are investigating a problem with the BombBomb application being unavailable and are working to identify the problem. The application should be available again shortly. Thanks for your patience.
Report: "Application Performance is Lagging"
Last updateEngineers have resolved the issue.
The application is back up and running. Our engineers will continue to monitor the status of the application.
Our engineers have identified the issue and are working to resolve. The application is still experiencing performance issues, which is intermittently available or running slowly. Thanks for your patience.
Engineers are investigating a problem with the BombBomb application being unavailable and are working to resolve. The application should be available again shortly. Thanks for your patience.
Report: "Application Temporarily Unavailable"
Last updateThis issue is resolved.
The application is back online, however, live tracking in the Chrome Extension is still down.
The application is back up and running. Our engineers will continue to monitor the status of the application.
Our engineers are investigating a problem with the BombBomb application being unavailable and are working to identify the problem. The application should be available again shortly. Thanks for your patience.
Report: "Application Temporarily Unavailable"
Last updateOur engineers have resolved the issue. The application is available.
Our engineers have identified the issue and are working to resolve the issue now
Our engineers are investigating a problem with the BombBomb application being unavailable and are working to identify the problem. The application should be available again shortly. Thanks for your patience.
Report: "Application Temporarily Unavailable"
Last updateOur engineers have resolved this issue.
All systems are running and we are slowly bringing back all functionality. Our engineers will continue to monitor the status of the application.
The application is now accessible, however, background tasks ie. Tracking is still unavailable.
Engineers have identified a problem with the BombBomb application being unavailable and are working to resolve. The application should be available again shortly. Thanks for your patience.
Our engineers are investigating a problem with the BombBomb application being unavailable and are working to identify the problem. The application should be available again shortly. Thanks for your patience.
Report: "Application Temporarily Down"
Last updateEngineers have identified and resolved this issue. The BombBomb application is back online!
Engineers are investigating a problem with the BombBomb application being unavailable and are working to resolve. The application should be available again shortly. Thanks for your patience.
Report: "Application performance is lagging"
Last updateThis incident has been resolved.
The problem has been fixed and we are monitoring it
We're experiencing performance issues with the application, which is intermittently available or running slowly. Technicians are investigating. The problem has been identified and we are working to resolve.
Report: "Intermittent Issues"
Last updateThis incident has been resolved.
As a precaution, BombBomb has queued any email sends going out through the web and mobile applications for now. All mail will be sent as our subsystem returns to full health. The Chrome Extension is unaffected.
Issues with Amazon's Simple Storage Service are occasionally causing issues with service delivery.
Report: "Application Temporarily Down"
Last updateThe application is back online. This problem is resolved.
The BombBomb application is coming back online. We are actively monitoring it.
Engineers have identified a problem with the BombBomb application being unavailable and are working to resolve. The application should be available again shortly. Thanks for your patience.
Report: "Application Temporarily Down"
Last updateThe application is back online. This problem is resolved.
The BombBomb application is coming back online. We are actively monitoring it.
AWS reports they’ve identified the issue and are working to remediate.
Engineers have identified a problem with AWS servers and are working to resolve. Application should be available again shortly. Thanks for your patience.
Report: "The WWW site is unreachable at the moment..."
Last updateEngineers were able to restore the WWW site. Thanks for your patience, and we apologize for any inconvenience this may have caused.
We've contacted our hosting partners at Flywheel to resolve the issue, and they are currently investigating. Users can still log in to the application at https://app.bombbomb.com/app/
Report: "Application performance is lagging"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're experiencing performance issues with the application, which is intermittently available or running slowly. Technicians are investigating.
Report: "Delays with Sending and Report Generation"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Application performance is lagging"
Last updateWe experienced performance issues with the application, which was temporarily unavailable or running slowly. Technicians have identified and resolved the issue within minutes.
Report: "Application performance is lagging"
Last updateEngineers were able to resolve the issue. The application was disrupted from 8:55am to 9:01am MST. Thanks for your patience, and sorry for the trouble!
Technicians are investigating
Report: "Email Sending Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're experiencing a delay in email sending. Technicians are investigating.
Report: "Application performance is lagging"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're experiencing performance issues with the application, which is intermittently available or running slowly. Technicians are investigating.
Report: "Application performance issues are resolving."
Last updateTechnicians have fully restored the application. Thanks for your patience. We apologize for the interruption.
Background tasks are again processing. Technicians are monitoring.
Database issue has been identified and system is being restored.
Application is severely affected by a database issue. Technicians are investigating.
Report: "Support site temporarily down."
Last updateThis incident has been resolved.
Please give us a call for help!
Report: "Application is temporarily down"
Last updateThis incident has been resolved.
Engineers have deployed the fix and the application has recovered. We are actively monitoring the situation.
Engineers have identified a problem with AWS servers and are working to resolve. Application should be available again shortly. Thanks for your patience.
Report: "Application performance issues"
Last updateService restored! Sorry for the trouble!
Performance is returning to normal. We're monitoring the situation.
Engineers have identified the problem and are working to resolve. New applications servers have been rotated in to replace the unhealthy servers. Latency is returning to normal levels.
Engineers have observed unusual loads that are overwhelming application servers. The site is intermittent and slow right now. Looking into it with urgency!
Report: "Web Application Outtage"
Last updateThis incident has been resolved.