Historical record of incidents for Lookback
Report: "Dashboard and Player outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "New redactions are not being processed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Redacting parts of a session is currently not possible. While the user interface in the player still lets you flag a part of the session for redaction, the redaction will not be processed at the moment.
Report: "Android app temporarily unavailable"
Last updateThe Android application has now been restored
We believe we have resolved the issue and are waiting for the app to be restored to the play store
Google has temporarily removed the Participate App from the Android App Store. We are working to resolve the underlying issue and restore the app. While the app is not listed participants who do not already have the app cannot download it, those that have it can continue using it.
Report: "Transcriptions are slow"
Last updateAll pending transcriptions have now been completed and new transcriptions will complete promptly again. Apologies for the disruption inconvenience.
The fix is working as intended the backlog is still being processed. Until this is completed new transcriptions will be be delayed.
We believe the root cause has been addressed. It may take a while until we have worked through the backlog that built up during the period the issue was unmitigated.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Playback unavailable"
Last updateAn outage affecting playback of previously recorded sessions.
Report: "Slow loading times"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Participate links not working"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Emails, notifications, transcriptions, and player feed not working"
Last updateWe have restored our queue system and all effected systems are now operational.
A fix has been implemented and we are monitoring the situation.
We've identified that our queue system has gone down, and are working on bringing it up again.
Report: "Not possible to create new organizations, slow loading of dashboard, organization settings, and player"
Last updateAll systems are operational. We're continuing to keep an eye out on any degraded performance when it comes to organization creation from new signups, as well as loading times.
A fix has been implemented and we are monitoring the results.
Slow loading times are gone, but we're continuing to investigate the core issue. Team creation might still be slow and not work.
We are currently investigating this issue.
Report: "Login unavailable"
Last updateThis incident has been resolved.
Our fix worked. Considering the matter resolved.
A fix has been implemented and we are monitoring the results.
It's currently not possible to login to the platform. We are investigating the issue.
Report: "Users cannot login to dashboard and player"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Elevated errors"
Last updateThe issue has been resolved and dashboard is stable again.
We have detected elevated errors in the Dashboard and are investigating this.
Report: "Information about the Log4j issue."
Last updateThis incident has been resolved.
AWS has confirmed that the services we rely on were not vulnerable to attacks due to Log4j.
Log4j is a Java-based logging utility found in a wide number of software products. The CVE-2021-44228 vulnerability (aka the “Log4Shell” vulnerability) was disclosed by the Apache Log4j project. If exploited, this vulnerability could potentially allow a remote attacker to execute code on the server. We don't use Java in our systems and as such have no dependencies on Log4j. The one exception to this is our Android app, Participate, which does not use Log4j. Our database providers are not affected. We rely on AWS as our datacenter and some of their systems may have been vulnerable in a way that affects us. We are investigating that further. We continue to monitor the situation.
Report: "Users cannot log in to lookback.io"
Last updateThis incident has been resolved.
Rollback of previous deploy has fixed the issue. Still investigating root cause.
Users are reporting an error "Something is wrong" when attempting to access the login page https://auth.lookback.io/login.
Report: "Chrome 92: Participants not being able to hear Moderators in Participate for Desktop (LiveShare)"
Last updateA fixed Chrome extension is approved in the Chrome Web Store, and published as version 2.24.3.
We've submitted a fix for Participate for Desktop to the Chrome Webstore. It's pending review. If approved, it'll roll out in a few hours.
Since the release of Chrome 92, we can reproduce an issue where Participants using Participate for Desktop (our Chrome extension) cannot hear the Moderator in LiveShare sessions. We recommend using Interview mode instead until a fix has been implemented.
Report: "Sessions not becoming watchable"
Last updateThis incident has been resolved.
We've identified the issue and are doing rollbacks. Non-uploaded sessions will be uploaded automatically.
We are continuing to investigate this issue.
After ending a session, it does not become watchable in the player or dashboard. This is for new recordings. We're investigating the issue.
Report: "Watching replay sessions doesn't work – stuck in "Loading...""
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We're continuing to troubleshoot this issue. A _potential_ root cause has been identified.
No sessions can be watched in the player. We are currently investigating.
Report: "Users are unable to access player, dashboard, or settings"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
The backend service that powers multiple services in the Lookback platform is not responding. We are currently investigating and attempting to roll back to a working state.
Report: "Experiencing intermittent issues when loading dashboard and player"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating.
Report: "Slowdown in main backend – affecting dashboard, player, and participant's sessions"
Last updateWe had a near outage in one of our main backends, which affected queries from - the dashboard - the player - and participant clients (iOS, Android, Chrome) It was a "near" outage, since the service didn't go down, but had a major slowdown in responding to queries. Queries would be queued up and cause a snowball effect. We rolled back to an earlier version, and are now investigating the root cause.
Report: "Recordings DB outage"
Last updateWe are currently investigating this issue.
Report: "Users are unable to start new sessions or end existing sessions"
Last updateThis issue has been resolved.
We are continuing to work on a fix for this issue.
Users are currently unable to start new sessions or end existing sessions. We are investigating.
Report: "Some moderators running sessions with iOS Participate app users on iOS are unable to hear Participants wearing headphones"
Last updateFixed in Participate iOS 4.3.2
We have tested a fix for the audio problems and have submitted a new iOS Participate release to Apple. We will update this incident once the review process has been completed and the app is available.
We have identified the root cause of the issue creating audio problems and have created a fix. A new iOS release is being tested. Once testing has completed we will submit the updated app for review by Apple.
We are continuing our investigation to identify and resolve the root cause of this issue. In the mean time we recommend that customers pause testing on iOS devices until a fix has been made available. We will continue to update this page as work progresses.
We are continuing to investigate this issue.
Customers have informed us that using the latest release of the iOS app they are sometimes unable to hear Participants on iOS 14 when the Participant is wearing headphones. We are currently investigating whether this is due to a recent app update as well as exact steps required to reproduce the problem.
Report: "Some customers are unable to access the dashboard, player, and cannot carry out sessions"
Last updateWe are not seeing any further errors at this time.
We are now seeing very few errors for customers and are continuing to monitor the situation.
We are currently investigating the inability for some customers to access the dashboard and run sessions due to an outage at a dependent service.
Report: "Some customers are having issues loading the dashboard or videos"
Last updateErrors have subsided and this incident has been marked as resolved.
We've introduced a fix and are no longer encountering this behavior, we will continue to monitor the situation.
We are currently investigating an issue where some customers are unable to load their dashboard or videos.
Report: "Users are unable to export recordings"
Last updateIssues with exporting for recordings and highlights have been addressed.
We have identified the issue with exports and are working on deploying a fix.
Some users are unable to export their recordings and highlights, we are investigating the cause and will provide updates as available.
Report: "Some users are unable to play recordings"
Last updateWe have identified the issue and a fix has been deployed to address all issues.
We are continuing to investigate this issue.
Some users are unable to play recordings, are having issues starting sessions, and are seeing errors in the player, we are investigating now.
Report: "Users are unable to start sessions"
Last updateWe have resolved this incident.
We have identified the root cause of the issue and are continuing to monitor the situation.
Functionality should now be restored for customers. We are continuing to investigate the root cause of this issue.
We are investigating reports of users being unable to start live, in-person, and self test sessions.
Report: "Some users are unable to play recordings"
Last updateWe have addressed this issue and believe mitigation steps are in place, we will continue to monitor.
We're continuing to monitor the situation and have not encountered further issues.
We have identified the cause of playback issues and have addressed it for a majority of customers, we will continue to monitor the situation.
This should be resolved for many users however we're continuing to investigate the root cause of this issue.
We are currently investigating some user's inability to play back recordings.
Report: "Users may be unable to download recordings"
Last updateWe have confirmed that download functionality is now fully operational for all users again.
We have addressed issues related to some recordings being unavailable for download and have confirmed that downloads are processing correctly again.
We have identified why some recordings cannot be downloaded and are working to address it now.
Some users have notified us that they are unable to download recordings. We have confirmed this and are investigating now.
Report: "Lookback.io is not responding"
Last updateWe have resolved the issues impacting Lookback.io as well as Self-Test and Live sessions and have confirmed that they are now functioning correctly.
We have identified the issue and have restored partial functionality to the site. Self-Tests and Live sessions cannot be uploaded at this time and we are continuing to troubleshoot the issue.
We are currently investigating a site outage due to a service update and will provide updates as they are available.
Report: "Session downloads unavailable"
Last updateWe have taken steps to address users being unable to download sessions and highlights.
We've received reports of being unable to download sessions and highlights and are currently investigating.
Report: "Processing errors caused by video pipeline scaling"
Last updateNew video workers that were brought online to process increased demand were failing. They were decommissioned and new, functional, workers were brought online to re-process failed recordings and increase capacity.
Report: "Unexpected server crash"
Last updateAll systems nominal.
Service appears to be restored. We're now taking some remediation steps while monitoring for continued functionality.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
One of our virtual servers (EC2 Instance for those "in the know") unexpectedly went down. This should have been a quick recovery, if not for cascading affects to how we orchestrate our services (i.e. Docker Swarm).
Report: "Server Outage"
Last updateWe've successfully made a rollback and thus resolved the outage on all services.
We're seeing a server outage affecting Live streaming and all uploading of recordings, and are investigating a rollback.
Report: "Server outage"
Last updateThis incident has been resolved. It was caused by an update to our servers that temporarily made them lose connection with each other.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Live server outage"
Last updateDue to an unexpected server failure Live service was non functional for 20 minutes.
Report: "Live server outage"
Last updateServers appear to be stable. Thanks for your patience!
We have revived the server and are monitoring it to ensure it's stable. The Live feature for Participate Chrome should be functional again.
One of our live servers is temporarily down. We're currently working on bringing it back up.
Report: "Outage of the Participate web service"
Last updateWe've identified the two-fold cause of bad database URLs and bad configuration setting. Both have been rectified and the service is operating normally.
Currently the Participate flow is broken due to an internal server error, possibly caused by a deploy configuration change. Currently investigating the source of the error.
Report: "AWS partial outage of CloudFront and S3 services (recording download & upload)"
Last updateAWS has reported the issue as resolved. Everything should be functioning normally.
AWS has restored some ability to retrieve data, but uploads may continue to be affected.
Earlier today AWS started experiencing problems (mainly in US-EAST region) which impacted the services we use to upload & download recordings—as well as other static files which are downloaded from & displayed on the site.
Report: "Observers are unable to join live recordings."
Last updateService seems to be operational now. More investigation will be done to determine root cause and monitoring will be added to ensure any system failures will be caught sooner.
We've restarted the service that went down which powers the "observers" feature. It should be functional now and we're monitoring it to ensure it stays up.
Live observers are getting stuck on "Waiting for participant to accept..." when attempting to join an ongoing live recording.
Report: "Participate service is down"
Last updateRecovering from a feature regression caused a partial outage. An older version of the service has been restored, and we will re-attempt deploying tomorrow.
Change in server configuration might have caused the outage.
The service powering our Participate feature is experiencing unexpected downtime. Actively working to resolve the issue & restore service.
Report: "Lookback.io is unreachable"
Last updateThis incident has been resolved.
Lookback.io is currently down for all users.
Report: "Lookback.io is not loading in Safari"
Last updateThis incident has been resolved.
No version of Safari can't load lookback.io at the moment. We have a fix to be deployed.
Report: "Linux worker outage"
Last updateBacklog has been cleared, there should be no more delays in processing.
There now seems to be a backlog of Mac recordings. Investigating now.
Processing servers have been brought back online and are consuming the backlog now.
There was a back-up of Mac & Android processing jobs, and the active jobs weren't completing. We're attempting a restart to restore functionality and address the backlog.
Report: "Upload API downtime"
Last updateThe source of the problem with Upload API has been identified and remedied. Uploads should now be restored to original capacity.
At roughly 2016-03-20 02:00 GMT, our server coordinating uploads went down. It was down for 1 day and 6 hours during the weekend. It is now up again, but may experience sporadic downtime. No data has been lost: all Lookback clients will reattempt uploads automatically in a while, or immediately if you restart the Lookback app.
Report: "Slow Uploads"
Last updateDue to a slow intercontinental link between Europe and US, uploading and processing of recording was very slow.