Lookback

Is Lookback Down Right Now? Check if there is a current outage ongoing.

Lookback is currently Operational

Last checked from Lookback's official status page

Historical record of incidents for Lookback

Report: "Dashboard and Player outage"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "New redactions are not being processed"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Redacting parts of a session is currently not possible. While the user interface in the player still lets you flag a part of the session for redaction, the redaction will not be processed at the moment.

Report: "Android app temporarily unavailable"

Last update
resolved

The Android application has now been restored

identified

We believe we have resolved the issue and are waiting for the app to be restored to the play store

investigating

Google has temporarily removed the Participate App from the Android App Store. We are working to resolve the underlying issue and restore the app. While the app is not listed participants who do not already have the app cannot download it, those that have it can continue using it.

Report: "Transcriptions are slow"

Last update
resolved

All pending transcriptions have now been completed and new transcriptions will complete promptly again. Apologies for the disruption inconvenience.

monitoring

The fix is working as intended the backlog is still being processed. Until this is completed new transcriptions will be be delayed.

monitoring

We believe the root cause has been addressed. It may take a while until we have worked through the backlog that built up during the period the issue was unmitigated.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Playback unavailable"

Last update
resolved

An outage affecting playback of previously recorded sessions.

Report: "Slow loading times"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating this issue.

Report: "Participate links not working"

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

Report: "Emails, notifications, transcriptions, and player feed not working"

Last update
resolved

We have restored our queue system and all effected systems are now operational.

monitoring

A fix has been implemented and we are monitoring the situation.

identified

We've identified that our queue system has gone down, and are working on bringing it up again.

Report: "Not possible to create new organizations, slow loading of dashboard, organization settings, and player"

Last update
resolved

All systems are operational. We're continuing to keep an eye out on any degraded performance when it comes to organization creation from new signups, as well as loading times.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Slow loading times are gone, but we're continuing to investigate the core issue. Team creation might still be slow and not work.

investigating

We are currently investigating this issue.

Report: "Login unavailable"

Last update
resolved

This incident has been resolved.

monitoring

Our fix worked. Considering the matter resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

It's currently not possible to login to the platform. We are investigating the issue.

Report: "Users cannot login to dashboard and player"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating this issue.

Report: "Elevated errors"

Last update
resolved

The issue has been resolved and dashboard is stable again.

investigating

We have detected elevated errors in the Dashboard and are investigating this.

Report: "Information about the Log4j issue."

Last update
resolved

This incident has been resolved.

monitoring

AWS has confirmed that the services we rely on were not vulnerable to attacks due to Log4j.

monitoring

Log4j is a Java-based logging utility found in a wide number of software products. The CVE-2021-44228 vulnerability (aka the “Log4Shell” vulnerability) was disclosed by the Apache Log4j project. If exploited, this vulnerability could potentially allow a remote attacker to execute code on the server. We don't use Java in our systems and as such have no dependencies on Log4j. The one exception to this is our Android app, Participate, which does not use Log4j. Our database providers are not affected. We rely on AWS as our datacenter and some of their systems may have been vulnerable in a way that affects us. We are investigating that further. We continue to monitor the situation.

Report: "Users cannot log in to lookback.io"

Last update
resolved

This incident has been resolved.

investigating

Rollback of previous deploy has fixed the issue. Still investigating root cause.

investigating

Users are reporting an error "Something is wrong" when attempting to access the login page https://auth.lookback.io/login.

Report: "Chrome 92: Participants not being able to hear Moderators in Participate for Desktop (LiveShare)"

Last update
resolved

A fixed Chrome extension is approved in the Chrome Web Store, and published as version 2.24.3.

monitoring

We've submitted a fix for Participate for Desktop to the Chrome Webstore. It's pending review. If approved, it'll roll out in a few hours.

identified

Since the release of Chrome 92, we can reproduce an issue where Participants using Participate for Desktop (our Chrome extension) cannot hear the Moderator in LiveShare sessions. We recommend using Interview mode instead until a fix has been implemented.

Report: "Sessions not becoming watchable"

Last update
resolved

This incident has been resolved.

identified

We've identified the issue and are doing rollbacks. Non-uploaded sessions will be uploaded automatically.

investigating

We are continuing to investigate this issue.

investigating

After ending a session, it does not become watchable in the player or dashboard. This is for new recordings. We're investigating the issue.

Report: "Watching replay sessions doesn't work – stuck in "Loading...""

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

investigating

We are continuing to investigate this issue.

investigating

We're continuing to troubleshoot this issue. A _potential_ root cause has been identified.

investigating

No sessions can be watched in the player. We are currently investigating.

Report: "Users are unable to access player, dashboard, or settings"

Last update
resolved

This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

The backend service that powers multiple services in the Lookback platform is not responding. We are currently investigating and attempting to roll back to a working state.

Report: "Experiencing intermittent issues when loading dashboard and player"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating.

Report: "Slowdown in main backend – affecting dashboard, player, and participant's sessions"

Last update
resolved

We had a near outage in one of our main backends, which affected queries from - the dashboard - the player - and participant clients (iOS, Android, Chrome) It was a "near" outage, since the service didn't go down, but had a major slowdown in responding to queries. Queries would be queued up and cause a snowball effect. We rolled back to an earlier version, and are now investigating the root cause.

Report: "Recordings DB outage"

Last update
resolved

We are currently investigating this issue.

Report: "Users are unable to start new sessions or end existing sessions"

Last update
resolved

This issue has been resolved.

identified

We are continuing to work on a fix for this issue.

identified

Users are currently unable to start new sessions or end existing sessions. We are investigating.

Report: "Some moderators running sessions with iOS Participate app users on iOS are unable to hear Participants wearing headphones"

Last update
resolved

Fixed in Participate iOS 4.3.2

identified

We have tested a fix for the audio problems and have submitted a new iOS Participate release to Apple. We will update this incident once the review process has been completed and the app is available.

identified

We have identified the root cause of the issue creating audio problems and have created a fix. A new iOS release is being tested. Once testing has completed we will submit the updated app for review by Apple.

investigating

We are continuing our investigation to identify and resolve the root cause of this issue. In the mean time we recommend that customers pause testing on iOS devices until a fix has been made available. We will continue to update this page as work progresses.

investigating

We are continuing to investigate this issue.

investigating

Customers have informed us that using the latest release of the iOS app they are sometimes unable to hear Participants on iOS 14 when the Participant is wearing headphones. We are currently investigating whether this is due to a recent app update as well as exact steps required to reproduce the problem.

Report: "Some customers are unable to access the dashboard, player, and cannot carry out sessions"

Last update
resolved

We are not seeing any further errors at this time.

monitoring

We are now seeing very few errors for customers and are continuing to monitor the situation.

investigating

We are currently investigating the inability for some customers to access the dashboard and run sessions due to an outage at a dependent service.

Report: "Some customers are having issues loading the dashboard or videos"

Last update
resolved

Errors have subsided and this incident has been marked as resolved.

monitoring

We've introduced a fix and are no longer encountering this behavior, we will continue to monitor the situation.

investigating

We are currently investigating an issue where some customers are unable to load their dashboard or videos.

Report: "Users are unable to export recordings"

Last update
resolved

Issues with exporting for recordings and highlights have been addressed.

identified

We have identified the issue with exports and are working on deploying a fix.

investigating

Some users are unable to export their recordings and highlights, we are investigating the cause and will provide updates as available.

Report: "Some users are unable to play recordings"

Last update
resolved

We have identified the issue and a fix has been deployed to address all issues.

investigating

We are continuing to investigate this issue.

investigating

Some users are unable to play recordings, are having issues starting sessions, and are seeing errors in the player, we are investigating now.

Report: "Users are unable to start sessions"

Last update
resolved

We have resolved this incident.

monitoring

We have identified the root cause of the issue and are continuing to monitor the situation.

investigating

Functionality should now be restored for customers. We are continuing to investigate the root cause of this issue.

investigating

We are investigating reports of users being unable to start live, in-person, and self test sessions.

Report: "Some users are unable to play recordings"

Last update
resolved

We have addressed this issue and believe mitigation steps are in place, we will continue to monitor.

monitoring

We're continuing to monitor the situation and have not encountered further issues.

identified

We have identified the cause of playback issues and have addressed it for a majority of customers, we will continue to monitor the situation.

investigating

This should be resolved for many users however we're continuing to investigate the root cause of this issue.

investigating

We are currently investigating some user's inability to play back recordings.

Report: "Users may be unable to download recordings"

Last update
resolved

We have confirmed that download functionality is now fully operational for all users again.

monitoring

We have addressed issues related to some recordings being unavailable for download and have confirmed that downloads are processing correctly again.

identified

We have identified why some recordings cannot be downloaded and are working to address it now.

investigating

Some users have notified us that they are unable to download recordings. We have confirmed this and are investigating now.

Report: "Lookback.io is not responding"

Last update
resolved

We have resolved the issues impacting Lookback.io as well as Self-Test and Live sessions and have confirmed that they are now functioning correctly.

identified

We have identified the issue and have restored partial functionality to the site. Self-Tests and Live sessions cannot be uploaded at this time and we are continuing to troubleshoot the issue.

investigating

We are currently investigating a site outage due to a service update and will provide updates as they are available.

Report: "Session downloads unavailable"

Last update
resolved

We have taken steps to address users being unable to download sessions and highlights.

investigating

We've received reports of being unable to download sessions and highlights and are currently investigating.

Report: "Processing errors caused by video pipeline scaling"

Last update
resolved

New video workers that were brought online to process increased demand were failing. They were decommissioned and new, functional, workers were brought online to re-process failed recordings and increase capacity.

Report: "Unexpected server crash"

Last update
resolved

All systems nominal.

monitoring

Service appears to be restored. We're now taking some remediation steps while monitoring for continued functionality.

monitoring

We are continuing to monitor for any further issues.

monitoring

We are continuing to monitor for any further issues.

monitoring

One of our virtual servers (EC2 Instance for those "in the know") unexpectedly went down. This should have been a quick recovery, if not for cascading affects to how we orchestrate our services (i.e. Docker Swarm).

Report: "Server Outage"

Last update
resolved

We've successfully made a rollback and thus resolved the outage on all services.

identified

We're seeing a server outage affecting Live streaming and all uploading of recordings, and are investigating a rollback.

Report: "Server outage"

Last update
resolved

This incident has been resolved. It was caused by an update to our servers that temporarily made them lose connection with each other.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Live server outage"

Last update
resolved

Due to an unexpected server failure Live service was non functional for 20 minutes.

Report: "Live server outage"

Last update
resolved

Servers appear to be stable. Thanks for your patience!

monitoring

We have revived the server and are monitoring it to ensure it's stable. The Live feature for Participate Chrome should be functional again.

identified

One of our live servers is temporarily down. We're currently working on bringing it back up.

Report: "Outage of the Participate web service"

Last update
resolved

We've identified the two-fold cause of bad database URLs and bad configuration setting. Both have been rectified and the service is operating normally.

investigating

Currently the Participate flow is broken due to an internal server error, possibly caused by a deploy configuration change. Currently investigating the source of the error.

Report: "AWS partial outage of CloudFront and S3 services (recording download & upload)"

Last update
resolved

AWS has reported the issue as resolved. Everything should be functioning normally.

monitoring

AWS has restored some ability to retrieve data, but uploads may continue to be affected.

monitoring

Earlier today AWS started experiencing problems (mainly in US-EAST region) which impacted the services we use to upload & download recordings—as well as other static files which are downloaded from & displayed on the site.

Report: "Observers are unable to join live recordings."

Last update
resolved

Service seems to be operational now. More investigation will be done to determine root cause and monitoring will be added to ensure any system failures will be caught sooner.

monitoring

We've restarted the service that went down which powers the "observers" feature. It should be functional now and we're monitoring it to ensure it stays up.

investigating

Live observers are getting stuck on "Waiting for participant to accept..." when attempting to join an ongoing live recording.

Report: "Participate service is down"

Last update
resolved

Recovering from a feature regression caused a partial outage. An older version of the service has been restored, and we will re-attempt deploying tomorrow.

identified

Change in server configuration might have caused the outage.

investigating

The service powering our Participate feature is experiencing unexpected downtime. Actively working to resolve the issue & restore service.

Report: "Lookback.io is unreachable"

Last update
resolved

This incident has been resolved.

investigating

Lookback.io is currently down for all users.

Report: "Lookback.io is not loading in Safari"

Last update
resolved

This incident has been resolved.

identified

No version of Safari can't load lookback.io at the moment. We have a fix to be deployed.

Report: "Linux worker outage"

Last update
resolved

Backlog has been cleared, there should be no more delays in processing.

investigating

There now seems to be a backlog of Mac recordings. Investigating now.

monitoring

Processing servers have been brought back online and are consuming the backlog now.

investigating

There was a back-up of Mac & Android processing jobs, and the active jobs weren't completing. We're attempting a restart to restore functionality and address the backlog.

Report: "Upload API downtime"

Last update
resolved

The source of the problem with Upload API has been identified and remedied. Uploads should now be restored to original capacity.

investigating

At roughly 2016-03-20 02:00 GMT, our server coordinating uploads went down. It was down for 1 day and 6 hours during the weekend. It is now up again, but may experience sporadic downtime. No data has been lost: all Lookback clients will reattempt uploads automatically in a while, or immediately if you restart the Lookback app.

Report: "Slow Uploads"

Last update
resolved

Due to a slow intercontinental link between Europe and US, uploading and processing of recording was very slow.