Is Rewind Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved.

We're investigating potential degraded performance with the Rewind Alerts app.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are investigating possible issues with the Rewind Staging web application.

This incident has been resolved.

A workaround has been implemented and we are monitoring the situation. We are rerunning the nightly backups that failed last night. We will leave status update up for the rest of the day. We are expecting tonight's nightly backups to progress as expected. This affects only QuickBooks customers.

We are exploring a possible mitigation and are in the process of deploying it. This affects only QuickBooks customers.

We are continuing to investigate. This affects only QuickBooks customers.

It appears that nightly backups for some of our QuickBooks Online customers did not occur last night. We are currently investigating. This affects only QuickBooks customers.

This incident has been resolved.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

As a temporary fix, we have disabled backups for Mailchimp landing page content objects. All other objects are being backed up successfully.

This incident has been resolved.

The migration continues. We have taken steps to reduce the slowness, although some backups are still taking longer than usual. We continue to monitor the situation.

As we are migrating our Shopify customers to a new Shopify API endpoint, we have noticed that some of our nightly backups (<2%) are slower than usual. This is a temporary situation and we are monitoring very carefully.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We're investigating restores getting stuck for the us east region. Our team is on it!

We have resolved the issues affecting performance.

We are currently investigating this issue.

This incident has been resolved.

We have identified the root cause of the issue and are currently deploying a fix.

We are currently experiencing intermittent slow downs on app.rewind.com. We are currently investigating and will update with more details.

This incident has been resolved.

We are currently investigating an issue with our subscriptions page.

This incident has been resolved.

Investigating issues with Advanced Restores on Shopify and BigCommerce

The Rewind web application is not displaying versions of backed up items for some platforms. Backups continue to run successfully and are not affected.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're investigating delayed backups for some customers in US region. Our team is on it!

This incident has been resolved.

We are continuing to monitor for any further issues.

We identified delays in the backups running in 'eu-central-1' region. We fixed the issue and monitoring the results.

This incident is resolved and nightly backups now resume as normal.

A fix is implemented and we are monitoring the results

The issue has been identified and a fix is being implemented.

We're investigating a large number of nightly rewind backups not being scheduled. Our team is on it!

This incident has been resolved.

We are currently investigating a performance issue with our core internal api service.

# BackHub audit log data loss incident on March 14, 2024 ## Summary On March 14th 2024 during a live Disaster Recovery \(DR\) test for the BackHub system, the data store for the live audit log was deleted in error. While investigating what had happened, it was discovered that no backup existed of the audit log data store and all audit log records for the BackHub infrastructure was lost. ## What Happened We perform bi-annual DR testing of all our application services and infrastructure. During this testing, we discovered that for the BackHub infrastructure \(which handles backups of Github\), we had not previously done a full AWS regional failure DR test and proceeded to do so. While the DR testing was successful and resulted in good updates to our DR procedures, the teardown of the DR testing region revealed a problem with one of the data stores used by BackHub. BackHub uses three main data stores for permanent storage: * For backup metadata and customer information, an AWS RDS database is utilized * For the actual Github backups themselves, AWS EBS storage is utilized * For the audit logging \(backup complete, backup failed, restore complete, etc.\), AWS SimpleDB is utilized All of the infrastructure for the environments are managed using Infrastructure-as-code, specifically with Terraform. ### SimpleDB SimpleDB is an AWS service that launched in 2007 as a ‘simple' NoSQL data store. It has since been superseded by DynamoDB as the main NoSQL offering from AWS and indeed is no longer accessible in the AWS console. Within Rewind, the BackHub infrastructure is the only component using SimpleDB as a data store. It is this service which is used to hold the audit log records for BackHub Within the BackHub infrastructure, the backup services are deployed in multiple regions for data residency purposes. However, the core ‘administration’ database is hosted in a single region with snapshots replicated for DR purposes. Within Terraform, the configuration for SimpleDB looks like this: ```json module "simpledb" { source = "../modules/simpledb" providers = { aws = aws.eu-west-1 } } ``` Meaning, whatever provider is used for the backup services, SimpleDB will always be referenced in the eu-west-1 region. Usually when performing a _terraform apply_ for a pre-existing resource, the apply will fail and the resource must be imported into the terraform state. However, that is not the case with SimpleDB - the resource is added to the existing state with no conflicts. For someone not familiar with the fine details of the Terraform template, it appears as if a new SimpleDB instance has been created in the newly configured DR region. The issue then comes when running a terraform destroy operation to remove the DR testing infrastructure - the SimpleDB instance in eu-west-1 is destroyed rather than what the operator expects which is the DR copy of the SimpleDB instance. ### SimpleDB Backups Rewind has extensive policies and procedures around backup and restore of critical infrastructure and data stores and this is regularly tested bi-annually in a process known as the “restore-a-thon”. During this process, we verify we can restore everything we have backed up - both in the same region and a replica region within AWS. However, despite all of this we found we had no backup of the SimpleDB database being used for audit logging and hence no restoration of the now destroyed database was possible. We also looked at the possibility of re-creating the audit log database from regular log messages emitted by BackHub but found that the log messages do not contain enough information to reconstruct the audit log records. ## Lessons Learned and Actions We are applying the following learnings and actions from this incident: * SimpleDB has no built-in backup and restore process * Action: we will create our own tooling to facilitate this * While we have guard rails in place to prevent deletion of data stores, SimpleDB was not in this policy * Action: We have added SimpleDB to the guardrails around deletion * SimpleDB backup and restore testing should be performed at the same interval as other data stores * Action: SimpleDB is being added to the regular backup and restore testing process * All data stores should be re-audited for backup and restore capabilities and procedures * Action: All data stores are being re-audited to ensure full backup and restore processes and procedures exist

We have identified and remediated the problem with the audit log for the Backhub system. A postmortem will be published for this incident.

We have identified the issue with the Backhub audit log and are considering mitigations

We are currently investigating an issue where some customers are missing audit log data in the Backhub product.

This incident has been resolved.

We have identified an issue affecting several of our platforms. We are working on a fix.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

We've identified an issue affecting backup performance for a subset of customers. Our team is on it!

This incident has been resolved.

We're experiencing delays in marking backups as complete for the Platforms - Confluence, Jira and Bitbucket in the European region. Our team has identified the problem and is working to resolve it.

This incident has been resolved.

We're investigating rewind app vault search issues for some customers. Our team is on it!

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're investigating an issue affecting a subset of our BigCommerce backup customers. Our team is on it!

This incident has been resolved.

We have identified an issue with account exports and are currently working on a fix.

Backups are operating normally. Thank you for your patience.

We've identified an issue affecting backup performance in our European region for Jira, Confluence, Bitbucket, and Github.

This incident has been resolved.

We've identified the issue and are working on a fix Our team is on it!

This incident has been resolved.

We are beginning to see webhooks being processed. We are continuing to monitor the situation.

We are continuing to work on a fix for this issue.

We're investigating elevated error rates for Rewind webhooks for all platforms. Our team is on it! Backups and restores are functional, however near-real-time backups are currently delayed. Shopify Staging one-time copies are functional, however continuous copies are currently delayed.

Thank you for your patience, JIRA and Confluence backups have resumed.

We are currently investigating an issue with JIRA and Confluence backups in the Australia region.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating the issue. Users are still able to sign in with their email login in the meantime.

This incident has been resolved.

We are investigating possible issues with the Rewind web application. Don't fret though, your backups are still running!

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating possible issues with the Rewind web application. Don't fret though, your backups are still running!

This incident has been resolved.

We're currently investigating an issue with our Jira backups Vault view

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We're investigating elevated error rates for Rewind backups for the Trello platform. Our team is on it!

This incident has been resolved.

We are currently investigating the issues with copies for Shopify Copy.

This incident has been resolved.

We are continuing to investigate this issue for Shopify Copy. BigCommerce Copy is back to operational.

We are continuing to investigate this issue.

We are currently investigating the issues with BigCommerce Copy and Shopify Copy.

This incident has been resolved.

A fix for the issue has been applied. Degraded performance may persist for a short time while systems stabilize.

We have identified the issue and are working on a fix.

We're investigating elevated error rates for Rewind backups on the Shopify platform.

This incident has been resolved.

We are currently investigating an issue with Rewind Backups for BigCommerce in the US region.

This incident has been resolved.

Our engineers have identified the root cause of the issue and are working on a fix.

We've identified an issue that caused backups for the Trello platform in the US region to not be performed as scheduled on the evening of Sept 8th. Our team has diagnosed the issue and is working on a fix.

This incident has been resolved.

Issue has been identified and backups are in progress for all accounts that were impacted by this service interruption. We will post an update once this is complete.

We are experiencing a service interruption to nightly backups for Intuit; action is underway to resume service, we will post an update as soon as one is available.

This incident has been resolved.

We are currently investigating an issue with Rewind Backups for Shopify in the US region.

This incident has been resolved.

We are continuing to bring service back online to full capacity.

We are currently investigating an issue with Rewind Backups for Shopify in the US region.

This incident has been resolved.

We are currently investigating an issue with backups for Github in the AWS EU region

This incident has been resolved.

We are continuing to investigate this issue.

We are investigating possible issues with the Rewind web application. Don't fret though, your backups are still running!

This incident has been resolved.

Capacity has been increased in the US region and we're seeing much improved processing rates. Near-realtime backups may be delayed by a few minutes

We're investigating elevated error rates for Rewind backups on the BigCommerce platform in the US region. Our team is on it!

This incident has been resolved.

We are continuing to work on a fix for this issue.

Rewind engineers have identified the issue and are working towards remediation. In the interim, please consider using Firefox, Safari, or logging in directly from https://app.rewind.io

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue with the Rewind Web Application.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved and all backup and restore services are functioning normally

We have identified the issue and engineers are applying a mitigation

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results

We're investigating elevated error rates for Rewind backups on the Shopify platform in the US region. Our team is on it!

This incident has been resolved.

We are continuing to investigate this issue.

We are currently investigating this issue.