Is Subscribe Pro Down Right Now? Check if there is a current outage ongoing.

Background job processing went down on 10/1 or us-east-1 region and 10/4 for eu-west-1 region. Both incidents were due to the same root cause: * A key scheduling service failed. * This was triggered by our core transactional database recycling after a minor version upgrade. * This happened around 3am eastern time on 10/4 in us-east-1 region. * Happened around 3am eastern time on 10/1 in eu-west-1 region. * We should have noticed this, however there was an error in some of our alerting / notification scripts. * We have since updated and thoroughly tested the notification scripts.

This incident was caused because we missed updating a key credential during deployment to our production platform. We will be reviewing our process around how we track such credentials during updates.

The issue has been resolved.

We are currently investigating an issue causing an outage of our Merchant App and API.

We have identified the core issue and a fix is in place. We will follow up with a postmortem.

All systems are currently operating normally. We are still monitoring. We will post a postmortem write up once we fully understand the root cause.

We have implemented a fix and are monitoring. Please contact support@subscribepro.com if you continue to experience performance or connectivity issues.

We are currently investigating an issue causing degraded performance of our Merchant App and API.

This incident has been reported resolved by our upstream provider, and we are also no longer seeing any issues, so we are also considering it resolved.

Our upstream provider has implemented a fix and transactions appear to be processing successfully now. We are actively monitoring the results.

Currently all payment transactions submitted through the Subscribe Pro Vault are failing. All other APIs and systems are operational. We have identified an issue with an upstream provider causing failing transaction requests. They have published an incident here: https://status.spreedly.com/incidents/1ly5j04m8mjs

Full service has been restored.

Our Merchant App and API are currently experiencing partial downtime due to a long running database update.

This incident has been resolved.

A fix has been completed and performance has returned to normal. We will continue to monitor.

We are currently investigating degraded performance of our Merchant App and API.

The issue causing a partial outage of our Merchant App and API is resolved.

We are currently investigating a partial outage of our Merchant App and API.

This incident has been resolved.

The issue with our upstream provider, Cloudflare, appears to be resolved. We are continuing to monitor.

We are continuing to investigate this issue.

We are currently investigating an outage of our Merchant App UI and API. Currently it looks like an issue with our upstream provider, CloudFlare. We will keep you updated as we learn more.

We identified an issue during our most recent deployment that resulted in traffic to our API and web UI not being processed. The relevant change was reverted and the issue has been resolved.

We are currently identifying an issue with our merchant and and API.

A fix has been deployed by Spreedly and the core issue has been resolved. Transactions are processing successfully.

We have implemented a stable workaround. The core issue is related to an SSL certificate on core.spreedly.com. We will update when the core issue has been resolved.

We are currently investigating an issue causing payment transactions to fail.

The issue has been resolved by our upstream provider.

A fix has been applied by our upstream provider. Services are currently up and being monitored.

This incident is due to issues at our upstream provider. We will keep you updated as it is investigated and resolved. https://status.platform.sh/

Network issue was marked resolved by Cloudflare at 4:25am eastern.

A fix has been implemented and we are monitoring the results.

We experienced a brief period of downtime (3-5 minutes) starting around 10:49pm eastern. The issue is related to a Cloudflare outage in Ashburn, VA. https://www.cloudflarestatus.com/ At this point, service has been restored and we are monitoring the issue.

Today our API and web UI were down for close to 15 minutes. The root cause was a long running database update which was necessary as part of an upgrade. When we estimated the length of time the update would need we were off by 1 order of a magnitude. Consequently the update took close to 15 minutes instead of a little over a minute and translated into downtime for the API and UI. These sort of database updates are something we do rarely. Never-the-less, we have identified methods we can use to do more extensive testing in the future, for similar updates. This would allow us to plan alternate, incremental ways to update the database and schedule those at off times.

This incident has been resolved.

A fix is in place and we are monitoring.

We are investigating a partial outage of our Merchant UI and API

This incident has been resolved.

We have fixed the issue causing some recurring orders not to be processed. We will continue to monitor for any further issues.

We are investigating reports that some recurring orders are not being processed as expected.

This incident has been resolved.

A fix has been implemented and we are monitoring. If you continue to experience issues accessing Merchant UI functionality, please contact support@subscribepro.com

We are currently investigating a 500 Internal Server Error that is preventing access to subscription and customer management pages in our Merchant UI.

We had an outage this morning that affected all Subscribe Pro customers. The root cause was a disk that filled up on our transactional database system. There were two other systems failures which caused this issue to lead to over an hour of downtime: * Our designated on-call personnel failed to respond as promptly as required. * We were operating with a mis-understanding of the auto-scaling settings around the disk that filled up. We have taken action to address both of these system issues: * We have fine tuned our procedures for on-call personnel and increased our training frequency. * We have adjusted and tested the relevant auto-scaling settings for database disk. We take any down time very seriously and have used this as opportunity to improve our systems and processes so that this type of issue can’t happen again. Thank you.

This outage has been resolved. Please contact support@subscribepro.com if you experience any further issues accessing our platform web UI or API.

A fix has been implemented and we are monitoring.

We are currently investigating an outage of our platform UI and API.

Cloudflare has considered this issue resolved. Affected services have resumed normal operation. You can find more information about this issue here: https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr

Cloudflare has implemented a fix. Service appears to be restored. We will monitor and provide an update when the issue is resolved.

Issue was determined to be caused by Cloudflare outage - https://www.cloudflarestatus.com/

The root cause was a Cloudflare outage. Cloudflare service has been fully restored. Subscribe Pro hosted code is now fully operational as well. https://www.cloudflarestatus.com/

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

Issue was determined to be caused by Cloudflare outage - https://www.cloudflarestatus.com/ We have temporarily bypassed the issue and hosted code should be available.

We're currently investigating a partial outage of our hosted widget code. The cause seems to be related to a Cloudflare connection issue.

The partial outage of recurring order processing has been resolved.

We have identified the issue and put a fix into place. We will continue to monitor to ensure everything continues to process as expected. If your accounts' subscriptions were delayed, they should process soon.

We are currently investigating a partial outage of our recurring order processing.

We experience brief intermittent downtime due to an issue with our logging cluster and application logging queue. Full service has been restored.

We are currently investigating a partial outage of our platform UI and API.

API and UI are operating normally.

We are currently investigating intermittent 502, 503 and 504 errors.

The outage was caused by a misconfigured SSL certificate chain. Our hosting provider replaced / reconfigured a number of certificates and unfortunately our environment was affected. This caused some traffic coming from Cloudfront to our production web clusters to fail and thus caused them to be unreachable from some locations.

The Cloudfront CDN appears to be operating normally for our services now.

Our upstream hosting provider continues to investigate the Amazon Cloudfront issue.

Our monitoring has detected serious issues with our cloud infrastructure provider which affects all sites using Cloudfront CDN. Our operations team has been notified and they are investigating the issue with our provider. We will update the status shortly but we do not yet have an estimated resolution time.

We are currently investigating a partial outage of our API and Web UI. The root cause seems to be our upstream hosting provider.

This incident has been resolved.

The issue has been resolved. We will continue monitoring.

We are currently investigating a partial outage of the Subscribe Pro Web UI and API.

This incident has been resolved.

We are continuing to monitor for any further issues.

We believe the issue is resolved. We will continue monitoring.

We are currently investigating a partial outage of the Subscribe Pro Web UI and API.

We investigated the root cause on Monday’s partial outage. In short some long running reporting queries were able to overwhelm database connections and make the platform unresponsive. Today we have implemented a system to stop long running queries and prevent them from impacting overall system performance. With this new system, an outage similar to Monday’s will no longer be possible.

This issue causing the partial outage of our Web UI and API has been resolved. The issue was related to our tabular reporting, which has been temporarily disabled until the root issue is addressed.

We believe the issue is resolved. We will continue monitoring and provide a post-mortem once we fully understand the issue.

We are currently investigating a partial outage of the Subscribe Pro Web UI and API.

Unfortunately this morning Subscribe Pro had an extended period of intermittent downtime for our subscriptions platform. We do sincerely apologize to all our clients for this for this outage. Historically, since our launch 4 years ago, we have maintained extraordinarily high availability. Although today tarnished that record a bit, we don't intend to let anything similar happen in the foreseeable future. What Went Wrong ---- 1. We deployed updates to our software this morning around 8:05am eastern time. These updates included some code changes and an additional index in our database. 2. Within 10 minutes after the deployment, we started noticing performance issues with our database server. 3. We began investigating the issue immediately. Initially we traced the performance issues to the creation of a the new database index. 4. We instituted emergency procedures to add more resources to our production environment and continued investigating the root cause. 5. Eventually we were able to determine the root cause was related to a code change in the software. And to determine that the database index change was a red herring. Moving Forward --- We understand how important 24x7x365 availability is to an eCommerce business and we do put in a lot of effort to make sure your sites achieve that. Today we were not successful in that, but we are using the experience to learn what we can do better in the future.

We have identified and resolved the root cause of the service outage. Please stay tuned for a full report of what went wrong.

We are experiencing a platform outage and are actively working to restore service.

Unfortunately we experienced around 30 minutes of downtime for our platform this afternoon. The issue occurred while we were deploying some new features to our platform to support our new Magento 2 integration. We estimated that the changes had low risk, but unfortunately there was an unexpected issue with our database migrations. Normally we should be able to recover from this type of an issue during a deployment within a few minutes. However in this case, all of the right people weren't immediately available and there were some questions about our procedures. We are revisiting our deployment process again and hope to avoid a similar situation in the future. We understand how important it is to you, our customers, that Subscribe Pro avoid any downtime possible as we are a key part of your eCommerce revenue stream.

The subscriptions platform was down between approximately 5:20 and 5:45pm eastern time.

We experienced downtime today between 1:30 and 1:52 eastern today. First, I'd like to apologize on behalf of our whole team. We understand that our platform is important and that you, our customers are relying on our platform to be up and running so you can accept orders and payments from your customers. The root cause was a test we ran on our database, to prepare for an upcoming infrastructure migration. We were running a test on our database to prepare for migration of our platform to a new hosting provider. We simply didn't anticipate the risk of running this particular test. We clearly should have anticipated the risk in this case. Currently we're rethinking our process of risk assessment for any actions we undertake on our production infrastructure. We plan to formalize this process this week. By making the risk assessment process explicit and formal, we expect to avoid issues like this one in the future.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

We have determined the cause of the slow running recurring order processing jobs and have taken steps to make the system run smoothly in the short term. We will be taking additional steps to prevent this from occurring in the long term, as well. Order generation has been turned back on, and our system appears to be caught up at this time. Please reach out to support@subscribepro.com if you continue to notice any issues or delays with order processing.

We are continuing to investigate the issue with slow recurring order job processing. We have disabled order jobs while we look into this issue further.

We are currently investigating an issue with delayed order processing for recurring orders.

At 1:30pm EST we identified an issue causing a delay in automatic ordering. Some order jobs may have been delayed. We have resolved the issue and will continue to monitor.

The problem with slow and delayed subscription processing has been resolved. We are still investigating the root cause.

We are investigating slow and delayed recurring order processing across all Subscribe Pro environments. We are investigating the issue and will report when everything is back to normal.

As of 5:08pm ET AWS has considered the issues resolved.

Due to currently ongoing issues with the Amazon S3 service, specifically targeting the us-east-1 server, some impact may be seen in our platform. In particular, we have noticed that some transactional emails are failing to be sent. We will continue to monitor the situation and provide updates as they become available. You can find more information at the Amazon AWS status site, here: https://status.aws.amazon.com/

This incident has been resolved.

We have some customers who are experiencing API authentication issues. In all cases this should be resolved by flushing Magento caches. We deployed a change to our platform this morning which upgraded the OAuth2 authentication system. This system is currently used by our Magento 1 extension. The upgrade invalidated all existing OAuth access tokens. The expectation was that the invalidated access tokens were cached for no more than 5 minutes by Magento 1 clients. However it appears that in a few cases, access tokens are cached longer. The issue should be resolved by flushing all caches in the Magento admin panel, specifically the 2 Subscribe Pro related caches.

Authorize.NET's issue affecting API connectivity today has now been resolved. https://status.authorize.net/#

Authorize.NET has implemented a resolution on their end and are monitoring to ensure that the issue is resolved.

Subscribe Pro has become aware of an issue that Authorize.NET has been experiencing since 11:35am EST that has been affecting our customers who use them as their payment gateway provider. The most current status on the Authorize.NET issue can be tracked at the below link. https://status.authorize.net/#

Thank you for your patience. The data in the Subscribe Pro Dashboard module is now available once again to view.

We are currently investigating an issue with the Subscribe Pro Dashboard module not displaying data to our customers. Please note that this does not affect the Subscribe Pro platform or API performance and functionality.

The site has returned to normal.

We are currently experiencing an unexpected outage. We are investigating.

This afternoon our platform started experiencing slow performance and intermittent 502 errors. The root cause was a difference in how the database engines on our old and new environment responded to specific indexes. We have resolved the issue by adding additional indexes.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

Today we have upgrade our reporting capabilities. Your dashboard will show no data until our new aggregate data indexes have been rebuilt. We expect full data to be available on the Dashboard by Monday. In the meantime, all data is still available via the Reporting menu. We experienced approximately 1.5 minutes of downtime between 10:20 and 10:22am eastern this morning during the upgrade.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.

This incident has been resolved.

We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.