Historical record of incidents for FastSpring
Report: "Paypal order failure"
Last updateBetween 1:50 AM and 5:50 AM PT, we observed a decline in PayPal orders, preventing some customers from completing their transactions. The issue was caused by a system-related problem on PayPal's end. As of now, all orders are processing normally.
Report: "API Limit"
Last updateWe’ve identified a 30-minute window where almost all API calls were blocked, from 11:36 PM on 02.10 to 12:06 AM on 02.11. However, after this period, API traffic appears to be functioning normally. At this time, the issue does not seem to be ongoing, but we are continuing to investigate the cause of the blockage during that specific window. We will provide further updates as we gather more information. Thank you for your patience.
We’ve identified a 30-minute window where almost all API calls were blocked, from 11:36 PM on 02.10 to 12:06 AM on 02.11. However, after this period, API traffic appears to be functioning normally. At this time, the issue does not seem to be ongoing, but we are continuing to investigate the cause of the blockage during that specific window. We will provide further updates as we gather more information. Thank you for your patience.
We are aware that some sellers are experiencing API rate limiting issues, which are disrupting day-to-day business operations. FastSpring is actively investigating this as a priority and working towards a resolution. We will provide updates as soon as more information becomes available.
Report: "PayPal Down"
Last updateThis issue has been resolved by PayPal.
PayPal appears to be down. We are investigating.
Report: "Paypal Outage"
Last updateWe see consistent signs of recovery for Paypal orders. Payments are operational.
We are seeing signs of recovery. Paypal is reporting that they have mitigated the issue and are also seeing recovery. We will continue to monitor.
We are investigating the impact of a Paypal outage starting Oct 16, 17:35 UTC. You may also refer to Paypal's status page here - https://www.paypal-status.com/incident/production
Report: "Google Pay Unavailable in India"
Last updateWe have encountered challenges with some buyers purchasing with Google Pay on mobile devices in India. To ensure an optimal buyer experience, we have disabled Google Pay in India; however, this is isolated to new transactions and should not affect any active subscriptions purchased using Google pay from India prior to the current challenge.
Report: "KakaoPay transactions not going through."
Last updateIn light of recent global news regarding wallet data, we are working with our partners to validate and certify our Kakao Pay offering, therefore we have disabled the payment method until further notice.
Further investigation revealed that KakaoPay has been disabled by our Payment Service Provider. We are working with them on the future course of action.
Report: "Payment Methods not available for Classic store"
Last updateThis incident has been resolved
We have a number of reports of Credit Cards not being presented a payment option for customer on Classic stores only
Report: "Slow response time in contextual stores from (09:00 UTC)"
Last updateThis incident has been resolved.
The incident has been resolved with no further issues observed since 9:30 UTC. Our team is continuing to monitor the situation closely, and we will keep the status page updated with any additional information.
Increased load times have been reported on Contextual Stores as of 09:00 UTC. We are currently investigating the cause of this issue and will keep this page updated.
Report: "Brief payment collection interruption"
Last updateBetween 18:11 UTC - 18:23 UTC, as part of our deployment, we noticed a brief interruption to payment collection from buyers. This has been restored.
Report: "Intermittent API timeouts, SBL timeouts, Login Timeouts."
Last updateThe issue with intermittent API timeouts, SBL timeouts, Login Timeouts have been resolved.
System metrics are stable. We will continue to monitor.
We are currently experiencing and investigating intermittent API timeouts, SBL timeouts, Login Timeouts.
Report: "Unable to accept PayPal EUR from European Economic (EEA) region"
Last updateThis incident has been resolved.
Paypal EU transactions are going through successfully. We are continuing to monitor the situation.
Processing of PayPal EUR transaction from EEA region is back to normal. We are still investigating the issues with last nights recurring charges. Thank you for your patience and co-operation at this time.
We are continuing to investigate this issue.
We are currently investigating this issue and will provide an update once there is more information.
Report: "PDF download on order receipts may result in error"
Last updateThis incident has been resolved.
This has been resolved.
Problem identified. We continue to monitor before marking incident as resolved.
PDF download on order receipts may result in error. Please use browser print/save to pdf as a workaround
Report: "Rebills running behind schedule"
Last updateAll subscription rebills are now up to date
The issue has been identified that that rebills are running behind schedule but we expect the rebills to catch up within the next 24-36 hours.
Report: "Stuck Paypal Orders"
Last updatePaypal orders are no longer getting stuck. Cleanup efforts are TBD.
The issue has been identified and a fix is being implemented. Due to the nature of the fix, it will require some time to put into place and test. We appreciate your patience.
Note: When applicable, a license is generated for each affected order. If your storefront settings are set to display the license on the order completion page, we recommend temporarily disabling this. For web storefront, navigate to Storefronts > Web Storefronts > Settings > Completion page. Uncheck the box and click Save at the top right. For popup storefront, navigate to Storefronts > Popup Storefronts > Settings, scroll down to the Completion Page section at the bottom, uncheck the bottom checkbox, and click Save at the top right. Licenses are not being delivered to the customers via email.
We continue to work with Paypal to resolve the issue. Note: Stuck orders would have been declined if allowed to move towards completion. None of these transactions would have resulted in income.
We have confirmed only a very small subset is affected. We are working with Paypal to resolve the issue.
Some small subset of Paypal transactions are not moving into a completed status. We are investigating the extent and time period of the problem.
Report: "Account Management and Email Disruption"
Last updateWe are aware of disruptions to the account management page and backend functions like creating products. We are also looking into delays in sending customer emails like receipts. The root cause was configuration changes to our infrastructure that have been reverted, and the problem is not ongoing.
Report: "Login Issues"
Last updateThis incident has been resolved. Effects include API timeouts, increased latency on stores, and an elevated error rate on stores. Some orders may have been delayed or affected
Problems have dissipated, but we are still determining the impact and affected components.
We are currently investigating issues related to login problems to https://app.fastspring.com/
Report: "Login and Latency Issues"
Last updateThis incident has been resolved.
We are currently experiencing API latency due to an issue with AWS. Log in, app speed, and API access may be impacted.
Report: "Some platform login and checkout issues"
Last updateIncident is being marked as resolved and root cause will be posted once it is determined.
All functions should now be working properly. We are investigating the root cause.
Between 19:20 and 19:40 UTC, some checkout sessions and FastSpring platform login sessions failed.
Report: "Database Problems"
Last updateRoot cause known and mitigated against future issues. We apologize for any disruptions.
We are continuing to monitor for any further issues.
All systems are responding. We will continue to monitor and get to the root cause.
All applications are timing out or slow. We are investigating.
We are experiencing an issue with our database, which is impacting services. More details to follow.
Report: "Classic Invoice Product Names"
Last updateAll invoices should display correctly. The issue is resolved.
We are working to clean up already viewed invoices, however, the problem is no longer occurring.
The root cause is now known and a fix for expected invoices is expected tomorrow.
Invoices viewed from now will include the product name. Previously viewed invoices may still not include product names. We are working to fix these.
Product names are not properly showing on invoices for Classic stores.
Report: "AWS degradation causing email failure"
Last updateThe AWS Lambda issue has been resolved, although it is throttled. Our email processing has resumed. We are continuing to monitor the services. Our queuing system is currently processing backed up events, including webhooks, however because of throttling from AWS on the backed out events, there could be delays in processing the events.
Update 1: the issue is affecting webhooks as well as fulfillments.
AWS is experiencing degradation which is causing our emails to fail. We are investigating the issue.
Report: "Popup and embedded secure payload failures"
Last updateFrom 2:45-6:03 pm PST, transactions on popup and embedded storefronts that utilized secure payloads were unsuccessful. This was due to some changes made to webhook payloads during a release, which caused conflicts with secure payloads. However, we have resolved the issue and all transactions have been processed successfully. We are currently conducting further investigation to determine the root cause and take the necessary measures to prevent such incidents from happening again in the future.
Report: "Store Errors"
Last updateBetween 17:53 and 18:03 UTC some stores sporadically experienced errors. The cause was quickly identified and mitigated with a configuration adjustment.
Report: "SVB Payout"
Last updateThis incident has been resolved.
To all our Customers, Given the current circumstances with Silicon Valley Bank (SVB), FastSpring will not be making any payouts to SVB accounts. Over the next 24 hours, FastSpring will contact each impacted seller individually to work out other arrangements.
Report: "Tags incorrect in subscription.activated webhook"
Last updateData cleanup is in process. Please contact support if you need subscription.activated webhooks regenerated from orders during this timeframe or if you have additional issues or concerns. We will not be refiring proactively to avoid causing you problems with duplicates, but we can do it on a case-by-case basis. Correct data can also be pulled via API call or we can supply a csv of the data.
We have fully identified and verified the root cause of the problem. We have also verified the data is not permanently lost. We are confident we will be able to regenerate the appropriate data and send the webhooks with the correct tags. We should have a cleanup plan in place with tools within 24 hours.
We have rolled back the code that caused this change in behavior. Our Engineering team is working on mitigating actions for the impacted susbcription.activated webhooks. If the order.completed webhook is enabled at store-level on your account, we recommend using it as source of truth for the order-level tags of subscriptions created during the incident timeframe.
Order-level tags are being passed incorrectly to the subscription.activated webhooks. It is important to note that order-level tags kept being passed correctly to the order.completed webhooks.
Report: "SEPA Payment Processing Disabled"
Last updateWe expect to release an improved SEPA solution later in 2023.
We are continuing to work on a fix for this situation.
We are continuing to work on a resolution for this issue. We are waiting on our banking partner to give the go-ahead to re-enable the payment method. We expect SEPA to still be unavailable for at least a week longer or perhaps more. We will keep you updated here.
We expect the outage might extend until approximately the end of next week (Sept-16). Any orders not currently fully settled should complete when the payment method is back online. We continue to post updates here.
SEPA payment processing has been removed as a payment option for new orders because of an issue with our processing partner. We will re-enable SEPA as a payment method when available. Subscription rebills are not affected. We project the problem will be resolved Monday. We will update as needed on this page.
Report: "APIs Unresponsive"
Last updateAPIs were unresponsive from 12:35am PST to 2:23AM PST. The issue is now resolved and is not ongoing.
Report: "API /accounts endpoint failures"
Last updateThe solution has been tested and verified.
A fix has been implemented and we are monitoring the results.
Some attempts to POST to the /accounts API endpoint are returning the following error code with a 400 HTTP status response - “email is passed as null. API cannot create account with empty email” We have identified the root cause and are actively working on a solution.
Report: "Incorrect Payment Notification"
Last updateAll incorrectly reported payout data has been reset and account balances are now correctly reflected
An incorrect payout notification sent on September 15 related to the sales for the week ending September 14th was sent in error. Although the amount stated as the payment has been deducted from your running balance reflected in the FastSpring platform, the funds were not deducted from your account and will not be paid out early. These funds will be included with your next scheduled payout as is appropriate. The underlying issue has been identified and resolved and will not impact upcoming payments. We are looking into the root cause of the problem which resulted in this process running out of schedule and are working on correcting the running balance.
Report: "Price edit issues"
Last updateStarting on May-18 through May-19, editing the price on a non-subscription product displayed the wrong dialogue. Saving the price unintendedly caused the non-subscription product to be converted into a subscription product. We have fixed the error and the problem should not continue to occur. We are attempting to identify problematic edits and revert those products back to non-subscriptions.
Report: "API Errors"
Last updateThis incident has been resolved. We will continue to monitor.
A fix has been implemented and we are monitoring the results. Calls to the API, backend login, checkout errors, and other platform functions may have responded slowly or timed out. The time range (in UTC) of the incident was 08:30 to 09:42 and 10:33 to 14:10
We are continuing to investigate this issue.
We are currently investigating an issue with our APIs that has led to an increase in error responses.
Report: "Rebill Issues"
Last updateAll active rebills from 4/18 and 4/19 have been completed. Any rebill failures from 4/18 or 4/19 will be retried normally on the next rebill run on 4/20. We continue to monitor the system to ensure timely rebill runs. The root cause remains under investigation.
Missing rebills from Apr-18 and Apr-19 are currently being rerun. The root cause is under investigation, but we have made changes to our architecture intended to avoid ongoing problems with our next rebill batch. Some subscriptions where a rebill was attempted on Apr-18 have entered an unusual state. We are working on a remediation plan to return the subscriptions to good standing.
We are currently investigating additional issues with some subset of rebills going back to Apr-18.
Report: "Missing rebills or rebill delay"
Last updateAny missing subscription rebills will be processed later today or tomorrow.
Not all subscription rebills from Apr-18 have been processed. We have fixed the root cause.
Report: "Missing Webhooks and Duplicate Buyer Notifications"
Last updateRoot causes have been identified and solutions are applied. New webhooks will now fire as expected. We will be working towards identifying missing events. Notifications to customers will not continue to be duplicated going forward.
We are continuing to investigate, and we have been testing out multiple solutions. We appreciate your patience.
We have identified the root cause of the issue. Solutions are currently being tested. We have also determined that multiple notifications were sent to buyers for some subset of orders. The root cause is the same.
We are currently investigating the issue.
Report: "API Errors"
Last updateThis incident has been resolved. We will continue to monitor.
We have deployed a solution. We will monitor the outcomes.
We have identified the source of the errors. We are testing a solution at this time.
We are seeing some intermittent errors on our platform that may cause some API calls to periodically fail.
Report: "Order-related webhooks not automatically sending"
Last updateWe have confirmed that the issue causing unsent webhooks is resolved and no new webhooks are affected.
We are currently investigating an issue where order-related and subscription-related webhooks were not automatically being sent starting on 16 January and ending on 17 January 2021.
Report: "CVE-2021-44228: Log4j2 Security Vulnerability"
Last updateFastSpring is actively following the security vulnerability in the Apache foundation's open source library Log4j2 utility CVE-2021-44228 (https://nvd.nist.gov/vuln/detail/CVE-2021-44228). The vulnerability could allow attackers to perform remote code execution on applications that use the affected library. The FastSpring team was able to react very quickly to this exploit and has updated all applications to include the latest fix provided by Apache https://logging.apache.org/log4j/2.x/security.html on Dec 10. In addition, the team also implemented additional extensive measures in our web application firewall to thwart any malicious traffic that may try to leverage the vulnerability. We will continue to monitor the situation and provide updates as necessary.
Report: "Service Degradation"
Last updateAll services are fully functional.
We are monitoring based on updates from our cloud service provider.
We are still working to determine the exact impact on our services. At this time our best understanding is - Stores & Checkout = Degraded performance - orders complete and checkout loads - possible email and invoice delays API = Unknown impact, presumed to be down (investigation underway) Jobs = Partial impact on emails, invoice, bank payment settlement (Delays expected) App/Dashboard = Partial impact (investigation underway)
We are experiencing degraded performance with our cloud service provider. Customers may experience issues while accessing our platform. While the issue is outside of our full control, we are monitoring the situation.
Report: "Login issues to backend (app)"
Last updateSome sellers attempting to log into app.fastspring.com were receiving errors between 09:30 - 10:40 UTC. Resources have been reprovisioned to correct the disruption.
Report: "Login issues to backend (app)"
Last updateSome sellers attempting to log into app.fastspring.com were receiving errors between 13:45 - 16:15 UTC. Resources have been reprovisioned to correct the disruption.
Report: "Webhook Issues"
Last updateDue to record sales volume, our rebill server experienced added load which resulted in delayed or missing webhooks for subscription rebill events. Webhook failures are designed to retry automatically; however, webhooks related to this event were manually re-delivered which was done using the many tools at hand. We have taken corrective action to scale our architecture and have added additional related monitoring.
Report: "Webhook Issues"
Last updateDue to record sales volume, our rebill server experienced added load which resulted in delayed or missing webhooks for subscription rebill events. Webhook failures are designed to retry automatically; however, webhooks related to this event were manually re-delivered which was done using the many tools at hand. We have taken corrective action to scale our architecture and have added additional related monitoring.
Report: "Webhook Issues"
Last updateDue to record sales volume, our rebill server experienced added load which resulted in delayed or missing webhooks for subscription rebill events. Webhook failures are designed to retry automatically; however, webhooks related to this event were manually re-delivered which was done using the many tools at hand. We have taken corrective action to scale our architecture and have added additional related monitoring.
Report: "Login issue affecting some customers"
Last updateThis incident has been resolved.
The issue has been identified and a fix has been implemented. Anyone who was previously experiencing issues can now log in normally.
We are investigating a platform login issue which affects some customers with accounts that are a few years old.
Report: "Classic Store Rebill Issue"
Last updateWe have resolved the issue with classic subscriptions failing to complete re-bill that affected May 7th re-bills. Subscriptions have now rebilled. We continue to monitor the issue.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue with Classic Store Rebills. We will update this issue as we have further information.
Report: "Popup cart loading issues"
Last updateThis incident has been resolved.
We have identified an issue where some popup carts are not loading. We are currently investigating the cause.
Report: "Some orders not being processed"
Last updateWe experienced a delay in processing some of our orders. This was a result of degradation of performance on of our servers. The issue is now resolved and all orders are processing as expected.
We have taken steps to ensure any processing delays are limited to a maximum of 15 minutes while we continue to search for a root cause.
We are continuing to investigate this issue.
We have identified an issue where some orders are not being processed immediately. We are currently investigating the cause.
Report: "Degraded Store Performance"
Last updateThis incident has been resolved.
We experienced an issue with a 3rd party service which led to some degradation of store performance. The problem has been addressed and is being monitored.
Report: "Slow loading or timeout with Store Builder Library"
Last updateOne of our features experienced higher failure rates due to an edge case. This impacted the capability of processing orders via SBL. We have identified the issue and resolved the incident. We are doing a root cause analysis of the incident and will take the necessary steps to avoid a recurrence of the issue.
We have found the problem and have put a solution in place, but it will take a few minutes for Contextual stores to return to normal. We are actively monitoring and update as soon as we are sure the issue is fully resolved.
We are continuing to investigate this issue.
We are noticing slow loading or timeouts with SBL, popup storefronts, and other components. We are currently investigating the issue and will provide updates here.
Report: "Support Portal Login"
Last updateWe have addressed configuration issues related to our customer support portal and it is now fully accessible. The root cause of the issue has been identified as unexpected side-effects from an update to a third-party system. We apologize for any inconvenience.
We are continuing to work on a fix for this issue.
We are currently experiencing technical difficulties with our customer support portal. As a temporary workaround please submit your support request to support@fastspring.com and include as much detail as possible. We will inform you when the portal is working again. Thank you for your patience; response and resolution times are currently slow due to the portal issue.