Is Spreedly Down Right Now? Check if there is a current outage ongoing.

This morning, from 11:43 AM EDT to 11:45 AM EDT, for about two minutes, we experienced a minor issue that caused intermittent 500 errors. Our team has fixed the problem, and we do not anticipate any further impact on customers.

The issue has been fully resolved, and the system is functioning as expected. No further incidents have been observed.

The issue with the missing stylesheet express-mobile-3.1.0.min.css causing the user interface to break in the Spreedly Express integration has been resolved. We are now actively monitoring the system to ensure the issue does not recur.

We are currently investigating an issue where the stylesheet express-mobile-3.1.0.min.css is returning a 404 Not Found error, causing the user interface to break in the Spreedly Express integration.

To ensure ongoing service quality, some database components will be undergoing maintenance during this time. No services will be taken offline as a part of this maintenance task, and our customers are not being asked to take any action in response to this maintenance task.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

# May 14, 2025 — HTTP 500 Errors and Google Pay Degradation ### ‌High-Level Summary On May 14, 2025 at 03:18 pm UTC, a configuration change was deployed to enhance application fault tolerance which, resulted in a brief period of HTTP 500 errors. While the breaking change was reverted within a minute of deployment, there were still instances of cached data which extended the impact for a total of 4 minutes of partial service outage, until 03:22 pm UTC. A secondary issue occurred as a result of the timing of this deployment coinciding with the regeneration of the Google Pay verification keys for decryption, which resulted in a 51 minutes degradation of Google Pay transactions until 04:09 pm UTC. ### What Happened A configuration change intended to enhance application fault tolerance to Redis failures inadvertently introduced serialization issues when accessing Redis keys. This resulted in new cache failing to be successfully read from the existing application instances, which resulted in HTTP 500 errors. Additionally, at the same time this change was deployed, the Google Pay verification keys were regenerated. Google Pay decryption stopped working due to the cached keys not having serialized properly. These keys then had to be deleted from the cache to enable the system to auto generate a new pair. ### Next Steps * Spreedly has implemented additional observability improvements via our automated alerting system to enhance monitoring of Google Pay services.

IMPACT STARTED AT: 11:18 AM EST IMPACT ENDED AT: 12:09 PM EST AFFECTED SYSTEMS OR SERVICES: Google Pay Transactions DETAILS OF THE INCIDENT: Google Pay Transactions Failing CURRENT ACTIONS: After closely monitoring Google Pay transactions and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. A post-mortem will be published. We apologize for any inconvenience or disruption.

AFFECTED SYSTEMS OR SERVICES: Google Pay Transactions DETAILS OF THE INCIDENT: Degradation with Google Pay transactions CURRENT ACTIONS: We’ve implemented a fix and are seeing Google Pay transactions succeed. Our team is monitoring the situation closely to ensure there are no additional impacts.

We are currently investigating some degradation with Google Pay transactions.

We are currently investigating some degradation with Google Pay transactions.

# **April 14, 2025 — Certificate Issue Impacts Core Requests** ‌ On April 14 at 3:47 PM ET, a platform-wide incident occurred that caused all core Spreedly requests to fail. The issue was identified and declared at 3:49 PM, mitigated by 4:13 PM, and fully resolved by 4:32 PM. Total impact duration was 26 minutes. During this window, merchants were unable to complete any transactions through the Spreedly platform due to system-wide connectivity failures. ## What Happened An intermediate certificate on our core service hosts expired April 14th at 3:47pm EST, which broke the certificate chain, preventing HTTP requests between Spreedly internal services and subsequently led to responses of HTTP 500 errors for all requests. ## Next Steps * Spreedly is completing an audit of all certificate chains to identify potential risks and update our catalog of expiration dates. * Spreedly has a plan in place to migrate the management of all certificate chains to our cloud provider.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

Spreedly has detected an issue that may impact customers, including potentially resulting in failed transactions. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available.

Spreedly has detected an issue that may impact customers, including potentially resulting in failed transactions. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available.

We are currently investigating an issue where the stylesheet express-mobile-3.1.0.min.css is returning a 404 Not Found error, causing the user interface to break in the Spreedly Express integration.

# March 15th, 2025 — Payment Method Storage Errors _On March 15th, 2025 at 06:12 UTC Spreedly systems encountered a system resource constraint which led to intermittent API call failures resulting in_ `404` _and_ `500` errors being returned to customers. _The unresponsive application containers were restarted allowing the system to return to normal at 06:47 UTC_ ## What Happened On 2025-03-15 between 06:12 and 06:47 UTC, calls to Spreedly’s payment method tokenization endpoints intermittently failed resulting in `HTTP 500` errors. Intermittent elevated rates of `HTTP 404` errors when attempting to use a payment method to transact were also encountered. These failures were the result of internal system resource limitations. ## Next Steps * Spreedly has implemented additional resource monitoring in our automated alerting system to prevent future overruns. * System resources have been adjusted to prevent similar situations in the future.

IMPACT STARTED AT: 2:12 AM EST IMPACT ENDED AT: 2:40 AM EST After closely monitoring Spreedly Core Services and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. A post-mortem will be published. We apologize for any inconvenience or disruption.

As of 3:10 AM EST, our system availability has been fully restored and we are seeing stability return. We have implemented a fix and stabilized Spreedly Core Services. Our team is closely monitoring the situation to ensure no further impact.

As of 3:10 AM EST, our system availability has been fully restored and we are seeing stability return. We have implemented a fix and stabilized Spreedly Core Services. Our team is closely monitoring the situation to ensure no further impact.

We are investigating an elevated rate of 5xx errors affecting Spreedly core services starting at 01:36AM EST. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Our team is actively working to identify the root cause and mitigate the impact. We will provide further updates as soon as we have more information. Thank you for your patience.

# February 16, 2025 — Intermittent API Failures Resulting in 404 and 500 errors ‌ _On February 16, 2025 at 08:55 UTC Spreedly systems encountered a system resource constraint which led to intermittent API call failures resulting in_ `404` _and_ `500` errors being returned to customers. _The unresponsive application containers were restarted allowing the system to return to normal at 09:55 UTC._ ### What Happened On 2025-02-16 between 08:55 and 09:55 UTC calls to Spreedly’s payment method tokenization endpoints intermittently failed resulting in `HTTP 500` errors. Intermittent elevated rates of `HTTP 404` errors when attempting to use a payment method to transact were also encountered. These failures were the result of internal system resource limitations. ### Next Steps * Spreedly has implemented additional resource monitoring in our automated alerting system to prevent future overruns. * System resources have been adjusted to slow similar situations in the future.

After closely monitoring Spreedly Core Services and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the root cause of the incident and any residual impact. A post-mortem will be published. We apologize for any inconvenience or disruption.

As of 4:44 AM EST, our systems have been resourcing this issue, and we are seeing stability return. We have implemented a fix and stabilized Spreedly Core Services. Our team is closely monitoring the situation to ensure no further impact.

We are investigating an elevated rate of 5xx errors affecting Spreedly core services. This issue began at 3:56 AM EST. Our team is actively working to identify the root cause and mitigate the impact. We will provide further updates as soon as we have more information. If you have any questions or concerns, please <a href="https://support.spreedly.com/hc/en-us/requests/new">reach out</a> and our support team will be happy to assist.

# December 12th, 2024 — Payment Method Tokenization Errors Server maintenance to reduce log volume caused an internal service to become unavailable, causing payment method tokenization requests to fail. ## What Happened At approximately 17:45 UTC on December 12th, 2024, a logging configuration change made to reduce log volume resulted in an internal service becoming unavailable. Traffic was shifted to an alternate service during the outage window to reduce impact. At approximately 18:01 the internal service resumed normal operation and service was restored. Payment transactions were negatively impacted during this time. ## Next Steps Spreedly Engineering is investigating the coupling of the configuration change to the internal service and will be taking steps to prevent the issue from occuring in the future.

After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

From 12:47 to 12:58 PM EST, we encountered an issue that affected customers attempting to tokenize a payment method and transacting with a payment method, resulting in a 500 error. We have identified the problem and implemented a fix. Our team is closely monitoring the situation to ensure there are no further impacts. If you need additional assistance, please reach out to our Support team.

Spreedly has discovered an issue that could potentially impact new payment method creation. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We will provide an update as soon as more details are available.

# December 11th, 2024 — Internal System Errors Server maintenance performed during Spreedly’s regular change window to increase capacity to an internal service caused two brief periods of service interruption to APIs that access decrypted information. ## What Happened At approximately 19:15 UTC, Spreedly Engineering initiated a capacity expansion of an internal service used for data decryption. From 19:28 to 19:30, a first wave of failed API requests during the maintenance window. A second wave of failed API requests occurred from 19:39 to 19:41 UTC when rebalancing request traffic to the capacity-expanded internal service. Service was restored at 19:41 UTC, and the system was fully operational. API calls that required decrypted data were impacted during the outage timeframes. ## Next Steps Spreedly Engineering is improving internal observability, implementing automated monitors, and investigating the use of automation for scaling capacity in the future to prevent this issue from recurring.

While maintaining a backend service to increase capacity, we saw intermittent errors during the timeframes of 2:28 PM EST and 2:30 PM EST and again during the timeframes of 2:39 PM EST and 2:41 PM EST. Our internal monitors picked up this issue, and we resolved it quickly. We have since seen the system return to normal, and we do not expect any further customer impact. This incident is considered resolved. We apologize for any inconvenience this may have caused.

# December 11th, 2024 — Global Collect Gateway Transaction Failures A change to the Global Collect gateway integration resulted in transaction failures for a subset of customers. ## What Happened At 20:50 UTC, an update was made to the Global Collect gateway integration. Initial monitoring and observability did not surface issues. At 23:53 UTC, Spreedly Engineering began researching an elevated number of transaction failures on the Global Collect gateway. After investigation, it was determined that the prior update to the Global Collect gateway was causing the increase in transaction failures. At 01:02 UTC on December 12th, 2024, the change was reverted and service to the Global Collect gateway was restored. ## Next Steps Spreedly Engineering is reviewing and improving internal observability to better surface gateway-specific transaction failure rate anomalies.

We have identified an issue with a code change that was deployed around 3:35 pm EST that affected the Global Collect gateway. We have rolled back this code change and are now seeing successful transactions. After a period of monitoring, we have not seen any recurrence of this issue. The incident is being considered resolved. We will publish a postmortem on our status page as soon as possible. We apologize for any inconvenience this has caused.

Spreedly is investigating an issue that could potentially impact our services, leading to 404 errors on the Global Collect gateway. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We will provide an update as soon as more details are available. Thank you for your patience.

## December 10th, 2024 — CVV Caching Issue At approximately 15:36 UTC on December 10th, 2024, Spreedly Engineering attempted a code change to an internal service that inadvertently prevented Spreedly from storing CVVs for payment methods when a /recache request was made. The adverse change resulted in a partial outage of Spreedly’s transactional API between 15:36 UTC and 17:07 UTC. Service was fully restored at 17:07 UTC. After a period of monitoring, Spreedly’s systems are stable, and no further impact is expected on customers. ## What Happened At approximately 15:36 UTC on December 10th, 2024, Spreedly Engineering attempted a code change to an internal service that inadvertently prevented Spreedly from extending the time a CVV exists in the cache. The adverse change resulted in a subset of transactions failing for customers who rely on Spreedly’s CVV recaching mechanism to complete a transaction. Spreedly Engineering team quickly mobilized and identified the adverse code deployment as the issue and immediately began reverting the change. The code reversion was completed at approximately 17:07 UTC, with customers reporting no additional impact after the code reversion. ## Next Steps Spreedly Engineering is reviewing internal observability to determine how to automate the detection of such an issue in the future.

After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. A post-mortem review will be published. We apologize for any inconvenience or disruption.

Spreedly experienced an issue with Caching the CVV. Once identified, we immediately began working on a resolution. We’ve implemented a fix and stabilized the system. Our team is monitoring the situation closely to ensure there are no additional impacts. Please <a href="https://support.spreedly.com/hc/en-us/requests/new">reach out</a> to our Support team if you need any additional help.

Spreedly has discovered an issue that could potentially impact our services, leading to errors with caching the CVV. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We provide an update as soon as more details are available. Thank you for your patience.

After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

We experienced a brief period of system instability during which multiple endpoints were impacted. This issue resulted in intermittent errors and disruptions in service. Starting at 3:15 PM UTC our team promptly identified the root cause of the issue, and as of 3:20 PM UTC, the problem was resolved. Since then, we have not observed any recurrence of this issue. We are closely monitoring the system to ensure continued stability and are taking steps to prevent similar incidents in the future. If you have any questions or concerns, please <a href="https://support.spreedly.com/hc/en-us/requests/new">reach out</a> and our support team will be happy to assist.

After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

We experienced a brief period of system instability during which multiple endpoints were impacted. This issue resulted in intermittent errors and disruptions in service. Starting at 2:28 PM UTC our team promptly identified the root cause of the issue, and as of 2:33 PM UTC, the problem was resolved. Since then, we have not observed any recurrence of this issue. We are closely monitoring the system to ensure continued stability and are taking steps to prevent similar incidents in the future. We apologize for any inconvenience caused and thank you for your understanding. If you have any questions or concerns, please <a href="https://support.spreedly.com/hc/en-us/requests/new">reach out</a> and our support team will be happy to assist.

After investigation, we have determined that there are no systemic issues with 3DS processing. Properly configured 3DS transactions are completing as expected. Thank you for your understanding.

We are investigating the potential degradation in 3DS2 processing, with specific focus on transactions not progressing beyond the pending state.

Since 2024-09-26 11:17 AM ET this morning, we experienced an issue where a change to enhance how our Access Token validation is performed caused a disruption in services attempting to use the existing methods. At 11:29 AM ET, the change was fully reverted. All services may have been affected during this timeframe. After a period of monitoring, our systems are stable, and no further impact is expected on customers. We apologize for any inconvenience this incident has caused.

Since 2024-09-26 11:17 AM ET this morning, an issue has been detected causing 500 errors to our core transactional API. We have reverted to a stable release and are currently monitoring the system.

After completing our investigation, we can confirm that all systems are operational, and the reported issue is not affecting functionality. Everything is running as expected, and there is no disruption to service. Thank you for your understanding. Please feel free to reach out if you have any further concerns.

We are currently investigating an issue affecting data synchronization with app.spreedly.com. Please note that this issue is not transaction-impacting, and all transactions are processing as expected. We will provide updates as we have more information.

After closely monitoring the situation and confirming that payment method creation is now stable and functioning as expected, we consider this incident resolved. No further customer impact is anticipated. We apologize for any inconvenience this may have caused and appreciate your patience as we worked to address the issue.

We are continuing to monitor for any further issues.

We are currently investigating an issue affecting payment method creation. Our team has identified the root cause of the issue affecting payment method creation and have implemented a fix. We are currently monitoring the situation to ensure that everything is functioning as expected. We'll provide more updates as soon as they're available

After closely monitoring our Core Transactional API and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

We experienced increased latency in our Core Transactional API from 17:32 UTC until 17:39 UTC. We immediately mobilized resources to address the issue. API performance has been stabilized. Our team is actively monitoring the situation closely to ensure there are no additional impacts. Please reach out to our Support team if you need any additional help.

We are aware that our Core Transactional API is experiencing some issues which may be impacting customers. We are actively investigating and will update with further details.

From 2024-07-31 12:19:00 (time) EDT until 12:20:00 EDT this afternoon, or approximately one (1) minute, we detected an issue that resulted in minor intermittent 500 errors. Our team has implemented a solution to rectify the problem, and we are currently monitoring the system, but consider this incident resolved.

After closely monitoring our reporting systems and confirming that all systems are stable and functioning as expected, this incident is considered resolved. 
 We apologize for any inconvenience or disruption.

The Spreedly team has identified and implemented a fix to resolve the delay in our API listing endpoint and dashboard reporting. This delay did not affect the state or success of customer transactions, but reporting during this time frame may not be accurately reflected. This incident is being closely monitored, and we apologize for any inconvenience or disruption.

We have identified a gap in reporting for some customers listing recent transactions using the API listing endpoints, which may affect /v1/transactions/ and list Payment Methods /v1/payment_methods. We are aware of the issue and are working to implement a fix.

# May 30, 2024 —Delays in listing transactions using the API listing endpoints # Executive Summary During a routine maintenance event, a piece of software responsible for reporting activity to systems used to generate customer reports and API listing endpoints silently failed to start. During the 40 minutes in which the software was non-functional, API calls for listing transactions and payment methods created in this timeframe would not have been reported. This reporting issue did not affect the outcome of customer transactions, only the ability for Spreedly systems to report that data to customers. After correcting the issue, Spreedly backfilled all available payment method and transaction data to reporting systems. # What Happened On 2024-05-30 at 15:15 UTC, production traffic was switched to a newly updated system responsible for generating customer activity data. This system silently failed to produce the necessary information, which resulted in a data gap. Upon automated alerting at 15:37, the team halted their maintenance work and reverted the system to a known good state by 15:50. After confirming that all systems were functioning normally and that no transactions were impacted, the team gathered all available data and replayed the data into the system so that reporting systems would accurately reflect that data. ## Next Steps Spreedly will continue to work to eliminate the possibility of silent failures of this functionality, testing to prevent non-functional reporting to receive production traffic, and introduce additional automated monitoring that will more quickly identify any loss of reporting volume. ## Conclusion Spreedly values the trust our customers place in us to handle one of the most critical aspects of their business, and welcome the opportunity to discuss any questions, concerns, or comments you have regarding this incident. Please reach out to our support staff or your account management team with your concerns.

The Spreedly team has identified and corrected the cause for the gap in reporting data for some customers. Reporting is affected for both API and Dashboard users during the period of time from 2024-05-30 15:15 - 2024-05-30 15:55 UTC. This gap did not affect the state or success of customer transactions, but reporting during this time frame may not reflect those transactions. We have confirmed that reporting outside of the specified time range is fully functional. We will communicate when reporting data has been corrected via the post-mortem for this incident.

We have identified a gap in reporting for some customers listing recent transactions using the API listing endpoints, which may affect /v1/transactions/<token> and list Payment Methods /v1/payment_methods. We are aware of the issue and are working to implement a fix.

We are marking this item as resolved. After further review of data, SCA authentications are processing successfully.

We are currently investigating an issue related to Strong Customer Authentication (SCA) authentication rejections. Requests for SCA authentication are being rejected with the error message "Service: Invalid Json response"

After a period of monitoring, this incident is considered resolved. We appreciate your patience and apologize for any inconvenience. We will publish a post mortem related to this incident.

We are monitoring the degraded performance for some users accessing Spreedly ID via Single Sign-On (SSO) connections using using OpenID Connect (OIDC). We have implemented a workaround for users experiencing this issue. If you are still impacted, please <a href="https://support.spreedly.com/hc/en-us/requests/new">reach out</a> and our support team will gladly assist you with gaining access. We apologize for any inconvenience this issue has caused and are working on a longer term solution

We are currently investigating reports of degraded performance for Single Sign-On (SSO) users accessing Spreedly ID. Users may experience delays or timeout issues when logging in or accessing the Spreedly dashboard or certain features. Our engineering team is actively investigating the root cause of the issue. please <a href="https://support.spreedly.com/hc/en-us/requests/new">reach out</a> to our support team if you are experiencing any log in issues.

We observed degraded performance for some users accessing Spreedly ID via Single Sign-On (SSO). Our team has investigated and we have implemented a workaround for users experiencing this issue. If you are still impacted, please reach out, and our support team will gladly assist you with gaining access. We apologize for any inconvenience this issue has caused.

We are currently investigating reports of degraded performance for Single Sign-On (SSO) users accessing Spreedly ID. Users may experience delays or timeout issues when logging in or accessing the Spreedly dashboard or certain features. Our engineering team is actively investigating the root cause of the issue. We will provide updates as soon as more information becomes available.

### **March 27th, 204 — ID Server Misconfiguration** A server configuration issue resulted in the ID service being unable to handle all incoming requests, which led to intermittent availability of Spreedly services, some transaction impacts, and 500 type errors being served to customers. ### **What Happened** On 3/27, at 2:30 PM UTC, Spreedly engineers began scheduled maintenance of the Spreedly ID service which handles authentication and authorization for other Spreedly services. A redeploy of the ID service, following the maintenance, revealed a latent server misconfiguration which led to application issues on specific servers in the ID cluster. At 5:41 PM UTC, the Spreedly Core service began intermittently responding with errors, as the ID service was no longer able to handle all incoming authorization requests. At 5:47 PM UTC, Spreedly engineers observed the increase in 500s from ID and began investigating. The misconfiguration was discovered and corrected at 6:22 PM UTC at which point the ID service began to recover. Errors from Spreedly Core services ceased entirely at 6:26 PM UTC. ### **Next Steps** Spreedly engineers have corrected the configurations that contributed to the application issues and are working to determine additional improvements that would prevent reoccurrence. Spreedly engineers are taking steps to lower the alerting thresholds for the ID service to provide earlier notification of similar issues. ### **Conclusion** We apologize for the disruption of the service, our goal is to maintain a reliable service for our customers so avoiding these kind of events is one of our top priorities. We’ll drive our efforts to keep improving our services.

After a period of monitoring, this incident is considered resolved. We appreciate your patience and apologize for any inconvenience. We will publish a post mortem related to this incident.

We identified an issue that resulted in intermittent transaction failures. A fix has been implemented at 1:22 EST, we are monitoring the results. We apologize for any disruption to service.

We are currently investigating an issue with access to the Dashboard application and related intermittent 500 errors. We apologize for any disruption to service.

After careful monitoring, we consider this incident resolved. We apologize for any inconvenience or disruption.

A fix has been implemented and we are monitoring the results.

Since 2024-02-29 11:02 am EST, we have identified an issue causing some Braintree 3DS transactions in the challenge flow to remain in a pending state. Our team is actively working towards a solution. We value your patience as we work to resolve this issue.

After closely monitoring our ApplePay integration through Vantiv and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. A post-mortem review will be published. We apologize for any inconvenience or disruption.

We experienced an issue causing some transactions that use ApplePay through the Vantiv integration to fail. Once identified, we immediately began working on a resolution. We’ve implemented a fix and stabilized the integration, and our team is monitoring the situation closely to ensure no additional impacts. Please reach out to our Support team if you need any additional help.

Since 2024-02-29 11:32 am ET this morning, an issue has been detected, causing some transactions that use ApplePay through the Vantiv integration to fail. Other service degradation may be related. Our team is actively working towards a solution to rectify the problem, and we will keep you updated on any progress made in resolving the matter. We value your patience as we work to resolve this issue.

After closely monitoring our fix on AU Callback Deliveries and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

Since 2024-01-13, there has been an issue detected causing AU Callbacks to fail delivery. We’ve implemented a fix and stabilized AU Callback Deliveries and our team is monitoring the situation closely to ensure there are no additional impacts. Please reach out to our Support team if you need any additional help.

Since 2024-01-13, there has been an issue detected causing AU Callbacks to fail delivery. All impacted customers have been notified. Our team is actively working towards a solution to rectify the problem, and we will keep you updated on any progress made in resolving the matter. We value your patience as we work to resolve this issue.

We have closely monitored the situation and received official confirmation that the issue has been resolved. As a result, we consider the incident resolved and do not anticipate any further impact on our customers. We apologize for any inconvenience or disruption.

Since 2024-02-14 09:14 ET, there has been an issue outside of our platform detected with our 3DS Global Provider. A fix has been implemented, and our team is monitoring the situation closely to ensure there are no additional impacts. Please reach out to our Support team if you need any additional help.

Since 2024-02-14 09:14 ET, there has been an issue outside of our platform detected with our 3DS Global Provider. Our service provider is in the process of implementing a fix today to address the intermittent issues being noticed by some customers when conducting 3DS Global transactions with Visa. Once we've received confirmation from our service provider that the fix is in place, we will monitor the changes and communicate this status.

Since 2024-02-14 09:14 ET this morning, there has been an issue outside of our platform detected with our 3DS Global Provider. Our service provider has identified the source of the issue and is actively working to address. Spreedly continues to work with the provider on this issue and our team is monitoring the situation closely to ensure there are no additional impacts. The next update will be provided as soon as a material update is available. Please reach out to our Support team if you need any additional help.

Spreedly identified an issue outside of our platform that could impact our services with 3DS. This may lead to degraded performance with 3DS Global. We are taking all necessary measures to investigate the matter with our service provider to determine the extent of the situation. We will provide an update as soon as more details are available. Thank you for your patience.

We have mitigated the 1% errors seen with the BIN MetaData service after a period of monitoring and are taking further steps to make our system more resilient. We consider this incident resolved. We apologize for any inconvenience.

We are continuing to investigate and are working towards mitigating this issue. We apologize for any inconvenience.

We are investigating an issue related to the BIN metadata service. This is a service which provides data in the Spreedly response, but does not affect transaction success rates. We are seeing that in 1% of requests which access payment methods, the bin_metadata portion of the response ‘BIN metadata service unavailable.’ will be returned.

We are currently investigating this issue.

This incident is considered resolved after closely monitoring our dLocal gateway connection and confirming that all systems are stabilized and functioning as expected. No further customer impact is expected. We apologize for any inconvenience or disruption.

We recently encountered a technical issue with Dlocal that resulted in errors for customers. Once identified, we immediately began working with the gateway on a resolution. We are pleased to report that we are now seeing successful transactions with Dlocal. Our team is monitoring the situation closely to ensure there are no additional impacts. Please reach out to our Support team if you need any additional help.

Spreedly has identified an issue outside of our platform that could impact our services with Dlocal. Our team is actively working towards a solution with the gateway to rectify the problem, and we will keep you updated on any progress made in resolving the matter. We value your patience as we work to resolve this issue.

Spreedly has identified an issue outside of our platform that could impact our services with Dlocal. This may lead to degraded performance with all transactions with this gateway. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We will provide an update as soon as more details are available. Thank you for your patience.

At 16:10 UTC a change was deployed to the Spreedly iFrame that introduced errors with Gateway Specific 3DS. A fix has been implemented at 18:52 UTC and the issue is now resolved.

After closely monitoring id.spreedly.com and dashboard.spreedly.com and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

Spreedly had detected an issue that prevented access to id.spreedly.com and dashboard.spreedly.com. Since that time the issue has been resolved, and we apologize for any inconvenience it may have caused. We will be working with one of our partners to determine the cause of the issue. Thank you for your patience and understanding.

**Summary** On 9/20/23, a third-party CDN provider had internal issues connecting to AWS data centers in Columbus, OH. This caused increased timeouts for transactions originating from these locations. **What Happened** * **20:37 Spreedly declares SEV-2 incident and notifies customers** * 21:01 - Fastly - Status posted and subscribers notified of elevated errors in CMH POP * 21:21 - Fastly - Remediation plan for third party service provider issue identified * 21:26 - Fastly - Remediation plan deployed * 21:30 - Fastly - Incident confirmed to be fully mitigated * **22:26 - Spreedly considers incident to be resolved** Based on [Fastly’s Statuspage](https://www.fastlystatus.com/incident/376079) resolution on this incident: “During this incident, customers would have experienced increased latency and elevated 5xx errors in our Columbus \(CMH\) POP, as well as, elevated errors when attempting to access configuration management and API services through our Control Plane from 18:40 to 21:30 UTC.” **Next Steps** No additional steps are required at this time **Conclusion** This is an unfortunate third-party service provider outage, and Spreedly has been in communication with the provider to determine the reason for the delay in notification.

This incident has been resolved.

We have closely monitored the situation and received official confirmation from our third-party provider that the issue has been resolved. As a result, we consider the incident resolved and do not anticipate any further impact on our customers. We apologize for any inconvenience or disruption.

Since 2023-09-20 2:41 PM ET this afternoon, there has been an issue outside of our platform detected with regional network service. Traffic has been rerouted in this region, and we continue to monitor traffic. Please reach out to our Support team if you need any additional help.

Spreedly has discovered an issue that could potentially impact our services, leading to Increased timeouts for a small number of transactions. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We provide an update as soon as more details are available. Thank you for your patience.

Spreedly has monitored the results of the deployed fix and have now deemed this incident to be resolved.

The issue has been identified and a fix is being implemented.

We are investigating an issue with the Spreedly Dashboard.

This incident has been resolved.

A fix has been implemented and are confident the issue is resolved. We are continuing to monitor our systems closely.

Spreedly is aware that some customers received 500 errors for PMD transactions. We have identified the fix and it has been implemented.

At 2023-08-31 11:18:00 AM ET this morning, a change was made causing 503 Errors. After rolling back our changes and close monitoring, we have confirmed that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. **A post-incident review will be published.** We apologize for any inconvenience or disruption.

We have closely monitored the situation and received official confirmation from Fastly that the issue has been resolved. As a result, we consider the incident resolved and do not anticipate any further impact on our customers. We apologize for any inconvenience or disruption.

A Fix has been implemented by Fastly and they are monitoring results Please reach out to our Support team if you need any additional help. For additional information: https://www.fastlystatus.com/

We continue to see no worsening symptoms of an impact from the outage.  We'll continue to monitor Fastly Status for resolution and intend on updating Spreedly status to resolved when Fastly has reported as resolved or if this moves into more formal incident territory.  We are reporting our systems as operational alongside the status page post.  Support is keeping watch on the inbound queue. We will provide an update as soon as more details are available. Thank you for your patience.

Spreedly is monitoring an issue we’ve identified with our third-party CDN, Fastly, who is reporting an issue that we believe is resulting in a very small number of intermittent 503 error responses. We're in communication with Fastly and are closely monitoring the situation. Fastly has communicated the issue has been identified and a fix is being implemented. We'll continue to provide updates as soon as more details are available.

After a period of monitoring, we have not seen any recurrence of this issue. We consider this incident resolved and do not anticipate any further impact on our customers. We apologize for the impact this caused to affected customers.

We have been in contact with Payeezy and they confirmed there was an outage on their side. It appears this has been corrected as we are seeing successful transactions with this payments gateway now. We will monitor for some time to make sure all has returned to normal.

Our team is actively investigating an issue that led to Payeezy payments gateway outage. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We will provide an update as soon as more details are available. Thank you for your patience.

We have identified an issue with the Payeezy payments gateway which is impacting customers transacting through this integration. We are currently investigating and will provide an update within the next hour.

We have closely monitored the situation and received official confirmation from Elavon, as well from our customers, that the issue has been resolved. As a result, we consider the incident resolved and do not anticipate any further impact on our customers or their capability to transact using this gateway. We apologize for any inconvenience or disruption.

We have been notified by Elavon that a fix has been implemented. We also see evidence of an increased success rate in our internal systems and will continue monitoring. Please reach out to our Support team if you need any additional help.

We are continuing to work on a fix for this issue.

We are continuing to reach out to Elavon in an effort to have this issue resolved as soon as possible. We will provide another update an hour from now. Thank you for your patience.

We continue to actively reach out to Elavon in an effort to achieve a resolution on this issue, however the current expectations are that we will see further traction tomorrow, Monday. We will provide another update by 9 AM ET, unless we have any information from Elavon sooner.

We continue to actively reach out to Elavon in an effort to achieve resolution as soon as possible. We will provide another update by 4 pm ET, unless we have any information sooner.

We will continue to reach out to Elavon to pursue a fix for this issue.

We will continue to reach out to Elavon to pursue a fix for this issue.

We have been in touch with Elavon throughout the day. The issue is known and we have requested Elavon dispatch additional resources to resolve their network issue. We will provide another update no later than 10 EDT tomorrow, or sooner if the issue is able to be resolved before then.

We continue to actively follow up on status of this issue with Elavon and while they are still investigating, we do not have an estimated resolution time yet. We will provide another update as soon as we receive additional information from their organization.

Our team has been assured that Elavon continues to actively investigate this issue. We will follow up with them and provide another update within the next couple of hours.

We have actively engaged with Elavon and are currently jointly investigating this issue. We will provide another update once we have more information.

We are continuing to work on a fix for this issue.

Since 2023-08-11 at 20:06 ET we started to detect intermittent errors when communicating with the Elavon gateway. This issue appears to be similar to a previous one we experienced on August 1st. Approximately a third of the transactions traffic is being blocked by the payments gateway. We have reached out to Elavon for immediate assistance in resolving this issue and will provide an update as soon as we have additional information

We have closely monitored the situation since Elavon notified us earlier that the issue was resolved, and we have confirmed that the errors that were previously experienced are no longer being experienced. We consider this incident resolved and do not anticipate any further impact on our customers. We apologize for any inconvenience or disruption. We will continue working with Elavon on root cause analysis and in jointly ensuring that we prevent a similar issue in the future.

We are continuing to monitor for any further issues.

We have been notified by Elavon that a fix has been implemented. We also see evidence of an increased success rate in our internal systems and will continue monitoring.

We are continuing to investigate this issue. We are in contact with Elavon and are confident that our investigation is progressing towards a resolution. Thank you for your patience and we sincerely apologize for any inconvenience.

We are continuing to investigate this issue. We are currently working with Elavon to work toward a resolution. Thank you for your patience.

Spreedly identified an issue outside of our platform that could impact our services with Elavon Gateway. This may affect customers transacting against this gateway. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We will provide an update as soon as more details are available. Thank you for your patience.

## Summary A planned operating system upgrade on July 12th resulted in slightly higher ongoing CPU usage than the prior OS. This compounded so that by July 19th, we began to exhaust CPU credits, which reduced our ability to successfully process a number of transactions. ## What Happened As prologue, on Wednesday July 12th, we exercised a planned-for operating system upgrade on the servers running the cryptography service. This involved our usual and practiced behavior of gracefully moving traffic from one cluster of servers \(i.e.: “blue” to “green”\), patching the offline cluster and then performing the same in reverse. Testing illustrated that the upgrade was a success and the system continued processing ~100m\+ internal service API requests daily without issue or incident. On July 18th at 18:26 UTC, internal monitoring alerted on an increased number of failures being generated from our cryptography service. Teams were immediately assembled to work the issue. Logs indicated a resource contention `(limiting connections by zone "default"`\) as individual service nodes exceeded the configured limit of simultaneous connections. We began the effort to alter the configuration of these limits and the task of quieting one cluster, effecting the repair, bringing the cluster online, and then performing the steps for the other cluster. Unfortunately, once reconfigured, a different resource contention was found, this time at the operating system level. We began the effort to alter that configuration and percolate the change through both “blue” and “green” clusters but the errors merely moved to another type of resource constraint. We reverted potentially relevant changes from earlier in the day, without positive effect on the system. Finally, we reverted the operating system upgrade from the July 12th activity which ultimately resolved the issue. By July 19th at 00:29 UTC, the error rate on all cryptography service nodes returned to zero and we left the incident in a monitor state for approximately 12 hours before declaring the incident resolved. A subsequent root cause analysis has determined that we overran the “CPU Credits” allocated to our AWS EC2 instance type on which the cryptography service was running. We believe that the OS upgrade incrementally consumed enough CPU capacity to eventually exhaust CPU credits, as credits are utilized at relatively small CPU usage, in excess of 20% ## Next Steps We are re-provisioning this service to use an AWS EC2 instance type that has no CPU credit limits which should prevent the problem from happening again. ## Conclusion We want to apologize to all customers who were impacted by failed transactions due to this incident. We understand how much you rely on our systems being fully operational in order to successfully operate your business. We also appreciate your patience while this issue was being investigated and resolved, thank you again for the trust you place on us every day.

After closely monitoring our Core Transactional API and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. We are completing our investigation with regard to the causes of the incident and any residual impact. A post-incident review will be published. We apologize for any inconvenience or disruption.

Our Engineering team has identified and implemented some changes to address the issue. As we actively monitor the fix, we are seeing a reduction in error rates indicating the fix is stabilizing the errors and addressing the issue. We’ll continue to monitor the fix closely and provide updates.

Our team is actively investigating an issue with one of the libraries that feeds our Core component resulting in some transactions receiving errors. Some customers may have experienced 500 errors. We are still investigating a potential solution. Thank you for your patience.

Since 2023-07-18 3:01 ET, there has been an issue detected causing intermittent errors on the Spreedly API. Our team is actively working towards a solution to rectify the problem, and we will keep you updated on any progress made in resolving the matter. We value your patience as we work to resolve this issue.

After a period of monitoring, we have confirmed that the issue has been resolved.

The intermittent errors with the Spreedly iFrame have subsided. We are continuing to monitor to ensure that the issue is fully resolved

We have identified an issue with an upstream service provider causing intermittent errors on the Spreedly iFrame. We are communicating with our service provider as they work to resolve the issue.

We are currently investigating an issue with the Spreedly iFrame. Users may experience that iFrame will intermittently not load when requested.

**April 27th, 2023 — Temporary resource contention in our environment resulted in 500 response errors** A code merge inadvertently prompted creation of a large queue which resulted in transitory transaction errors. **What Happened** A code merge inadvertently impacted batch processing services which created a large queue and which put pressure on a critical path database service. This resulted in some transitory errors for transactions, the majority of which were re-tried successfully.After the issue was resolved by restarting the processing service and the queues returned to normal, we refined our queue monitoring process to include additional monitoring conditions. **Next Steps** Work has been started to optimize our database performance in order to prevent similar issues in the future. **Conclusion** We apologize for this disruption to service, and will continue to drive internal improvements to avoid similar impact in the future.

After closely monitoring our resource utilization and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected in association with this incident. We are completing our investigation with regards to the causes of the incident and any residual impact. A post incident review will be published. We apologize for any inconvenience and disruption to service this caused for impacted customers.

While the previously experienced transaction errors are resolved, we are still monitoring our resource utilization closely until our system is confirmed stable. Customers may still experience some delays to services such as Dashboard reporting and callbacks. We will provide an update within 24 hours.

We identified this issue was caused by a temporary resource contention in our environment and we have addressed it. We are monitoring to ensure this has fully resolved the issue.

We are actively investigating an issue which resulted in 500 errors responses being returned from Core. We have identified some customers have been impacted. We will update with additional information as this investigation progresses.

Spreedly has detected an issue that may impact customers, including potentially resulting in failed transactions. While we are providing an early notification in an effort to alert as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available.

We have received confirmation from Mastercard that a service disruption, which was identified for their Smart Interface and caused customers to experience intermittent transaction failures during the impact period, has now been restored. We appreciate your patience while we tracked this issue with our partners.

Our team is still tracking an issue that started June 12th, 2023, at 07:03 am EST (12:03 UTC). Customers may continue to experience transaction failures within our 3DS2 Global Solution. Our Global 3DS2 provider and Mastercard have both confirmed this is an ongoing issue impacting Mastercard's card types. We understand that the Mastercard team is working to mitigate the issue and restore service to all customers as soon as possible. We will provide an update as soon as more details are available. Thank you for your patience at this time.

We have identified an issue which is impacting some Mastercard 3DS2 transactions for a subset of our customers using our 3DS2 Global solution. These transactions are failing with the following error "Remote Error: Intermediate layer error - error in intermediate layer (message received invalid. could not read the message. authentication response)". We have logged a ticket with our third party 3DS2 provider and they are actively investigating. We will provide updates as these become available to us.

After deploying a fix, our system is now stabilized and functioning. This incident is being considered resolved. We apologize for any inconvenience and disruption to service this caused to our customers transacting via Stripe.

Spreedly is actively addressing an issue with our Stripe integration which is impacting customers transacting through this end point. We are expected to be in this degraded performance state for about 20 minutes, while we work to correct this issue.

### April 6th, 2023 — Failure during vendor maintenance resulted in core errors A vendor service in the critical path for some Core transactions had an unrelated failure during maintenance which resulted in errors to customers. ## What Happened One of our external vendors regularly performs routine maintenance on databases in order to maintain security and reliability. While performing such a maintenance, on April 6th, 21:37 UTC the vendor automatically migrated our application from one database copy to another \(a standard operation\). This failover coincided with an incident in their application engine, which prevented our application from restarting with the new database. As a result, a critical secondary service was unavailable and resulted in customer errors from 21:41 to 22:05 for some classes of transactions \(such as tokenizing new cards\). Once the vendor resolved their application engine issue, our application successfully started and normal operation resumed. ## Next Steps We are in the process of moving this critical secondary service databases to a new hosting provider, which will provide more control over database scale and maintenance windows. ## Conclusion We apologize for this disruption to service, and will continue to drive internal resiliency and availability initiatives to reduce the impact of 3rd party outages.

We have confirmed the third-party component maintenance is complete and our systems are functioning normally. This incident is now resolved.

We have identified the source of this degradation to be related to a third-party component maintenance. The impact has been alleviated and core transactions are now functioning normally. We will continue to monitor for a period of time to ensure no further impact.

We are aware that some customers are experiencing a degraded performance with our Core Transactional API and we are actively investigating. We will update as soon as we have more details on this issue.

We have confirmed that this issue has been resolved, and customers are no longer experiencing 500 core errors. A post-incident review will be published. We apologize for the impact this issue caused to affected customers.

We have confirmed that the implemented fix has resolved the 500 errors for customers. We will continue to monitor to ensure resolution.

We have identified the issue that resulted in customers receiving 500 core errors and have implemented a fix to resolve this issue.

Spreedly’s actively aware that some customers may be experiencing Core 500 errors. We are currently investigating.

After a period of monitoring, we have observed that 3DS2 Global transactions are processing successfully. The incident is being considered resolved. We apologize for any inconvenience and disruption to service.

Our 3DS provider has identified that the outage has been resolved. We are continuing to monitor 3DS2 Global transactions to ensure that SCA Authentications are processing successfully.

Since 2023-03-01 08:33:13 UTC this morning, transactions using Spreedly's 3DS2 Global solution have experienced SCA authentication failures due to our 3DS service provider responding with an HTTP 502 generic error code. This error impacts only transactions using the 3DS2 Global solution. We are communicating with our 3DS provider about the outage, and we will provide updates as they work toward a resolution.

On January 25th, an application code deployment resulted in health check failures which impacted our computing ability to service other API requests properly. ## What Happened On 2023-01-25 22:41 UTC , an application code deployment removed application health endpoints that were used for critical infrastructure monitoring purposes. As a result, otherwise healthy instances were falsely marked as requiring termination, and new replacement instances were created. Beginning at 2023-01-25 23:09 UTC there were at times too few instances available to serve the volume of traffic, which impacted our ability to properly service API requests. Spreedly reverted the problematic code and instantiated new instances, which restored service levels for all customers. ## Next Steps Spreedly is committed to holistically reviewing and reimagining our change release processes and culture, with a focus on better documentation and cross-team training on cross-API interface requirements, deployment monitoring, and increased automatic paging for actionable alerts. ## Conclusion We deeply apologize to our customers for this interruption to service and the impacts on their business which they have entrusted to Spreedly.

We have confirmed that this issue has been resolved and customers are no longer experiencing 502 Bad Gateway errors. A post incident review will be published. We apologize for the impact this issue caused to affected customers.

We have implemented a fix and services are restored. We are currently monitoring and confirming with impacted customers that they are no longer experiencing these errors.

We have identified an issue resulting in customers receiving a 502 Bad Gateway Error. Our team is working to resolve this issue currently and we will update shortly.

After a period of monitoring, Spreedly is confident that the cloud provider hardware failure was fully corrected automatically. All systems are confirmed to be fully operational.

Spreedly has identified the source of the errors as a hardware failure in our cloud provider. The system autocorrected and is currently stable. Engineers are continuing to monitor this situation.

Spreedly is aware some customers may be receiving errors on the Core Transactional API, and is actively investigating.

From ~6:02-6:08pm EST (~6 mins) on 1/19/23 Spreedly experienced an outage of our Content Delivery Network (CDN) and Web Application Firewall (WAF) provider (Fastly) as referenced here: https://www.fastlystatus.com/, which impacted approximately ~2/3 of requests to core.spreedly.com during this time period. Customer transactions and those logged into Spreedly ID may have had some of their requests timeout and unable to login into Spreedly ID.