Is Clearhaus Down Right Now? Check if there is a current outage ongoing.

Starting April 11th 11:00 CEST, approximately 8% of subsequent-in-series Mastercard transactions were rejected with status code 40416 by specific issuers. For the affected issuers, a significant percentage of subsequent-in-series are rejected. No Visa transactions nor non-series Mastercard transactions were affected by this. On April 14th, the problem was noticed, and investigations have been ongoing together with Mastercard, an affected issuer, and our upstream provider. On April 17th, the problem was confirmed to be an issue with our upstream provider. On April 18th 22:00 CEST, our upstream provider released a fix. On April 19th, we confirmed that the fix was working for PAN-initiated transactions, but the problem still persisted on token-initiated subsequent-in-series Mastercard authorizations. We continued to work with our upstream provider to get this fully fixed. On April 24th 12:00 CEST, we managed to implement a workaround to mitigate the problems for token-initiated subsequent-in-series Mastercard authorizations. This fully resolved the impact of this incident. We continue to work with our upstream provider to coordinate the removal of the workaround when a fix is in place. Partners and merchants are recommended to retry both PAN- and token-initiated subsequent-in-series Mastercard authorizations that failed with 40416.

On the night of 2025-03-02 we saw timeouts and connectivity issues towards our upstream provider OmniPay. Mostly we’ve been able to route traffic to a healthy data-center, but between 2025-03-02T05:27Z and 2025-03-02T05:35Z, all data-centers were not responding. The incident was resolved within 8 minutes, and we have not seen any further connectivity issues since then. We are actively working with our upstream provider to determine how similar issues can be prevented in the future when they do network updates.

Between 2024-11-20T02:27:09Z and 2024-11-20T02:40:09Z we experienced connectivity issues towards our upstream provider, reaching up to 26% error rate at the peak of the incident. Only a few transactions were affected but we are posting this update for transparency. The incident was resolved within 14 minutes, and we have not seen any further connectivity issues since then.

After taking remedial action, performance of the degraded components has returned to normal during the course of the week. Additionally, we have taken measures to improve visibility into the affected systems in order to strengthen our infrastructure and prevent similar events in the future. We are in contact with the service provider to continue to investigate the root cause of the issue and explore additional measures to increase the robustness of our systems.

Due to a service provider we are experiencing degraded performance in the Merchant API. This does not affect our ability to process transactions.

At 2024-10-03T11:26Z a warning triggered an investigation of an increase in the error rate for authorizations. The investigation showed that we were receiving a subtle amount of declines from issuers reasoned with “format error” starting 2024-10-02T05:45Z. These rejects are normal and have numerous times shown issuers to be misbehaving – which is why we have a non-zero threshold that must be met before we become aware. Together with the “format error” must be an identification of the malformatted field. This led us to identify an issue with an update we applied during 2024-10-02 resulting in AUD authorizations to be consistently declined in the time span 2024-10-02T19:55Z - 2024-10-03T17:05Z. This, however, only explained a minority of the declines. The remaining declines indicated a field that we have never populated, however, since our investigation indicated it to be an issue across multiple issuing banks, we quickly implemented a change to populate the field. Both a fix for AUD and for populating the extra field were deployed, and they turned out to fix both issues. The field population issue caused sporadic declines in the time span 2024-10-02T05:45Z - 2024-10-03T19:08Z. In summary, the two issues above caused an average 0.15 percentage point drop in approval rate in the time span 2024-10-02T05:45Z - 2024-10-03T19:08Z. We will work hard to identify how the AUD error could happen and will ensure to introduce checks to avoid a similar error to happen in the future. We will also try to work with issuers to identify the cause of the rejects based on the previously unpopulated field.

## Introduction As a follow-up on the incident on 2024-09-12, this is our findings from analyzing the sequence of events. Again, we sincerely apologize for the disruption, downtime and inconvenience. ## The issue During the timespan 2024-09-12T03:30Z - 2024-09-12T11:45Z, we experienced connection failures and timeouts to an upstream provider. This affected live transactions: authorizations and voids for both Visa and Mastercard as well as credits for Mastercard. After confirming that the issue was entirely external, and that we were thus unable to mitigate it ourselves, we reached out to our provider, opened an issue with them and escalated immediately. Our automated failover moved traffic to the provider’s secondary data center when the first transactions failed towards their primary data center. While this is normally sufficient to mitigate small hiccups, this showed that the secondary data center was also affected by the incident. No transactions towards the second data center were approved and we therefore decided to only target the primary data center as we saw upwards of 10-15 % of the transactions going through there. We periodically sent small bursts of transactions to the secondary data center to monitor availability. There was a small window of 10 minutes where transactions were processed through the secondary data center before it became dysfunctional again. We contacted our provider to inform them that the secondary data center had been in a working state. After approximately 8 hours, our provider was able to mitigate the issue in both data centers, which resolved the incident, and approval rates went back to normal. For a short period of time during the incident our services were exhausted which exacerbated the condition. This resulted in the transaction gateway responding HTTP 504 Gateway Timeout which affected all transaction types. We are sincerely sorry for the inconvenience and we are working to prevent this from happening in the future. ## Remediation While we do not have any direct involvement in the incident at our provider we have learned a thing or two to take with us: 1. We will investigate the possibility to void authorizations with status code 50000 such that clients can void these transactions to ensure the state at upstream is the same representation as at the client. At the same time we will highlight that the transaction representation in the Merchant API will help in this regard to understand the state of a transaction. For critical transactions \(e.g. credits or “large enough” authorizations\) we recommend to use the [Merchant API](https://developer.clearhaus.com) to check the state of a transaction when you did not receive a response \(no matter if the failure happened on our end, somewhere in the middle, or on your end\). 2. To avoid HTTP 504 responses, we will go over our resource allocation and adjust to have a better tolerance. In addition we will improve alerting on our application load balancer to highlight when our internal systems are responding slowly and thus indicating a resource exhaustion in our system. 3. Investigate if we can improve upon the automated failover mechanism to balance traffic so we do not need to manually send small bursts to test availability. This could potentially have made us earlier aware when the secondary came up both for midways in the incident and in the very end of the incident. Already implemented improvements: 1. Extend internal dashboards to give improved visibility in the continuous success rates on the individual upstream data centers.

Our upstream partner has resolved their issue and our monitoring shows that the connection has been stable and without problems since our last update. We will investigate mitigating actions and the root cause of this incident and will publish a postmortem once we have a complete overview from the upstream partner. To summarize, we saw on average around 90 % of authorizations impacted during the time period 2024-09-12T03:30 - 2024-09-12T11:45 UTC. All impacted authorizations were responded with status code 50000 by our gateway API for transaction processing. We sincerely regret the inconvenience this serious incident has caused!

We have seen a significant increase in approval rates and the connectivity is now back to normal. Due to the severity of this issue and the previous sudden change in approval rate we are still actively monitoring this situation.

We are again seeing an improvement in approval rate starting 11:46 UTC and are continuing to monitor the situation as we previously have seen such improvement temporarily.

We are awaiting further communication from our provider. We are actively monitoring the situation. There is no ETA for when this issue is resolved. We will update as soon as we know more.

We saw an improvement in approval for a short period of time from around 08:47 UTC to 08:59 UTC. Unfortunately, this has worsened again, and we are back on approval rates below 15 %.

Unfortunately, we still see very high failure rates on authorizations. Our upstream provider is still trying to mitigate this critical incident. We are seeing examples where amounts are being reserved on cardholders' accounts despite the authorization failing. Therefore we recommend that subsequent-in-series recurring and unscheduled (e.g. subscriptions) are not attempted until after this incident is resolved. Furthermore, we are exploring options to potentially release these reservations.

We have identified that the issue is with our upstream provider. We have attempted mitigating actions but unfortunately with very limited impact. We are still seeing very low approval rates that fluctuates approximately from 0 % to 15 %. Our upstream provider is trying to resolve the connectivity issue.

Our upstream provider has confirmed and is aware of a connectivity issue. Currently almost all authorizations, voids and Mastercard credits are impacted.

Since 2024-09-12T03.39Z we have observed connectivity issues towards an upstream provider. We have reached out and are investigating with the provider.

The cause of this incident was that an issuer/Visa disabled the impacted BINs although they were actively being used. Luckily, issuer/Visa succeeded in reenabling the impacted BINs over the weekend. This has propagated through systems, and made both new transactions and previously impacted transactions able to clear. After careful analysis we can confirm that all previously impacted transactions are now cleared successfully and they are also included in settlements.

Visa has identified that the BINs involved in clearing failures are: 414865, 432921, 435786, 443254, 449269, 451143, 451349, 455853, 455854, 457101, 460312, 457154, 461061, 462003, 464254, 479870, 481686, 494109, 457153, and 457154.

Some captures and refunds for card numbers starting with 457153 and tokens that are generated from such card numbers failed clearing this night even though Clearhaus accepted the transactions. There are two consequences for such impacted captures and refunds: 1. They do not yet appear on cardholders’ bank accounts; this is especially unfortunate for refunds. 2. They are not yet being settled on merchants’ accounts; this is especially unfortunate for captures. No action from you is required or recommended; this incident is for your information. Please notice that authorizations on both impacted cards and tokens appear to be functioning normally. Though, 3-D Secure authentications are also impacted. Transactions on any other card are not impacted! Visa has acknowledged the issue and is working with the bank that issues these cards and tokens to ensure that clearings can again go through. We await further updates from Visa and also await the issuer to resolve the problem and do expect these clearings to eventually go through. We will keep you updated whenever there is news to share. We intend to update you no later than 2024-09-03T12:00Z.

This incident has been resolved.

Our monitoring shows all Visa transactions are functional, and we therefore close this incident. The time window of failures lasted from 2024-08-02T05:59Z till 2024-08-02T08:13Z.

Visa has announced that the service disruption is fully mitigated and we can positively confirm that the impacted Visa transactions are now being processed again. We will continue to monitor the error rates.

While we are awaiting an update from Visa we have to the best of our ability confirmed that transactions are not being processed upstream; thus, retrying a transaction is not expected to result in multiple processing if retried when this issue is resolved. However, we will advise you to be careful especially when retrying credit transactions.

At 2024-08-02T05:59Z we started noticing consistent failures in communication with Visa for both credit and debit transactions; neither Mastercard nor other Visa transaction types (e.g. authorizations and captures) are impacted. Visa has announced a service disruption, and we will keep you updated as we get information to share from Visa.

The hotfix has been replaced with a proper solution that has been monitored for the last couple of hours. We are hereby closing this incident.

All impacted authorizations have now been voided. These voids are visible via my.clearhaus.com. We expect to update and close this incident when we have tested and applied a proper solution in the coming days.

A significant amount of the authorizations that we responded with 50000 was approved upstream. We are currently attempting to void these approved authorizations and expect this to be completed within the next few hours. We will update this incident when the voids are completed.

The issue started 2024-04-25T10:35:00Z and ended 2024-04-25T11:48:00Z when we applied a hotfix.

We are currently seeing an increase in declined Visa Authorizations and are currently investigating.

The missing references have been brought back. The affected captures, refunds and voids are in the time span 2024-01-08T12:33:47Z - 2024-01-13T10:16:38Z.

Since 2024-01-08 references have been missing on some captures, refunds and voids in the Merchant API and on my.clearhaus.com. We apologize for the inconvenience and are doing our best to quickly bring them back. Transaction processing is not affected.

We experienced an increased amount of declines on authorizations, voids and Mastercard credits with status code 40400 backend problem in the time span 2023-11-03T00:59:31Z to 2023-11-03T01:09:38Z due to problems confirmed by an upstream provider. Neither captures, refunds, debits, debit refunds nor Visa credits were affected.

We have worked with Visa to resolve the issue. Some authorization responses in the affected timespan did not include Visa transaction IDs (TIDs). Should any client require the missing TIDs, do not hesitate to reach out. If missing TIDs do not pose a problem, no action is required. We apologize for any inconvenience.

Our upstream service provider has confirmed the issue is a result of a system upgrade on their end, and they are working on a resolution. Some subsequent-in-series authorizations may still fail for the time being, and we ask clients to postpone these authorizations until the issue has been resolved. Fortunately, very few authorizations are affected by this issue.

We have identified the issue and have managed to contain it to some degree. It is now the case that some attempts at creating a subsequent-in-series Visa authorization referencing a previous authorization created in the time span 2023-10-13T09:49:16Z - 2023-10-13T15:08:48Z might fail. We are working towards complete resolution.

Some attempts at creating a subsequent-in-series Visa authorization referencing a previous authorization created after 2023-10-13T09:49:16Z are currently failing.

We noticed at 2023-10-03T04:32:30Z that our DNS provider had issues. They identified and were able to resolve the issue by 2023-10-03T05:25Z. Since the TTL is one hour on the DNS records, the immediate impact of this issue is expected to be limited. However, during the period of the DNS issue, clients may have experienced sporadic connectivity problems or delays in accessing our services depending on their reliance on DNS.

All affected transactions have been settled and the issue is resolved.

We are investigating an issue on Mastercard transactions for the period between 2023-08-09T19:00Z and 2023-08-10T19:00Z. Only captures, refunds and credits are impacted whereas neither authorizations nor voids are impacted. Transactions appear successful in Clearhaus’ systems — also with respect to bookkeeping and payouts to merchants. There is no impact from a merchant’s perspective; only via clearing transactions not being effectuated on their cardholders’ accounts. Our investigation so far confirms that: - A majority of captures from 2023-08-10T09:00Z to 2023-08-10T14:00Z are impacted. - A majority of refunds from 2023-08-09T19:00Z to 2023-08-10T14:00Z are impacted. - All captures and refunds from 2023-08-10T14:00Z to 2023-08-10T19:00Z are impacted. - All credits from 2023-08-09T19:00Z to 2023-08-10T19:00Z are impacted. It is greatly important to note that this update is solely for informational purposes, as no action is required from any other party than Clearhaus and our upstream provider. We have identified the impacted transactions and are working with our upstream provider and Mastercard on a resolution. Since transactions appear successful in Clearhaus systems, we expect to be able to resolve the issue without your involvement. If you need to do anything, we will reach out directly.

No further timeouts have been observed. We have been in contact with our service provider and confirmed all systems to be operational. We apologize for the inconvenience.

From 2023-07-03T18:30:13Z to 18:48:41Z today, we saw an intermittent increase in timeouts towards our upstream service provider affecting authorizations and voids. Mastercard credits were also impacted. Other transactions like captures and refunds were not impacted. The issue appears to have subsided, but we will continue to monitor the situation and post another announcement should the issue reemerge.

In collaboration with Mastercard we are investigating and exploring the options for cleaning up the remaining inconsistencies. Fortunately only very few merchants are impacted, so further communication will happen directly with the affected merchants; they should expect an update no later than 2023-06-19T16:00:00Z. Therefore, we mark this incident as resolved here. We have run extra checks to confirm that the fix of inconsistencies for the impacted Mastercard credits from 2023-06-07T08:00:00Z till 2023-06-09T15:00:00Z is successful. We are deeply sorry about the inconvenience this incident has caused to partners, merchants and cardholders. We strive to do our best so we will surely use the learnings we have had over the last week to improve our systems to help avoid similar consequences in the future.

We can positively confirm that the inconsistency has been fixed for the impacted Mastercard credits from 2023-06-07T08:00:00Z till 2023-06-09T15:00:00Z whereas the inconsistency could not be resolved for the transactions in the beginning of the interval (from 2023-06-06T19:00:00Z till 2023-06-07T08:00:00Z). We are further investigating the remaining inconsistencies and will again be working with Mastercard to resolve this. Please expect an update on the resolution of the remaining inconsistencies no later than 2023-06-16T10:00:00Z. Still, Mastercard credits for merchant accounts with MCC 7800, 7801, 7802, 7994, 7995 or 9406 created since processing was resumed Monday are working without issues.

We have been working with Mastercard (and have also in parallel run tests) to resolve the inconsistency for the impacted Mastercard credit transactions from 2023-06-06T19:00:00Z till 2023-06-09T15:00:00Z. Preparations for a final solution are happening today, and the results should be available to us tomorrow after a nightly run at Mastercard. Thus, please expect an update on the resolution of the inconsistency no later than 2023-06-15T10:00:00Z. We have been following the status of Mastercard credits for merchant accounts with MCC 7800, 7801, 7802, 7994, 7995 or 9406 that have been processed since processing of these transactions was resumed Monday, and are happy to report that we see no errors or inconsistencies.

We can confirm that our solution was successful and are thus resuming processing of the impacted Mastercard credit transactions. Please notice that we are still researching how to resolve the inconsistency of approved credits processed from 2023-06-06T19:00:00Z till 2023-06-09T15:00:00Z. Please expect an update on this no later than 2023-06-14T16:00:00Z.

The solution is developed in testing, reviewed and tested, and will undergo testing for production readiness. If everything goes as planned, we will run production tests through Mastercard systems Sunday 2023-06-11, and will need to wait for the result till Monday 2023-06-12. No later than 2023-06-12T16:00:00Z will we provide an update regarding our confidence in the solution. If we are confident that our solution is correct, we will resume processing together with the update, otherwise we will most likely need to postpone processing of the impacted Mastercard transactions for another 24 hours. Thus, expect the next update no later than 2023-06-12T16:00:00Z.

We have been looking into this issue throughout the day and will continue to do so over the weekend. A solution for the impacted transactions is under development and has been partially submitted for internal review. The solution is currently being tested. Progress will be slightly slower over the weekend, but we are working on resuming processing of the impacted Mastercard credit transactions as soon as possible. You can expect the next update before 2023-06-11T00:00:00Z.

We can confirm that this incident is relevant only for Mastercard credits on merchant accounts with MCC 7800, 7801, 7802, 7994, 7995 or 9406. Impacted transactions are currently being actively declined from our end with status code 50000 (Clearhaus error). The focus is on finding a solution to resume processing, and we are actively working to achieve this. The time frame for a solution is unfortunately not yet known. You can expect an update before midnight UTC. After processing has resumed we will detail and communicate a plan for resolving the inconsistency of approved credits processed from 2023-06-06T19:00:00Z till 2023-06-09T15:00:00Z. We will explore all options to most effectively (and conveniently to you) handle such inconsistencies. At this stage, we are not yet able to fully investigate the state of these transactions in Mastercard systems. Further updates also in this regard will be posted here as soon as we have them. The impact of the inconsistency is that an approved credit transaction for an account with above-listed MCCs appears as successful to the partner/merchant but may appear as failed to the cardholder. Please be aware of this when handling cases from cardholders.

We are noticing problems processing Mastercard credits for specific merchant accounts, namely those with MCC 7800, 7801, 7802, 7994, 7995 or 9406. Early investigation shows that such transactions since 2023-06-06T19:00:00Z have failed even though our systems returned with status code 20000 (approved). Further investigation will be ongoing in the following hours. All other transaction processing is not impacted.

Approval rates are back to normal as of 2023-06-12T14:21:00Z.

Today after 11:00Z we started seeing low approval rates for all live Mastercard transactions (authorizations and credits). Mastercard are having issues with multiple systems, so unfortunately there is nothing we can do other than to monitor the state.

We haven't observed any failures towards our upstream service provider since 2023-04-30T16:58:27Z.

We are observing intermittent problems with an upstream service provider. It started around 2023-04-30T16:13:36Z. Our clients receive status code 50000 and message “failed processing transaction with upstream, state unknown”. Upon manual inspection, we are able to deduce that none of the failed authorizations were successful. Neither debits, captures, refunds, nor Visa credits were impacted.

No further errors observed.

From 2023-04-08T17:03:47Z to 2023-04-08T17:09:21Z we saw authorizations being consistently failing. Our clients received status code 50000 and message “failed processing transaction with upstream, state unknown”. Upon manual inspection, we are able to deduce that none of the failed authorizations were successful. Voids and Mastercard credits were equally impacted. Neither debits, captures, refund, nor Visa credits were impacted.

A fix has been put in place by our upstream partner. The first failure happened at 2022-10-14T10:00:09 UTC and the last failure happened at 2022-10-14T18:06:51 UTC.

Creating a workaround for the issue is unfortunately not possible as the received values are sufficiently malformed. The issue has been escalated at Visa. We are monitoring the issue.

We see an increased error rate for a small number of Visa BINs and have identified the issue. It is affecting 0.14% of the Visa authorizations sent upstream. The issue started at 2022-10-14T10:00:00Z. The incident has been communicated upstream to relevant parties. We are currently implementing a solution to work around the issue.

Payouts have been carried out this morning and merchants should receive the funds during the course of the day.

All checks and imports are now done. Delayed data is now available on the Merchant API and in my.clearhaus.com. Our ultimate cut-off for wiring funds to our merchants today was 12:30 UTC. This unfortunately means that funds will be delayed by a banking day.

We have been able to receive the missing settlement, dispute, and transaction data from our upstream service provider. We are running checks and importing the data into our systems. Expect the next update at 13:00 UTC.

Updated settlement, dispute, some transaction information, and payouts continue to be delayed for the time being. Our upstream provider has not been able to fix the issue. This does not affect our ability to process transactions. Expect the next update at 12:00 UTC

Updated settlement, dispute, some transaction information, and payouts continue to be delayed for the time being. Our upstream provider has not been able to fix the issue. This does not affect our ability to process transactions. Expect the next update at 11:00 UTC

Updated settlement, dispute, some transaction information, and payouts continue to be delayed for the time being. Our upstream provider has not been able to fix the issue. This does not affect our ability to process transactions. Expect the next update at 10:00 UTC

Updated settlement, dispute, some transaction information, and payouts continue to be delayed for the time being. Our upstream provider has not been able to fix the issue. This does not affect our ability to process transactions. Expect the next update at 9:00 UTC

One of our upstream service providers is experiencing issues with their systems which is keeping us from providing updated settlement, dispute, and some transaction information. This unfortunately also delays upcoming payouts from being carried out. This does not affect our ability to process transactions. Expect the next update at 8:00 UTC

Our upstream partner confirmed the issue. No further disruptions are expected.

From 2022-06-24T22:32:00Z to 2022-06-24T22:43:00Z we saw a dramatic increase in authorizations being declined with 40400 “backend problem” from upstream. It is back to normal, but we are inquiring to understand what happened.

A planned database migration had an unforeseen impact and resulted in transaction processing of authorizations and credits having a failure rate of upwards 60 % from 2022-06-22T08:23:24Z to 2022-06-22T08:24:41Z. We apologize for the inconvenience.

We have not seen more occurrences of the issue. In addition, It has been confirmed that the issue was caused by upstream network instability. We expect the situation to have stabilized now.

We have seen the issue take place from 2022-06-20T12:58:15Z to 2022-06-20T13:04:40Z and from 2022-06-20T14:29:52Z to 2022-06-20T14:45:49Z.

In continuation of the previously announced incident, we are seeing it reappear. Further investigation is ongoing. Again, only authorization and void traffic is impacted.

Our investigation has no indication of failures that neither we nor our upstream partners are in control of, so this incident can only be attributed to intermittent network connectivity issues. We notice that transactions were successfully retried.

In the time span 2022-06-20T06:43:04Z - 2022-06-20T06:49:55Z we experienced connection resets and timeouts upstream towards Visa and Mastercard for approximately one third of authorizations and voids. No other transaction type was affected. Our investigations show no structure in the errors so far. We are conducting further investigations in an attempt to understand this.

Mastercard credit transaction processing is again fully functional. Transactions in the time span 2022-06-07T10:13:06Z - 2022-06-07T11:26:05Z were affected. Please consider resubmitting previously declined transactions.

A solution has been deployed.

Mastercard has today updated the requirements for processing credit transactions. The update we have applied in our end is incompatible with Mastercard's update for certain merchant configurations. We are working on resolving the issue. The first impacted Mastercard credit transaction was processed at 2022-06-07T10:13:06Z. No other transaction type is impacted. Please hold off processing Mastercard credit transactions if possible until further notice.

Today, 2022-05-18, between 04:03:04 and 04:44:49 UTC our upstream service provider experienced a service disruption, resulting in lost communication with Visa systems. As a result, all Visa authorizations in the mentioned interval were declined with status code 40400. No other transactions were impacted. We apologize for this inconvenience.

Upstream has acknowledged the incident and resolved it by switching processing to secondary systems. Processing has been stable since 2021-11-09 23:01 UTC. No further interruptions are anticipated.

We are again noticing Mastercard authorizations and voids being approved since 2021-11-09 23:01 UTC.

We have been experiencing an upstream processing issue for Mastercard authorizations and voids since 2021-11-09 21:54 UTC.

Sorry for the late update! Regarding reversal processing: We have confirmation that Visa reversals have been processed successfully. We are thereby closing this incident. We are deeply sorry about the inconvenience this incident has caused to partners, merchants and cardholders. We strive to do our best, and therefore we will surely improve our systems to avoid similar incidents in the future. A deeper analysis is ongoing and an action plan is to be developed. When we have this ready, you can expect this incident to be updated with a postmortem; expect this early December.

Regarding incident announcement to issuers: Visa closes the incident by announcing to issuers that reversals will be processed with processing date October 13th. Expect another update no later than 2021-10-13 12:00 UTC for when we have confirmation that Visa reversals have been processed successfully.

Regarding reversal processing: We have confirmation that Mastercard reversals have been processed successfully. Visa reversals are ready for tonight’s processing cycle since no option turned out feasible for inclusion in an extended cycle. Visa will circulate an announcement to issuers when the reversals are successfully received by Visa. We will update this incident as soon as the announcement is circulated; expectedly today. Expect the next update no later than 2021-10-13 12:00 UTC.

Regarding reversal processing: We have only good indications for Mastercard reversals so far and expect them to process cleanly. Visa reversals will unfortunately not make it for tonight’s clearing cycle. We are exploring all possibilities for having Visa reversals included as soon as possible. Expect another update no later than 2021-10-12 12:00 UTC.

Regarding incident announcement to issuers: We do not expect a Mastercard announcement to be circulated. Regarding reversal processing: Mastercard reversals have been sent for processing in tonight's clearing cycle. Visa reversals are slower to produce, and we are not sure if we will make it for tonight’s clearing cycle. Expect another update no later than 22:00 UTC.

Regarding incident announcement to issuers: Visa has circulated the announcement.

Regarding incident announcement to issuers: We are working with Visa to have the information reach relevant issuers. The announcement is expected to be circulated among issuers today. We are chasing Mastercard if such an announcement is possible and necessary.

We have confirmation that we are able to reverse the duplicate transactions. For a pair of double transactions, the first transaction will stay whereas the second will be reversed; and it will be reversed with the posting date being the same as the second transaction. This ensures that cardholders should experience no consequences as for currency exchange rate fluctuations between the day of the second transaction and the reversal. We are doing our best to have the reversals executed with schemes during Monday’s clearing cycle, i.e. the night between Monday and Tuesday. This incident will receive an update on this no later than Monday afternoon, 2021-10-11 15:00 UTC. It is important to note that since we will automatically process all reversals (both purchase transactions, refund transactions and credit transactions), please do not manually reimburse cardholders or alike. We have reached out to schemes to get the information to issuers to hold back potential chargebacks that could otherwise be created due to this incident. This incident will receive an update on this no later than Monday morning, 2021-10-11 09:00 UTC.

We are still working on reversing the transactions. If you suspect your cardholders are impacted by this and want to be certain, please reach out to us via email to support@clearhaus.com and include your Merchant ID (you can find it in the upper right for instance when seeing details for a transaction); then we will respond with a list of impacted transactions.

During transaction clearing, already cleared transactions were cleared again. All transactions (captures, refunds and credits) processed between 2021-10-06 19:00:00 UTC and 2021-10-07 18:59:59 UTC were affected for both Visa and Mastercard, with the exception of Visa credits. We are investigating the issue and working on reversing the transactions. Expect the next update no later than 2021-10-09 20:00 UTC.

A fix is now in place. We have confirmed that the fix fully resolves the issue.

We have seen a few handful of Google Pay transactions fail since 2021-09-12T22:00Z with an error indicating that the intermediate key in the signing chain was wrong when this was actually not the case. We are currently working on a fix.

This incident has been resolved.

We are continuing to monitor for any further issues.

Between approximately 10:03 and 10:10 UTC, a small number of requests to the Merchant API failed. This did not affect our ability to process transactions.

An update to gateway.clearhaus.com that was rolled out shortly after 2021-06-02T13:50:00Z and rolled back by 2021-06-03T09:38:00Z had a regression for the processing of subsequent-in-series Mastercard authorizations resulting in a slightly increased rate of declines (with status code 40416). The vast majority of such authorizations processed in the above-mentioned interval was processed without properly marking the authorization as “exempted for SCA”, and a few issuers happened to decline as a result of the missing indication. Approximately 1,300 such authorizations were declined during the period due to this regression. These merchant-initiated authorizations may be retried by the merchant or the PSP, so we advise merchants or PSPs to retry subsequent-in-series Mastercard authorizations that were declined with status code 40416 during the above-mentioned period. Please be aware that PSPs may already have systems in place to auto-retry subsequent-in-series authorizations. We apologize for the inconvenience this has caused! If you have any questions, please feel free to reach out to us.

As a result of upstream issues, almost every authorization request between 20:43:01 and 20:48:00 UTC received status code 40400 (backend problem) indicating a timeout or the client timed out before we responded.

We experienced an intermittent outage from 20:44:39 till 20:48:20 and are investigating.

Everything has been back to normal since 2020-12-17T15:08:03 UTC. The start time of the interval is adjusted from 2020-12-17T15:05:49 to 2020-12-17T15:04:49 to allow for the beginning of every potential transaction not processed in a timely manner.

During a recurring maintenance task, a database was affected by heavy load and could not keep up. This resulted in approximately 50 % of transactions processed from 2020-12-17T15:05:49 to 2020-12-17T15:08:03 UTC were not handled in a timely manner.

The service has been stable since last update. We have received confirmation from Mastercard that they had an issue on their end and that the issue has been resolved.

We are continuing to monitor for any further issues.

We notice that Mastercard is again accepting connections and 3-D Secure version 2 is again functional.

Initial investigation indicates that the connectivity error happens because Mastercard’s directory server is not accepting our client certificate for mutual TLS. We are continuing our investigation, trying out ideas for mitigation, and working with Mastercard to resolve the issue. Notice that this incident is not affecting transaction processing (acquiring), 3-D Secure version 1 (MPI), nor 3-D Secure version 2 for Visa.

We are continuing to investigate this issue.

Around 2020-11-25T14:23:35 UTC, we began seeing issues connecting to the Mastercard 3DSv2 (Identity Check) Directory Server. We are currently investigating and have reached out to Mastercard.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Transactions are delayed in Dashboard and Merchant API. This does not affect our ability to process transactions or settle merchants.

This incident has been resolved.

Between 2020-11-12 08:18 UTC and 2020-11-12 08:45 UTC we noticed a small number of requests related to user management failed due to issues with a service provider. We’re continuing to monitor our service provider and Merchant API requests.

We're currently experiencing service provider issues affecting user management. This does not affect our ability to process transactions nor settle merchants.

The Visa directory server is now also responding to requests in a timely manner.

The Mastercard directory server is now responding to requests in a timely manner. We are still noticing slightly more than normal error rates for the Visa directory server.

Since 2020-11-10 21:45 UTC we have experienced an excessive number of read timeouts towards 3-D Secure directory servers. The issue is primarily affecting Mastercard.

The situation has been normal since the previous announcement.

In the time span 2020-11-01 13:25 UTC - 2020-11-01 15:12 UTC we experienced an excessive number of timeouts towards 3-D Secure directory servers. The issue affected both Visa and Mastercard. On average the issue caused every fifth request to fail. It appears the situation has returned to normal.

This incident has been resolved.

Settlement data in the Dashboard and in the Merchant API is delayed. This does not affect our ability to process transactions nor settle merchants.

The outage was caused by the general internet problems experienced today, August 30th 2020. It appears the problem was caused by a CenturyLink misconfiguration.

At 2020-08-30 14:21:56 UTC Visa's directory servers once again became reachable and started responding successfully, and the MPI is no longer experiencing timeout issues. We will continue to monitor the situation.

The Gateway API experienced sporadic timeouts towards upstream service providers approximately 2020-08-30 12:57-13:40 UTC due to a network issue. We have worked around the issue while we are waiting for our network provider to solve it. The MPI has experienced an increase in timeouts towards Visa's directory servers starting at approximately 2020-08-30 10:02 UTC. From approximately 2020-08-30 11:44 UTC and onwards all connections towards Visa's directory servers are experiencing timeouts. We have reached out to Visa. Mastercard’s directory servers are not affected.

We are seeing increased error rates and are investigating.

We are sorry about the inconvenience caused by this incident. Investigation has shown that a busy database and unfortunate timing of events lead to the incident. Here we will explain what caused the incident, quantify the impact seen from our side, explain how we mitigated the issue, as well as explain the improvements and actions we have already taken and those that are in the pipeline. ## The root cause We have recurring jobs that clean up data; for instance, it is important to get rid of sensitive data like card numbers etc. Such a job ran Monday 2020-08-17 leading up to the incident. The data being cleaned up by such jobs are increasing over time, thus the jobs have been made to run efficiently. Very unfortunate, the amount of data and the efficiency of the execution ended up just exactly depleting our primary database’s ability to handle bursts of traffic. The job ran a bit slower than usual in the end because it was throttled, however, it also left the database without the ability to handle bursts. The ability to burst is regained over time, but the transaction pattern that followed just exactly kept the burst balance down. Due to the database’s inability to handle bursts and due to database-wise heavy transaction rules we started seeing errors. Internally we are in the process of cleaning up such heavy rules that are in some cases superfluous, however, the task is not yet complete. \(See action 1.\) ## Quantification We will only handle the three major transaction types. The analyzed period is 14:50-17:00 UTC. Authorizations: The average authorization time increased tenfold, and during the worst period the average was 20 times higher than usual. 90% of the authorization requests were responded to within 20 seconds, and 74% within 10 seconds. Less than half a percent of the authorization requests were not responded to. Captures: Captures were impacted the most. During the analyzed period captures frequently exceeded the maximum processing time of 60 seconds. The average capture processing time of approximately 200 ms was bumped to a whopping 17 seconds during the incident. 74% of the capture requests were responded to within 20 seconds, and 65% within 10 seconds. Approximately 18% of the capture requests were not responded to. Refunds: The average processing time for a refund went from our usual 200 ms to approx 4 seconds. 73% of the refund requests were responded to within 20 seconds, and 45% within 10 seconds. Approximately 4% of the refund requests were not responded to. ## Mitigation The unfortunate timing of the recurring clean-up job and transaction pattern initially led us to believe that the issue was elsewhere, namely that it was caused by the heavy transaction rules. Therefore, we started identifying accounts having heavy rules burdening the database and adjusted their rules. While this decreased both the number of timeouts and the average transaction processing time, it could not solve the problem that the database was unable to handle bursts. It simply did not scale to manually go through accounts and their rules. When this was clear to us, we started investigating how to avoid the throttling in a better way. Essentially there was no upgrade path because all upgrade paths would put further pressure on the database for an extensive amount of time. We initiated the process to test the least pessimistic looking upgrade path, but decided to try to come up with alternatives in parallel. We did come up with another strategy that proved effective, namely a coarse-grained, automatic filtering of the transaction rules and thus removal of most burst-dependent queries from the database. When we saw that we had effectively mitigated the issue and saw the ability to burst being regained, we planned and executed a rollback of the transaction rule filter in a controlled manner, thereby bringing the systems back to normal. ## Improvements We always strive to be better and learn from our and others’ mistakes. We have already learned quite a bit from this incident and have initiated multiple improvements to our organisation, incident handling procedures, and IT systems. We have a number of actions that need to be taken immediately and some that shall be taken in near future after further planning and investigation. ### Immediate actions 1. Adjust the process to clean up unnecessary heavy rules in order to speed it up. The clean-up is expected to be completed within a few weeks. We are currently adjusting our tooling to accommodate a faster process. What rules are _necessary_ is a risk decision, so unfortunately it is not easily automated for all accounts. 2. Extend our alerting to notify the on-duty team when the ability to burst is being used heavily and when it has crossed an unacceptable threshold. Adjust the clean-up process of e.g. sensitive data to ensure that it does not impact our ability to process transactions. 3. Emphasize in our incident handling procedure that early announcement shall be considered. When the incident happened, all focus was put on mitigation. Though, a swift announcement to confirm that we noticed the incident and are fully focused on mitigating would have helped you, our partners and customers. 4. Improve our insights into specifically how our databases are keeping up with increasing workloads. Our current metrics and insights showed to be insufficient or too coarse. ### Improvements in the future 1. Increase various properties of our databases; the intention is to be able to handle larger bursts. 2. Separate the transaction rule handling from the transaction processing. ## Questions If you have follow-up questions, please reach out to us by email \([support@clearhaus.com](mailto:support@clearhaus.com)\) or phone \(\+45 8282 2200\). Sincerely, Casper Thomsen, Operations

The rollback went fine and systems are now fully back to normal. We expect no further incident updates. Guidance for our partners, added 2020-08-18 Since Clearhaus systems only have our view of the transaction state, to ensure consistency, we encourage our partners to retrieve a list of transactions from either the Merchant API or the Dashboard. The period 14:50-17:00 UTC on 2020-08-17 will cover all potential inconsistencies caused by this incident. Captures and refunds are the primarily affected transaction type. If inconsistencies are found, we recommend that consistency is established and that merchants are encouraged to review the affected transactions for the period.

Firstly, we can confirm that the Merchant API and Gateway API was consistent with the exception of exactly 1 authorisation; this has been re-sent, and the PSP in question has been informed. The mitigation that we put in place and announced here 15:27 UTC was effective in mitigating the issue. This temporary mitigation is being rolled back in a careful manner over the coming hours while we are tightly monitoring the systems.

Our monitoring shows that our current mitigation has been effective. The processing time and error rates should be back to normal. If you have experienced timeouts for capture or refund processing, and will attempt to resolve quickly we have this guidance: If the transaction is for the full amount, the transaction can be retried. If the capture was already processed for a given timeouted transaction, you will on the retry receive HTTP 400 with status code 40000 and status message “no remaining amount”. Likewise for refunds. At this time we are not 100 % sure if the Merchant API is consistent with the Gateway API. We will investigate this part and ensure to correct the data behind the Merchant API in case of any inconsistencies. Please await an update within a few hours regarding this.

Since approximately 14:50 UTC we have experienced increased processing times and increased failure rates, however, we believe the issue is now resolved. We will provide a post mortem when a thorough investigation has taken place.

This incident has been resolved.

Between 2020-08-19 16:32 UTC and 2020-08-19 16:43 UTC, we noticed a small number of requests related to access management failed due to issues with our identity provider. We’re continuing to monitor our identity provider and Merchant API requests.

We're currently experiencing longer than usual response times on Merchant API. This does not affect our ability to process transactions or settle merchants.

This incident has been resolved.

Between 2020-07-17 10:00:01 UTC and 2020-07-17 10:57:05 UTC our service provider experienced numerous timeouts towards Visa Europe. Unfortunately, this entailed that approx 37% of our authorisation and void requests for Visa failed in the time span.

We are sorry for the transactions your customers could not complete during the time span our service was not fully functional. Our upstream service provider has determined that an unfortunate chain of events led to the incident. While redirecting traffic from their primary to their secondary site in preparation for maintenance, they were not aware that one or more network ports on the secondary site were out of order, causing traffic to be lost. Under normal circumstances they would have been alerted about the condition, however, alerting for the secondary site were experiencing various issues leading up to the incident. They are aware that the chain of events is completely unacceptable and have introduced additional checks and balances to prevent it from occurring in the future. As noted in the incident updates, we discovered a way to improve transparency of such incidents, namely to display failing transactions in our dashboard. This improvement is currently being implemented and is intended to be in place within a few weeks. The investigation of this incident identified additional, potential improvements \(regarding the uncertainty about the transaction state of a failed transaction\) which even touches upon extending our API with “idempotency”. If you are a partner and have an opinion or request in regards to this, please feel free to reach out to us, perhaps even through an issue on our [public documentation](https://github.com/clearhaus/gateway-api-docs). Sincerely, your operations team

We have been monitoring transactions vigorously and have gotten confirmation from our service provider that the issue is remedied and they are processing normally. We have initially confirmed that our fail-over mechanisms and manual actions to mitigate were appropriate, but we will continue to find potential improvements.

Our service provider has confirmed the issue and we are actively working to understand what caused it to occur. Please be advised that the failing requests do not appear in the dashboard; however, the HTTP responses were appropriately returned.

Between 2020-06-16 11:06:22 UTC and 2020-06-16 11:28:24 UTC authorizations and voids failed with status code 50000. We are continuously investigating and will post an update as soon as possible. For now, service has been restored.

Upwards of 9 percent of transactions in the time span 2020-04-27T05:37:45+00:00 - 2020-04-27T06:50:11+00:00 failed due to an issue with currency conversion. The issue has been solved and we will ensure the gateway’s currency conversion becomes more fault-tolerant as soon as possible.

A database issue caused failing requests in the following timeframe: 12:16:43 - 12:24:35 UTC This did not affect our ability to process transactions.

A database failover caused failing requests in the following timeframe: 11:03:04-11:11:14 UTC This did not affect our ability to process transactions or settle merchants.

A few authorisation attempts in the time span 2020-01-23T09:23:24+00:00 - 2020-01-23T09:25:50+00:00 failed due to an incorrect firewall configuration. The configuration has been corrected.

This incident has been resolved.

The issue has been resolved.

Due to a service-provider outage, back-office authentication is currently unavailable. Our ability to process transactions and funding is not affected.