Is OpenPath Down Right Now? Check if there is a current outage ongoing.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

We are currently looking into this automated incident report. At the moment is looks like the monitoring system was failing from 2:13AM and resolved itself at 3:01AM.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 0 request(s) ran and 0 test(s) failed.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 2 test(s) failed.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

We will be updating our portals today with service patches. These service patches will not affect any payment transactions coming through the system; however, you may get logged out of your portal during these updates or see short unavailability of these portals between 30 to 60 seconds.

Standard Patch Releases

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 2 test(s) failed.

Standard Patch Releases

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 2 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

We are aware of ongoing issues with the Transaction API and are actively working to identify the cause.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

Technical Issue Report #27 Summary On February 4, 2025, at 3:45 AM, a technical issue occurred, lasting 4.5 hours, which resulted in misplaced decimals in transaction amounts and led to customer overcharges. All affected transactions have been reversed. The issue stemmed from a format change related to a new 3DS feature. It was detected at 7:45 AM through merchant decline alerts, and a hotfix was deployed by 8:25 AM. In total, nine customers experienced financial impact, while 178 were unable to transact. Post-Mortem 1. Incident Summary 1. Date & Time of Incident: Tuesday February 4th, 2025 at 3:45 AM 2. Duration: 4.5 Hours 3. Impact: Consumers using the OpenPath and PaymentLync were being charged too much due to decimal placement 4. Severity Level: **Critical** 5. Reported By: Jason Martin at OpenPath 2. Root Cause Analysis 1. What Happened? An OpenPath update was released which had several new updates, one of the updates for PaymentLync caused Consumers to be billed two decimal places over, so $25.00 would become $2,500.00. 2. Root Cause: To facilitate the new 3DS data only feature with PaymentLync, OpenPath changed the Format of the amount being sent from 2500.0 to 2500. This change was made because at the time of testing the 3DS endpoint was only accepting integers. 3. Detection & Response 1. How Was It Detected? We received automated alerts of Multiple declines from Merchants at 7:45 AM and began investigating, when he found the issue. 2. Initial Response Actions: At 8:10 AM we met with the development team and created a hotfix which was deployed and live at 8:25 AM. 3. Time to Detection: 4 Hours 4. Time to Resolution: 20 Minutes 4. Impact Assessment 1. Users Affected: 9 Customers financially impacted which needed to be made whole again. 178 Customers that were unable to transact but were not financially Impacted. 2. Systems Affected: OpenPath PaymentLync Transaction API 3. Business Impact: Revenue loss, Reputational Damage for OpenPath, PaymentLync and Merchants.

There are a high amount of declines for NSF on transactions flowing through PaymentLync.

This issue has now been resolved.

This is a continued monitoring issue from the previous week. The issue is not affecting transactions, or any systems and the alerts are false positives. The resolution from last week solved the issue happening multiple times a day; however, is still triggering once to twice a day. We currently have some monitoring fixes pending review a deployment today January 2nd, 2025.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

Healthcheck configuration in our Platform AppService has been successfully updated. The service is now fully operational and functioning as expected.

Intermittent Platform API error was identified and our dev team worked in the fix

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

We have deployed a new group of API servers to our existing groups of API servers. This new group is occasionally producing false positives with our automated monitoring systems. We are currently working on correcting this issue; however, at this time it is not affecting any of our services.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This issue has been resolved, for the Postmortem please see: https://asset.openpath.io/download/pdf/connection-pool-exhaustion-issue.pdf

To restore services as quickly as possible, we temporarily diverted traffic. At this time, all systems appear to be functioning as expected. We will now conduct a deeper investigation into the issue and provide updates here as soon as new information becomes available.

We are currently reviewing this issue and will update as soon as we have more information available.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

A 504 timeout occurred between Azure Application Gateway and the Transactions API App Service, which had stopped and failed to auto-restart. I manually restarted it and added a Health Check to ensure auto-restart in the future. No service was impacted, as requests were handled by active VM instances.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 2 test(s) failed.

Upon investigation, there was a 20 second timeout on the Microsoft Azure Application Gateway, this timeout did not impact any systems.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 2 test(s) failed.

There was a 2 minute outage on 1 API server in our server cluster that occurred during a standard update, the issue was identified immediately and resolved right away.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

We identified a long running process on our Subscription endpoint of the Platform API, this issue has been resolved and patch to ensure the issue does not happen again will be applied today.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 5 request(s) ran and 0 test(s) failed.

This incident has been resolved.

We have identified the issue caused with backup of queued data and have throttled back the execution of these queues, which has restored all performances.

We are noticing some slow downs in performance which we believe is related to our data queue flow. We are currently investigating. We are currently monitoring the impact; however, transaction data and client interfaces all appear to be functioning.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

All systems are currently working we are Monitoring and Testing for the next 24 Hours, transactions that went through will also be visible in the Enterprise Back Office at this time.

This issue has been downgraded to Minor, transactions are flowing correctly, and data is flowing; however, the reports in the Enterprise UI are still not reflecting the current transaction flow. We are rectifying this issue but have verified that the data is in the database.

All migrations have completed and we are connecting back up the services.

An issue with our databases is causing performance issues with all our platforms. We are currently moving our systems over a redundant replicated database, to restore systems to normal usage. This task has been started and we are waiting for it to complete. Will update as soon as the roll over is complete.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 2 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 1 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 0 test(s) failed.

This incident has been resolved.

Monitor System Down Checking, on Collection System Down Checking, did not run successfully. 4 request(s) ran and 4 test(s) failed.