Central 1

Is Central 1 Down Right Now? Check if there is a current outage ongoing.

Central 1 is currently undergoing Planned Maintenance

Last checked from Central 1's official status page

Historical record of incidents for Central 1

Report: "INC203030 - RESOLVED - EFM Outage June 12"

Last update
resolved

On June 12, 2025, between 11 am and 2:10 pm PT (2 and 5:10 pm ET), a service disruption affecting AWS resulted in approximately 5,200 transactions not being successfully transmitted to the Enterprise Fraud Management (EFM) platform. Please note that Interac continued to monitor transactions for suspicious activity throughout the duration of the outage. The issue has now been fully resolved, and all services are functioning as expected. We are continuing to monitor the environment closely and will provide further updates, if necessary. We sincerely regret any inconvenience this may have caused and appreciate your understanding. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "INC202834 - Missing AS (AFT) Settlement for June 9 ON03 File"

Last update
investigating

**This notice is for those organizations in AB, SK, MB, and ON, that receive an ON03 AFT posting file from Central 1** The ON03 AFT posting file for June 9 has not yet been settled to your Central 1 account. Our technical support team is investigating. We will provide an update by Wednesday, June 11, at 12pm PT (3 pm ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "INC202531 - Missing May 31 Everlink POS and SCD Settlement"

Last update
identified

This notification is for Ontario and British Columbia credit unions that use Everlink as their ATM/POS service provider. Everlink’s settlement for May 31st was not processed. We will provide our next update by Friday, June 6th, 9 a.m. PT / 12 p.m. ET or sooner if the incident has been resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "CHG159871 - Digital Banking OIDC Release 1.0.53"

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Please be advised that Digital Banking OIDC Release 1.0.53 has been scheduled for Tuesday June 3, 2025 at 1 a.m. PT (4 a.m. ET).Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878

Report: "Discontinuation of M2MC and M2MU Report File Delivery – Effective May 30, 2025"

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

As part of Central 1’s initiative to streamline backend payment processing and retire legacy systems, we have completed the transition to discontinue immediate Me-to-Me (M2M) and Business-to-Business (B2B) payments.Please see Processing Me-to-Me and Business-to-Business transactions through Central 1 for more details.What’s changing?*All M2MC and M2MU report files have been consistently empty due to the deprecation of immediate M2M/B2B transactions.*Starting Friday, May 30, 2025, Central 1 will stop generating and delivering empty M2MC and M2MU files to credit union SFTP folders.This change helps reduce unnecessary file traffic and aligns with the completed M2M/B2B retirement project. If your credit union is still receiving these files, no action is required—they will simply no longer appear after May 30.Central 1 Support at support@central1.com or 1.888.889.7878, press 1.

Report: "RESOLVED INC202095 - Proxy Error in Online Tracing (OLT) when Creating Reports"

Last update
resolved

Central 1 identified an issue in the bill payment Online Tracing (OLT) application where users encountered a proxy error when generating reports. The incident has been fully resolved. Our team has worked diligently to address the issue, and all systems and services are now functioning as expected. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

Report: "RESOLVED INC202095 - Proxy Error in Online Tracing (OLT) when Creating Reports"

Last update
Resolved

Central 1 identified an issue in the bill payment Online Tracing (OLT) application where users encountered a proxy error when generating reports. The incident has been fully resolved. Our team has worked diligently to address the issue, and all systems and services are now functioning as expected. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

Report: "INC196440 - Service Disruption: Impact on Outbound Wires and Potential OIDC Service Interruption"

Last update
postmortem

**Postmortem: INC196473 – Azure Network Outage Affecting Outbound Wires and Potential OIDC Interruption** On the morning of March 3, 2025, Microsoft experienced a widespread network disruption impacting connectivity to Azure and Microsoft 365 services for customers accessing their Toronto edge locations. The incident stemmed from a hardware failure on a core network device, followed by a secondary failure on its redundant pair. These cascading issues led to packet loss across services such as Azure ExpressRoute, Virtual Machines, Storage, App Services, and Kubernetes clusters. Central 1, like many Microsoft customers in Canada, was indirectly impacted. We initially observed a delay in outbound wire processing and potential risk to authentication services \(OIDC\). Fortunately, throughout the incident, OIDC login and Digital Banking Services remained fully operational. Our teams monitored the situation closely while Microsoft engineers worked to resolve the root issue. Microsoft confirmed that the disruption was due to increased memory pressure on a line card within their Toronto edge networking equipment. When both the active and redundant devices failed, packet loss began to affect customer traffic. The Microsoft Engineering team isolated the faulty line card and successfully rerouted traffic to healthy paths by 10:37 a.m. PT \(18:37 UTC\). Connectivity and services were restored, and stability has since been confirmed. Following this event, wire and e-Transfer services have resumed as expected. Central 1 took no direct action to resolve the issue, as the fault was fully within Microsoft’s infrastructure. We will continue to monitor performance but consider this incident resolved. For customers interested in Microsoft’s findings and recommendations, their Preliminary Incident Report is available, with a final RCA expected within 14 days. No further mitigation is required on the part of Central 1 or its clients. For any questions about this postmortem please contact [support@central1.com](mailto:support@central1.com)

resolved

Microsoft was able to resolve their incident impacting their customers connectivity to resources through network routes in Canada. They identified an issue with a network device and took steps to move traffic to healthy routes. Central 1 observed no impact to OIDC login or other Digital Banking Services throughout the Microsoft incident. Wires and e-Transfers have been released, and all other services have recovered. We will continue to monitor services throughout the day and resolve the incident after confirmed stability.

identified

Microsoft has begun recovering their DNS services. Central 1 wanted to highlight additional impacts that may have been experienced during the Microsoft outage to our services which have begun to recover. e-Transfers held for decisioning in EFM are delayed to be processed. You can still mark the record as "fraud" or "not fraud," and the e-Transfer will be queued and released once DNS services recover. There is no impact on sending or receiving e-Transfers that are not flagged as potential fraud, which represents the vast majority of transactions. Additionally, CAS Online may not be available for some users, as certain services rely on Azure (Microsoft). There is no impact to Digital Banking logins, which remain fully operational. Central 1 continues to explore and implement workarounds to restore services while Microsoft works to resolve their DNS issues. We will provide our next update by 12 p.m. PT (3 p.m. ET).

identified

Central 1 is currently experiencing a service disruption due to ongoing issues with Microsoft Azure, which Central 1 relies on for certain services. As a result, outbound wire transfers are currently paused, with transactions queuing until the issue is resolved. Our team is actively investigating the full extent of the impact, and we are closely monitoring the situation. Please note that our OIDC (OpenID Connect) service may also be affected if there is a DNS impact, as the Time-to-Live (TTL) threshold approaches. We will provide further updates as more information becomes available. If you have any questions, please reach out to our support team. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

Report: "CHG153870 - Digital Banking Core Release 744"

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Please be advised that Digital Banking Direct Core Production Release 744 has been scheduled for Wednesday May 28, 2025 at 1 a.m. PT (4 a.m. ET).Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878

Report: "Resolved - INC201774 – Issue with “Accept” Function for Incoming Wires in PaymentStream Direct (PSD)"

Last update
resolved

Change CHG159937 was implemented last night in production to correct the issue with the "Accept" functionality in PaymentStream Direct (PSD) for incoming wires. Please, contact Client Support if you continue to have issues with the “Accept” functionality in PSD. Central 1 - Support@central1.com - 1.888.889.7878, press

identified

Change CHG159937 will be implemented in production this evening between 7 and 10 p.m. PT (10 p.m. to 1 a.m. ET) to correct the issue with the "Accept" functionality in PaymentStream Direct (PSD) for incoming wires. We will provide our next update by Tuesday, May 27, at 11:00 a.m. PT (2:00 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press

identified

Central 1 is currently investigating an issue with the "Accept" functionality in PaymentStream Direct (PSD) for incoming wires. When attempting to use the “Accept” button on an incoming wire, users may encounter an error. The purpose of the “Accept” button is to reflect a status change in PSD, moving the wire from “Pending” to “Completed” once the credit union has posted the funds to a member’s account. This function does not affect the actual settlement of funds, which has already occurred when the wire enters PSD. There is no impact to your members—funds have already been received and can be posted to member accounts as usual. A fix is scheduled for deployment early next week, and we will continue to provide updates as we progress toward resolution. We will provide our next update by Monday at 11:00 a.m. PT / 2:00 p.m. ET. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "Resolved - INC200050 - Incoming\Outgoing wires incident"

Last update
postmortem

Postmortem: INC200050 – Incoming/Outgoing wires incident – Priority High On Monday, April 28, Central 1 experienced a system disruption impacting incoming and outgoing wires between 4:30 and 7 a.m. PT \(7:30 to 10 a.m. ET\). Central1 could not connect to IBM’s Managed File Transfer \(MTF\) environment \(swift connection\), which meant that incoming and outgoing wires could not be processed.  IBM implemented a fix to their MTF server at 7 a.m. PT \(10 a.m. ET\). After the fix was applied, new wires were created and using the automated workflows to process. The outgoing wires that were created during the outage remained in pending status until they were manually released. All the impacted wires were released by 12 p.m. PT \(3 p.m. ET\). Root cause analysis: The cause of this issue was a planned change executed on April 26 by IBM to upgrade their host proxy for the Managed File Transfer \(MTF\) environment. During the upgrade, the new Amazon EKS \(Elastic Kubernetes Service\) Virtual Machines were not automatically registered with the proxy Load Balancer. Central 1 was not informed of this maintenance, as it is part of IBM’s regular maintenance with no expected impact on the service. PRB011511 – Root cause analysis actions: IBM has taken several corrective and preventive actions to reduce the risk of recurrence: ‌ o   The EKS upgrade procedures have been enhanced to include post-upgrade validation of proxy registration and target group health checks. ‌ o   The MFT Sanity Check process has been updated to validate connectivity and proxy accessibility explicitly. ‌ o   Proxy Load Balancer monitoring and alerting have been enhanced, ensuring real-time detection if similar issues arise. ‌ o   A support case was opened with AWS to further investigate the root cause of the failed auto-registration of the EKS worker nodes. ‌ o   Internally, IBM has reinforced its communication protocols to ensure timely client notifications when anomalies are detected, even before client impact is confirmed. Central 1 remains committed to delivering reliable service and continuously improving our systems and processes. If you have any questions about this postmortem, please contact [support@central1.com](mailto:support@central1.com)

resolved

The outgoing wires that were not released this morning have now been processed. Central 1 - Support@central1.com - 1.888.889.7878, press 1

monitoring

We continue to work with our technical partner to release the outgoing wires sent before the fix at 7 a.m. PT (10 a.m. ET) this morning. An update will be provided to you at 1 p.m. PT (3 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

monitoring

Our technical partner continues to manually release the outgoing wires sent prior to the wires fix this morning. All incoming and new outgoing wires are processing. An update will be provided to you at 11 a.m. PT (1 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

Incoming wires and new outgoing wires are now processing. We are working with our technical partner to get 81 outgoing wires sent prior to the fix released for processing. An update will be provided to you at 9 a.m. PT (12 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

investigating

Incoming wires are now processing. We continue to experience an issue with outgoing wires. We are working with our technical partner and will let you know as soon as the outgoing wires start processing. An update will be provided to you at 8 a.m. PT (11 a.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

investigating

Central 1 currently cannot process incoming or outgoing wires. We have engaged our technical partners in the investigation of this high-priority incident. An update will be provided to you at 7 a.m. PT (10 a.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "Resolved - INC201774 – Issue with “Accept” Function for Incoming Wires in PaymentStream Direct (PSD)"

Last update
Resolved

Change CHG159937 was implemented last night in production to correct the issue with the "Accept" functionality in PaymentStream Direct (PSD) for incoming wires. Please, contact Client Support if you continue to have issues with the “Accept” functionality in PSD. Central 1 - Support@central1.com - 1.888.889.7878, press

Update

Change CHG159937 will be implemented in production this evening between 7 and 10 p.m. PT (10 p.m. to 1 a.m. ET) to correct the issue with the "Accept" functionality in PaymentStream Direct (PSD) for incoming wires. We will provide our next update by Tuesday, May 27, at 11:00 a.m. PT (2:00 p.m. ET).Central 1 - Support@central1.com - 1.888.889.7878, press

Identified

Central 1 is currently investigating an issue with the "Accept" functionality in PaymentStream Direct (PSD) for incoming wires.When attempting to use the “Accept” button on an incoming wire, users may encounter an error. The purpose of the “Accept” button is to reflect a status change in PSD, moving the wire from “Pending” to “Completed” once the credit union has posted the funds to a member’s account. This function does not affect the actual settlement of funds, which has already occurred when the wire enters PSD.There is no impact to your members—funds have already been received and can be posted to member accounts as usual.A fix is scheduled for deployment early next week, and we will continue to provide updates as we progress toward resolution.We will provide our next update by Monday at 11:00 a.m. PT / 2:00 p.m. ET.Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "INC200385 – Online banking and e-Transfer service degradation"

Last update
postmortem

**Postmortem: INC200385 – e-Transfer and OAuth/OIDC Login Outage Due to UCP Saturation** On Friday, May 2nd, Central 1 experienced a significant service degradation beginning at approximately 3:15 p.m. PT \(6:15 p.m. ET\) affecting e-Transfer transactions, digital banking login via OAuth/OIDC, RSA step-up authentication, Remote Deposit Capture \(RDC\), and select PaymentStream Direct \(PSD\) services. The degradation was traced to issues originating from our Universal Connection Proxy \(UCP\) infrastructure. Duration: 1 hour and 45 minutes for e-Transfer; 61 minutes for other impacted services. **Point of failure:** The trigger for this incident was a service degradation of service at Interac, a third-party provider critical to e-Transfer and authentication functions. Interac’s system failure \(which they have attributed to a change with unanticipated impact\) led to a surge of failed connection attempts which saturated UCP’s thread pool. This mirrored the failure pattern observed in previous incidents \(e.g., INC198347 on March 31, and INC189678 on November 1, 2024\), where upstream instability created cascading system pressure within our UCP gateway, degrading dependent services. Previous mitigation efforts implemented after these prior events \(including thread monitoring, timeout tuning, and increased infrastructure capacity\) proved insufficient against this hard failure scenario at Interac. At 4:16 p.m. PT \(7:16 p.m. ET\), Central 1 made a deliberate decision to disable the UCP connection to Interac, to enable recovery of other systems. Within moments, all affected Central 1 \(OAuth/OIDC login, RDC, and PSD\) services stabilized. E-Transfers remained offline until service recovered at Interac, and until Central 1 restored the UCP connection to Interac at 5:08 pm PT \(8:08 pm ET\).  **Corrective Actions & Next Steps:** While efforts were made to insulate our systems from upstream failure, this event has highlighted further resilience gaps within the UCP architecture. The following actions are underway: PTASK0010310 – _UCP Monitoring Enhancements_ DUE: June 2025 * Ongoing implementation of deeper thread-count and connection-failure monitoring. PTASK0010307 – _Latency & Timeout Tuning DUE:_ July 2025 * Further reduction of timeout thresholds for Interac connections to improve fail-fast behavior. PTASK0010308 – _Circuit Breaker Design Review DUE: July 2025_ * Technical assessment underway to implement a dynamic circuit breaker model that would prevent cascading failures during third-party outages. NEW – PTASK0010411 – _UCP Auto-Isolation Rules for Third-Party Failures DUE: August 2025_ * Implement logic to programmatically isolate known failing endpoints \(like Interac\) while preserving other UCP services. Central 1 understands the significant disruption that even brief outages cause to our clients and their members. We remain committed to strengthening the resilience of our systems against third-party instability and will continue to evolve our architectural safeguards to minimize service disruptions and isolate failures more effectively. If you have any questions about this postmortem, please contact [support@central1.com](mailto:support@central1.com).

resolved

Login issues and other UCP services were resolved at 4:16 p.m. PT (7:16 p.m. ET). Interac was able to recover their e-Transfer service allowing Central 1 to restore service at 5:08 p.m. PT (5:08 p.m. ET). The incident has been resolved, and Central 1 will work with Interac on the postmortems root cause analysis to mitigate these types of events.

identified

Central 1 has disabled our connections to Interac. All other services have recovered. e-Transfers will remain offline until Interac advises that the e-Transfer service has recovered. Central 1 will then re-enable the feature. Next update by 5:45 p.m. PT (8:45 p.m. ET) or sooner.

identified

The root cause of the incident is an incident with Interac. Technical teams are working to disable the connections to Interac to relieve pressure which will allow Digital Banking, Remote Deposit, RSA, and PSD services to recover. Next update by 5:30 p.m. PT (8:30 p.m. ET) or sooner.

investigating

Central 1 is experiencing a degradation in services starting at 3:15 p.m. PT (6:15 p.m. ET). Digital Banking Login via OAuth or OIDC is degraded, as well as legacy login with Increased Auth fail-closed. e-Transfer and Remote Deposit services are also degraded for all customers during this time. Technical teams are engaged to restore service as quickly as possible. Central 1 – Support@centrral1.com | DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "RESOLVED: INC200663 – Wires and Remote Deposit Capture Degradation - 7 May 2025"

Last update
postmortem

**Postmortem - INC200663 – PaymentStream Direct \(PSD\) wires outage**  On Wednesday, May 7 at 8:08 a.m. PT \(11:08 a.m. ET\), a routine change intended to improve Kubernetes service capacity unexpectedly removed critical Wires dependencies, which restricted our PaymentStream Direct Wires \(PSD Wires\) application from communicating with internal dependent systems.  As a result, while users could submit wires for processing, and Central 1 could receive inbound wires from external Financial Institutions, we could not process them, thus they remained in pending status until system recovery ws completed. The nature of the Incident also prevented an automated recovery, requiring extensive manual intervention to restore application functionality.  Full service was restored by 11:25 a.m. PT \(2:25 p.m. ET\). Point of failure: The incident was caused by human error during a production deployment. A missing configuration allowed namespace deletion, resulting in widespread service impact and a complex manual recovery process. **PRB011524 – Root cause analysis actions** * Re-enable Sync Protection – COMPLETED * Review and update Helm Chart with all required secrets and configurations – COMPELTED * Review shared pipelines between Production and Disaster Recovery environments – DUE: June * Include a step in the CI/CD pipeline to regenerate MuleSoft activation tokens – DUE: June * Commit reviewers will validate changes outside of Charts – DUE: June * Implement audit log monitoring to alert when secrets or namespaces are deleted – DUE: June * Improve Disaster Recovery environments for these services – DUE: July We recognize the disruption this caused and are committed to learning from this incident to reduce risk and improve our resiliency going forward. If you have any questions regarding this postmortem, please contact [support@central1.com](mailto:support@central1.com)

resolved

The wires degradation incident has been resolved, and all wires are now sending and being received through our services without delays. Wires created in PS Wires during the incident have been processed. Additionally, there was an impact to some of our customers remote deposit capture (Deposit Anywhere) service between 8:10 to 9:10 a.m. PT (11:10 a.m. to 12:10 p.m. ET). Members would have seen an error knowing that their was a deposit issue. Central 1 will continue to monitor our services to ensure stability and prioritize the postmortem within the next two weeks. Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

We continue to triage the ongoing wires disruption. We have isolated the root cause of the disruption and are working to restore full service. Wires can still be created and reviewed in PS-Wires, however they will not be processed until this issue is resolved. Central 1 can also receive external wires into our environment but cannot process them for our clients to accept until this matter is resolved. An update to this incident will be provided by 11:30 a.m. PT (2:30 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

Please be advised that Central 1 is experiencing wires processing degradation, preventing the completion of send/receive wires; wires can still be created and reviewed in PS-Wires. The wires API is failing due to a processing error. We are actively investigating to restore full wire service and will provide an update by 10:30 a.m. PT (1:30 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "CHG159303 - Secure File Transfer Protocol (SFTP) service upgrade – Phase 2: Cipher Updates on May 26"

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Central 1 will be updating the types of cryptographic ciphers we support for our SFTP service on Monday, May 26th, to align with the latest security standards. This could have an impact to your organization’s ability to connect if you need to upgrade your SFTP connectivity software and/or processes.This was communicated by Central 1 via our Client Centre (Test your Secure File Transfer (SFTP) service connection by May 26). We believe with the diligence of all participants that there should be minimal disruption during the upgrade. If you have any questions, or issues connecting to the SFTP server please email support@central1.com or call the Central 1 Support line at 1.888.889.7878 Option 1.

Report: "CHG159104 - File Transfer Protocol (FTP) Server Maintenance"

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Central 1 will be performing routine security patching on the FTP server this Sunday, May 25, from 9 to 10 a.m. PT (12 to 1 p.m. ET). During this time, the FTP server will not be available. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "RESOLVED: INC201689 – PaymentStream Direct wires errors"

Last update
resolved

PSD wires service was degraded between 8:42 to 5:59 a.m. PT (11:42 to 11:59 a.m. ET) this morning. After an application restart the service has remained stable and will be closely monitored throughout the day. Central 1 is investigating the point of failure with high urgency and will provide a postmortem with our findings within two weeks.

investigating

Central 1 is currently investigating errors with PaymentStream Direct (PSD) wires. Reported errors include a missing wires navigation, template errors, or problems creating wires. Central 1 is investigating this with high urgency and will provide an update by 10:30 a.m. PT (1:30 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "RESOLVED: INC201689 – PaymentStream Direct wires errors"

Last update
Resolved

PSD wires service was degraded between 8:42 to 5:59 a.m. PT (11:42 to 11:59 a.m. ET) this morning. After an application restart the service has remained stable and will be closely monitored throughout the day.Central 1 is investigating the point of failure with high urgency and will provide a postmortem with our findings within two weeks.

Investigating

Central 1 is currently investigating errors with PaymentStream Direct (PSD) wires. Reported errors include a missing wires navigation, template errors, or problems creating wires.Central 1 is investigating this with high urgency and will provide an update by 10:30 a.m. PT (1:30 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "CHG153954 - MDB and Forge Commercial Release 513"

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Please be advised that MDB and Forge Commercial Production Release 512+513 has been scheduled for Wednesday May 21, 2025 at 1 a.m. PT (4 a.m. ET).Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "INC201163 – RESOLVED - Some Corporate Remote Deposit Capture (cRDC) Clients receiving error on deposits"

Last update
resolved

The fix (CHG159573) for this incident was successfully deployed last night. The affected Corporate Capture clients should no longer receive an error when submitting their deposits.

identified

We are continuing to work on a fix for this issue.

identified

Central 1 has identified the point of failure for the deposit errors some cRDC members were experiencing. A fix (CHG159573) will be going in this evening to resolve the incident. We will monitor the service tomorrow and provide a final update confirming the error has been fixed.

identified

Some business members using Corporate Remote Deposit Capture (cRDC) are receiving a “call to deposit notification service failed” error message. This is occurring if the Corporate Clients’ deposit account at your organization is less than 12 digits. Currently there is no workaround. Our technical support is actively investigating this incident, and we will provide an update by 3:00 pm PT (6:00 pm ET). Central 1 Support - Support@central1.com - 1.888.889.7878, press 1

Report: "Alert! - INC200817- Prev-test-comp environment currently unavailable"

Last update
resolved

Login issues were resolved at 11:43 a.m. PT (2:43 p.m. ET)

investigating

MemberDirect/Forge is experiencing technical issues within the prev-comp-test (PCT). Staff may see HTTP ERROR 504 or a blank screen when trying to access. We are working on restoring our test environment and will provide an update by 1 p.m. PT (4 p.m. ET). Intellect - DigitalBanking_Support@Central1.com - 1.888.889.7878, option

Report: "RESOLVED: INC198965 - Brief Interac e-Transfer outage"

Last update
postmortem

**Postmortem: INC198965 – e-Transfer 3.4 and 3.5 Brief Outage** On Wednesday, April 9th, Central 1 experienced a brief disruption impacting e-Transfer services between 12:50 and 12:54 p.m. PT \(3:50 to 3:54 p.m. ET\). Services recovered automatically without requiring intervention. Our UCP system was temporarily unable to communicate with Interac through our Vancouver \(VAHC\) network connection. Despite this, the UCP system remained responsive throughout the incident, and Interac confirmed they observed no issues on their end. The incident was escalated to our Network team for a deeper review of underlying network-layer activity. Root cause analysis: Further investigation by our Network team determined that a firewall error briefly disrupted connectivity between UCP and Interac. The firewall rule was quickly corrected, restoring full service. As part of our corrective actions, we are enhancing our network-layer logging to improve visibility and speed up future investigations should similar events occur. Central 1 remains committed to maintaining reliable service and strengthening our diagnostic capabilities. **PRB011500** – Root cause analysis actions PTASK0010325 – Network process enhancements - COMPELTED * To prevent recurrence, we’ve implemented improved network governance practices and completed process enhancements to increase safeguards around firewall rule changes. Additionally, we are enhancing our network-layer logging to improve visibility and accelerate future investigations. Central 1 remains committed to delivering reliable service and continuously improving our systems and processes. If you have any questions about this postmortem, please contact [support@central1.com](mailto:support@central1.com).

resolved

Central 1 experienced an Interac e-Transfer service outage today 12:50 to 12:54 p.m. PT (3:50 to 3:54 p.m. ET) (< 4minutes). During the outage users would have seen an errors when trying to send or receive an e-Transfer using MemberDirect/Forge digital banking (3.4/3.5) or API services. Services have remained stable and Central 1 is investigating the root cause which will be published in a postmortem within two weeks. Please contact Support if you experience any further issues. Central 1 - support@Central1.com - 888.889.7878

Report: "RESOLVED - INC198347 – Online banking and e-Transfer service degradation"

Last update
postmortem

**Postmortem: INC198347 - e-Transfer Outage \(3.4, 3.5 and API\) for 17 minutes** On Monday, March 31st, Central 1 experienced a temporary degradation in e-Transfer services, Remote Deposit Capture \(RDC\), and digital banking login via OAuth and OIDC between 3:16 and 3:33 p.m. PT \(6:16 to 6:33 p.m. ET\). The issue resolved without intervention, and services remained stable for the remainder of the day. The root cause was traced to third party latency causing congestion on our UCP \(Unified Connection Proxy\).  This is the same failure point as the November 1st, 2024 outage \(INC189678\), where similarly, an issue with Interac triggered cascading effects across our systems. In both incidents, a degradation in Interac’s service caused a spike in failed connection attempts, which led to a saturation of threads in Central 1’s UCP pool. The UCP acts as a gateway managing high volumes of simultaneous connections to third-party partners, and the surge in thread consumption introduced latency across multiple services. Following the November incident, we implemented additional monitoring and increased capacity within our UCP infrastructure to better insulate against upstream partner failures. However, those measures did not prevent this recurrence. As a result, we have launched a deeper review of our high-availability architecture and UCP failover design. Several targeted improvements are now underway, including adding server capacity in TOHC to mirror the configuration in VAHC, tuning timeout thresholds for upstream calls, and exploring the reintroduction of a circuit breaker model to isolate problematic connections. **PRB011492** **– Root Cause Analysis** PTASK0010310 – Improved Monitoring – Due date: June * Shared system team to investigate with monitoring team for any additional monitoring that could be implemented to monitor thread counts PTASK0010307 – Adjusting latency for UCP timeout value – Due date: July * Investigate reducing the timeout value to our Interac connection. PTASK0010308 – Investigate a Circuit Breaker Implementation in UCP – Due date: July * Additional review of a circuit breaking approach that would notice that connections are failing and would stop creating new connections that would start consuming threads. PTASK0010309 – Interac incident Report - COMPELTED * Ensure Interac provides an Incident Report to review their mitigation efforts to prevent these service disruptions from occurring. While the March 31st incident was brief, we understand even short disruptions can significantly affect our members. We remain committed to learning from every incident and evolving our systems to handle the unpredictable nature of third-party dependencies with greater agility. Our goal is to minimize the blast radius of external failures and reinforce the reliability our clients depend on. If you have any questions about this postmortem, please contact [support@central1.com](mailto:support@central1.com). Jason Seale - Director of Client Support Services, [jseale@central1.com](mailto:jseale@central1.com) | 778.558.5627

resolved

Central 1 experienced a degradation in services between 3:16 to 3:33 p.m. PT (6:16 to 6:33 p.m. ET). Digital Banking Login via OAuth or OIDC would have been degraded, as well as legacy login with Increased Auth fail-closed. e-Transfer and Remote Deposit services would have also been degraded for all customers during this time. Services have recovered and remain stable. Central 1 is prioritizing an investigation on our UCP service and will provide a postmortem within the next two weeks. Central 1 – Support@centrral1.com | DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "INC199785 – Cash Ordering Returns and NDDS Deposit Reporting Delay"

Last update
resolved

The CAD and USD cash returns (Cash parcel) and NDDS deposits for Apr 23 settlement has now been posted to your organizations current account. The transactions are not on today's AIBC, AIBU, AIOC, AIOU, CDJ, settlement reports or the PREC or OREC files. The postings will appear on tomorrow’s reports with the effective date of April 23. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "Resolved - INC199724 – FX Notes Plus not available"

Last update
resolved

This incident has been resolved.

investigating

The FX Notes Plus application is now available. If you continue to experience log on issues, please contact Central 1 Support. Central 1 - Support@central1.com - 1.888.889.7878, press 1

investigating

Exchange Bank of Canada (EBC) has notified Central 1 they are experiencing system issues and users are receiving a “User has been disabled” message when trying to access the FX Notes Plus application. Central 1 and EBC are investigating and we will provide an update by 10 a.m. PT (1 p.m. ET) or sooner if the incident is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "RESOLVED: INC198745 – OIDC failed login on Sunday, April 6"

Last update
postmortem

**Postmortem: INC198745 – Login errors for OIDC service** On Sunday, April 6th, customers using OIDC for login experienced a service interruption starting at 1:00 a.m. PT \(4:00 a.m. ET\). The issue was resolved by 4:20 a.m. PT \(7:20 a.m. ET\), lasting approximately 1 hour and 20 minutes beyond our scheduled maintenance window. **Point of failure**: During a planned change to our server environment early Sunday morning, a system component restarted in a way that preserved temporary files it shouldn't have. These leftover files interfered with the login system's startup process, preventing it from functioning correctly. While our team had anticipated this kind of restart and included a script designed to clear out these temporary files, it didn’t work as expected in this case. This specific type of restart hadn’t been done in our production environment before, so the issue had not previously surfaced. A permanent fix was developed and deployed on April 9th, ensuring that all temporary files are properly removed no matter how the system is restarted. This will prevent the issue from reoccurring and improve the reliability of our login system. **PRB011499 – Root Cause Analysis** PTASK0010313 - Deploy fix for OIDC git initialization script to Production – COMPLETED * The git initialization script used to fetch the latest OIDC per-CU configuration files at startup will be updated to address a container restart scenario. PTASK0010314 - Document Kubernetes behavior during restarts - COMPLETED * Updated playbooks to ensure the difference in start behavior of the Kubernetes cluster/app when performing restarts vs an upgrade. We recognize the impact this disruption had on our members and their business operations. We are taking this incident seriously and implementing targeted improvements to enhance system resilience and accelerate detection and recovery in the future. If you have any questions about this postmortem, please contact [digitalbanking\_support@central.1com](mailto:digitalbanking_support@central.1com)

resolved

Please be advised there was an OIDC outage yesterday, Sunday, April 6 impacting login service for OIDC customers for 3 hours and 20 minutes. Outage began at 1:00 a.m. PT (4:00 a.m. ET) and started to recover by 4:20 a.m. PT (7:20 a.m. ET). Services fully recovered by 6 a.m. PT (9 a.m. ET) A postmortem is being completed within the next two weeks. Client Support Services T 888 889 7878 Option 2 / DigitalBanking_Support@central1.com

Report: "INC199059 – RESOLVED - FX Drafts Settlement Delay - April 10"

Last update
resolved

The settlement file for the April 10 FX Drafts has been received by Convera and posted effective April 10. There are no further outstanding settlements for FX Drafts.

monitoring

It was previously reported that the FX Drafts settlement was missing for April 9 and 10. Although the delayed settlement for April 9 was received and posted, the settlement for April 10 remains outstanding. Central 1 has not received the required amount files from our settlement partner, Convera. Settlement for April 11 was successfully received and posted. The matter has been escalated to Convera for urgent investigation. At this time, no workaround is available, as the necessary data has not been provided for manual processing. We will provide the next update by Tuesday, April 15 at 10:00 a.m. PT (1:00 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "Resolved - Alert! INC199059 – FX Drafts Settlement Report Delay"

Last update
resolved

The blank FX Drafts Settlement Reports for April 9 and 10 have been created by Convera with the missing transactions. Organizations will be debited for their FX Drafts transactions for April 9 and 10, today (Monday, April 14). Central 1 - Support@central1.com - 1.888.889.7878, press 1

investigating

Please be advised that the FX Drafts Settlement Reports for April 9 and 10 are currently blank. This issue is affecting all credit unions, as Central 1 has not received the required amount files from our settlement partner, Convera. As a result, credit unions were not debited on April 9, and the same issue is expected to occur on April 10 due to no file being received today either. An incident has been escalated to Convera for urgent investigation. At this time, no workaround is available, as the necessary data has not been provided for manual processing. We will provide the next update by Monday, April 14 at 8:00 a.m. PT (11:00 a.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "INC197972: MDB and CAS intermittent login issues"

Last update
postmortem

**Postmortem: INC197972 - Commercial Banking/Current Account System \(CAS\) Disruption – Mach 24** On Monday morning, March 24th credit union business members started to report errors when trying to log into MemberDirect Business \(MDB\). The point of failure was found during triage where two out of three MDB servers did not have a healthy database connection causing login verification to fail. The servers were restarted and they immediately connected to the databases resolving the incident by 10:45 a.m. PT \(1:45 p.m. ET\). Point of failure: During the Sunday, March 23rd, maintenance window of 1 to 3 a.m. PT \(4 to 6 a.m. ET\) here was routine patching to the MDB servers and security patching of the database server. Both patches resulted in restarts of the MDB servers and the patched SQL servers. Two of the three MDB servers restarted before the SQL server was available. The condition prevented the two MDB servers from initializing properly. Concurrent patching has been successful previously without incident, however, the precise timing issue did not happen and therefore the risk of incident was not exposed. Changes are being introduced into the patching runbooks to prevent a recurrence. MDB has several monitoring tools in place, and while alerts were generated, the issue was not immediately actionable due to the specific nature of the failed database connections. Site24x7 triggered alerts on failed login attempts, and while Zenoss confirmed server and network availability, it does not detect failed database connection states. Dynatrace, which offers deeper application-layer visibility, has since been activated for MDB and will now detect and alert on similar conditions moving forward. **PRB011489 – Root Cause Analysis** [PTASK0010303](https://central1.service-now.com/problem_task.do?sys_id=4e528bb133e4a650bbaf35bb9d5c7bb2&sysparm_record_target=problem_task&sysparm_record_row=1&sysparm_record_rows=3&sysparm_record_list=problem%3Df162eb7833286a10bbaf35bb9d5c7ba7%5EORDERBYnumber) - Application DB startup script – Due May \(to validate a solution, not implement\) * Review possible improvements for DB starts up – ex: after the application and a viable connection is not made it will kill the DB connection and reestablish a new one. [PTASK0010304](https://central1.service-now.com/problem_task.do?sys_id=bc53cf7533e4a650bbaf35bb9d5c7b9b&sysparm_record_target=problem_task&sysparm_record_row=2&sysparm_record_rows=3&sysparm_record_list=problem%3Df162eb7833286a10bbaf35bb9d5c7ba7%5EORDERBYnumber) - NOC escalation of MDB error - DONE * Add MDB DB errors to the NOC call list [PTASK0010305](https://central1.service-now.com/problem_task.do?sys_id=4684c37d33e4a650bbaf35bb9d5c7b2f&sysparm_record_target=problem_task&sysparm_record_row=3&sysparm_record_rows=3&sysparm_record_list=problem%3Df162eb7833286a10bbaf35bb9d5c7ba7%5EORDERBYnumber) - Manual Review of MS Patching and SQL patching * Will manually review DB connectivity after the next patching window \(April 2025\) or ensure that new patching staggering updates are working We recognize the impact this disruption had on our members and their business operations. We are taking this incident seriously and implementing targeted improvements to enhance system resilience and accelerate detection and recovery in the future. If you have any questions about this postmortem, please contact [digitalbanking\_support@central1.com](mailto:digitalbanking_support@central1.com).

resolved

The intermittent incident has been resolved following an emergency server restart on two of our devices. If you encounter any further issues, please don't hesitate to contact Support. Client Support Services T 888 889 7878 Option 1 support@central1.com

investigating

Currently credit union employee’s and members may experience intermittent errors when trying to log into Forge/MemberDirect Commercial or CAS Online accounts. Trying to login multiple times has worked as the error is intermittent. Our teams are actively working to resolve the situation as quickly as possible. Thank you for your patience and understanding. Client Support Services T 888 889 7878 Option 1 support@central1.com

Report: "INC198955 - Interac e-Transfer OAS system unavailable"

Last update
resolved

Central 1 has confirmed that OAS is performing optimally once again. The OAS application issue is now resolved. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

Central 1 is aware that users are not able to login to OAS production. We have reached out to Interac support to investigate and will provide updates as we receive them. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "INC198834 - RESOLVED - Clearing Transaction Posting and Report Issue"

Last update
resolved

All outstanding settlement amounts have now been posted to Central 1 accounts with an effective date of April 7. These will be reflected in tomorrow's banking reports and PREC/OREC files. We apologize for any inconvenience caused by this incident and thank you for your patience. Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

The Central 1 team is developing a fix for this morning’s clearing account posting issue for codes QC, BC, and RDC transactions. The transactions were also not reported on the daily banking reports (AIBC, AIOC, DAIL, ONDL, OREC, and the PREC) for April 8, 2025. We will provide an update on this incident at 2 p.m. PT (5 p.m. ET) or sooner if this incident is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

The Central 1 team has found the root cause of this morning’s clearing account posting issue for QC – Clearing Debit, BC – Branch Capture Deposit Credit, and the RDC – Remote Deposit Capture transactions. The transactions were also not reported on the daily banking reports (AIBC, AIOC, DAIL, ONDL, OREC, and the PREC) for April 8, 2025. We will provide an update on this incident at 12:30 p.m. PT (3:30 p.m. ET) or sooner if this incident is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

The Central 1 team is actively investigating a CAD account posting issue for QC – Clearing Debit, BC – Branch Capture Deposit credit and RDC – Remote Deposit Capture. The transactions have not been applied to your current account and are missing on the daily banking reports (AIBC, AIOC, DAIL, ONDL, OREC, and the PREC) for April 8, 2025. We are thoroughly analyzing the systems involved and working toward resolving the issue. We will provide an update on this incident at 10 a.m. PT (1 p.m. ET) or sooner if this incident is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "Alert! - INC198521 - Rollback of Digital Banking Core Release 740"

Last update
resolved

Please be advised that Digital Banking Core Release 740 has been rolled back due to a technical issue found post release. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "RESOLVED - INC198370 CRA Rejections Due to Missing Forms - March 31, 2025"

Last update
resolved

Due to month end volumes, some Corporate CRA transactions made on March 31 were rejected due to Missing Form. The affected transactions can be found on your March 31 BPRT report that was delivered this morning. Please contact your clients to have them make the payment again, today. We have resolved the issue and will be monitoring the system today. We apologize for any inconvenience caused by this incident. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "INC198028 - Forge Public Website Forms Error"

Last update
resolved

The workaround has resolved the issue. Forms on the public website are now being delivered. If you encounter any further issues, please don't hesitate to contact Support. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

Please be advised that the technical team has found the root cause of the issue and is attempting a workaround. We will provide an update by or before 5:00pm PT (8:00pm ET). Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

Technical teams continue to investigate. Next update at 3:00pm PT (6:00pm ET). Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

The technical teams continue to investigate the incident. We will provide an update by or before 9:30am PT (12:30pm ET) March 26th. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

Please be advised that we are currently experiencing technical issues with Public Website Form submissions. At this time members submitting forms on the Public Website may see "An unexpected error occurred, we apologize for inconvenience. Please try again later". Technical teams are investigating. We will provide an update by or before 5:00pm PT (8:00pm ET). Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "Resolved - INC197784 - Cannot Add Additional Email Addresses in PS AFT Originator Information Screen"

Last update
resolved

The incident has been resolved. Users are now able to successfully add additional email addresses to the Originator Information screen in PS AFT. We apologize for the inconvenience the recent disruption may have caused. Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

Central 1 is aware that clients are unable to add additional email addresses to the Originator Information screen in PS AFT. The incident has been identified, and our teams are fully engaged in resolving the issue. We understand the impact this may have on your experience and are working diligently to rectify the situation. Thank you for your understanding and support during this incident. We will provide the next update by Monday, March 24, at 9 am PT (12 pm ET) Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "Resolved - INC197204 - QA eCommSwitch Unavailable"

Last update
resolved

The Central 1 QA eCommSwitch used for bill payment testing is now available. Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

The bill payment service has been restored for eCommSwitch QA testing. Our Technical Team continues to work on fixing the CRA functionality in QA. An update will be provided tomorrow morning at 6 a.m. PT (9 a.m. ET.). Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

The Central 1 QA eCommSwitch used for bill payment testing is currently not available. An update will be provided at 12 p.m. PT (3 p.m. ET.) or sooner if the QA site becomes available. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "Resolved - INC197186 - Banking Reports and the OREC\PREC files Delayed"

Last update
resolved

The AIBC (AIOC), AIBU (AIOU), DAIL (CDJ) banking reports, and PREC and OREC files are now available. Central 1 - Support@central1.com - 1.888.889.7878, press 1

identified

The AIBC (AIOC), AIBU (AIOU), and DAIL (CDJ) banking reports, along with the PREC & OREC files, are not available this morning. All clearing reports and files are in your organization’s FTP folder. Central 1’s technical Team is investigating. We will provide an update on this incident at 7:30 a.m. PT (10:30 a.m. ET) or sooner if this incident is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1

Report: "Resolved - INC197108: ECCX reports not available for March 11th run"

Last update
resolved

The ECCX (Failed Transactions) report is included in this morning's MERG0312P.DAT file and FTP folders (ON). Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878

identified

Central 1 continues to work on a fix to ensure the ECCX report is available in tomorrow’s file and report delivery. We will provide an update tomorrow morning by 7 a.m. PT (10 a.m. ET).

identified

Please be advised that due to a potential software error related to daylight savings, the script for last night's ECCX 'e-Transfers Failed Transactions Report' did not fully complete. As a result, today's ECCX reports are currently empty. Our team is actively working on identifying a fix, and we aim to include today's ECCX report in tomorrow's report. All other reports were successfully completed and submitted as scheduled. We will provide our next update today by or before 2 p.m. PT (5 p.m. ET). Central 1 Client Support Services T 888 889 7878 Option 1

Report: "INC194858: Saturday Feb 1st e-Transfer 3.5 API service interruption"

Last update
postmortem

**Postmortem: INC194733 - API e-Transfer 3.5 outage on Saturday, February 1 | P2** On Saturday, February 1st Central 1 experienced an e-Transfer 3.5 outage \(ISO20022 send and receive\) linked to an API volume threshold that was exceeded. Impact of 40 minutes for clients using ISO20022 for e-Transfers. Service was restored when ID Mapper Limits were removed as a temporary measure until new limits were established.  **Point of failure**: ID Mapper API exceeded 1 million set limits. It has been sitting at 1 million for at least the last three years without previously exceeding the limit. The team is able to see that the limit has been continuously growing with normal operational growth however, they believe that the recent onboarding of a new client and month end activity caused us to reach that limit.  IDMapper API did not have monitoring on it’s service delaying the detection of the point of failure. **Actions**: PRB011471 – Root Cause Analysis * PTASK0010287 - IDMapper API limit on AWS – Due April 2025 * Product Support Team to work with Product team to determine and establish a new IDMapper limit.  * PTASK0010288 - API usage for API gateway – Due April 2025 * Building out a monitoring tool to allow the reporting and threshold monitoring on ID Mapper API usage moving forward.  This will be used to create the same monitoring capability across all API gateways. * PTASK0010290 - NOC escalation and Re-training – Due April 2025 * PTASK0010289 - Manually Monitor ID Mapper API usage during Month end - COMPLETED * A manual process will be out in place to monitor the IDMapper usage by the PS Software Support Team until the until the API usage monitoring and alerting is put into place.  We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all of our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements.  If you have any questions about this postmortem, please contact [digitalbanking\_support@central1.com](mailto:digitalbanking_support@central1.com). **Jason Seale |** Director of Client Support Services | [jseale@central1.com](mailto:jseale@central1.com) | 778.558.5627

resolved

Please be informed that Central 1 experienced a service disruption with our e-Transfer 3.5 API services on Saturday, February 1, from 11:23 a.m. to 12:03 p.m. PT (2:23 to 4:03 p.m. ET). The service was restarted to clear sessions which resolved the issue. Please note, 3.4 API e-Transfer and all other e-Transfers were not impacted. Central 1 will provide a postmortem within the next two weeks. Client Support Services T 888 889 7878 Option 2 - DigitalBanking_Support@central1.com

Report: "INC197038: Interac Etransfer 3.4 Autodeposits Service Delay"

Last update
resolved

The 3.4 autodeposit queue has been successfully cleared by Interac, and autodeposits that were in "Processing Autodeposit" status in OAS are now "Completed" and the incident is now resolved. Should you encounter any further issues or have concerns, please don't hesitate to contact Support for assistance. Central 1 Client Support Services T 888 889 7878

identified

Please be informed that Interac is currently experiencing a service disruption affecting incoming 3.4 autodeposits. While there are no errors at this time, these autodeposits remain stuck in the "Processing Autodeposit" status in OAS. All other functionalities, including 3.5 autodeposits, sending, registrations, contact management, and receiving with Q&A, continue to operate normally. This incident has been escalated, and all relevant teams are actively working to resolve the issue and process the affected autodeposits promptly. We will provide our next update by or befor 12 p.m. PT (3 p.m. ET). Central 1 Client Support Services T 888 889 7878 Option 1

Report: "INC194469 - Azure Data Center and OIDC logins Service Interruption"

Last update
postmortem

**Postmortem: INC194469 - Azure Data Center and OIDC logins Service Interruption | P2** On Tuesday, January 28th, Digital Banking experienced a network outage in our Azure data center causing a 5-minute online banking service outage for eastern clients \(clients connecting to our Toronto Data Center \(TOHC\). Online Banking services were immediately failed over to the Vancouver data center \(VAHC\) but clients using our OpenID Connect \(OIDC\) service \(replacing OAuth\) were unable to log in between 6:45 to 8:49 p.m. PT \(2-hour 4-minute outage\) until Central 1 manually failed OIDC to VAHC. Additionally, Atlantic credit unions utilizing our digital APIs \(instead if ISO\) had logins failing between 6:45 to 9:30 p.m. PT \(2-hour 45-minute outage\) until Central 1 manually failed services over to VAHC. **Point of failure**: the initial cause of failure was a faulty power bar in VAHC. The faulty power bar partially failed, and power was flapping on an off extending the timeline to detect root cause. The initial failure exposed a secondary issue, as specific services did not automatically failover as expected \(OIDC and Digital APIs via FUSE\). To restore services power was moved to another power source and failover configurations were restored.  A list of remediation activities will be tracked as part of the action items for this issue. **Actions**: PRB011469 – Root Cause Analysis * PTASK0010282 - Faulty power bar replacement - COMPLETED * PTASK0010283 - Proactive power bar replacement – DUE April 2025 * PTASK0010284 - Monitoring and Alerting Gap Analysis – DUE April 2025 We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements.  If you have any questions about this postmortem, please contact [digitalbanking\_support@central1.com](mailto:digitalbanking_support@central1.com). **Jason Seale** Director of Client Support Services [jseale@central1.com](mailto:jseale@central1.com) | 778.558.5627

resolved

Services have remained stable since loadbalancing back across regions yesterday, January 29th. A postmortem will be completed within the next two weeks.

monitoring

Our Digital Banking services have fully stabilized following earlier authentication disruptions. While most clients experienced no outage due to our automated recovery in the Vancouver data center, some clients using OIDC authentication may have encountered login issues for up to two hours before services were restored. We have now fully transitioned OIDC authentication to our Vancouver UCP server and continue to monitor system performance. Our teams are actively working to resolve the network issue in our Eastern data center. We will only restore the connection once our changes have been thoroughly tested to ensure stability. Our next update will be provided by 9:30 a.m. PT (12:30 p.m. ET).

monitoring

Please note, that we have moved the OIDC authentication services to our Vancouver UCP server and logins using OIDC are now working. We continue to investigate the issue and monitoring the traffic. We will provide an update by or before 9:30 AM PT (12:30 PM ET). Central 1 Client Support Services T 888 889 7878 Option 2 / DigitalBanking_Support@central1.com

investigating

Digital Banking is currently experiencing a network outage in our Azure data center and have failed Online Banking services over to the Vancouver data center. But, this issue is currently affecting logins that use OIDC authentication. Our Central 1 teams are actively working on resolving the issue. We will provide an update by 9:30 PM PT (12:30 AM ET). Central 1 Client Support Services T 888 889 7878 Option 2 / DigitalBanking_Support@central1.com

Report: "INC195332: Intermittent “Missing form” CRA payments"

Last update
resolved

Central 1 is resolving this incident as CRA services have remained stabled after our network updates. Thank you for your understanding and patience.

monitoring

The CRA form latency incident has been resolved after Central 1 applied a network update successfully to our Vancouver Data Centre last night and our Toronto Data Centre this afternoon. We will continue to monitor the service for stability and provide a final update by Friday, February 28, at 3 p.m. PT (6 p.m. ET).

identified

Central 1 technical team will be performing an emergency network update tonight to our Vancouver Data Centre at 1 a.m. PT (4 a.m. ET). This update has already been successfully applied to our QA environment, resulting in performance improvements. If this resolves the CRA form timeout issue we will be applying the fix to the Toronto Data Centre on the following evening. We will provide an update by or before Thursday, February 27, at 3 p.m. PT (6 p.m. ET). Thank you for your understanding and patience.

identified

Central 1 continues to investigate the ongoing CRA form timeouts. Our teams are currently testing proposed fixes in QA to determine if it can resolve the issues. Should this update prove successful in our QA environment, we will proceed with applying the same update to our production UCP servers by Wednesday this week. We will provide an update by or before Wednesday, February 26, at 3 p.m. PT (6 p.m. ET). Thank you for your understanding and patience.

identified

Our teams continue to investigate and are also adding additional logging to isolate the point of failure over the weekend. We will provide an update no later than Monday, February 24, at 3 p.m. PT (6 p.m. ET). Thank you for your understanding and patience.

investigating

Central 1 is continuing our investigation for the 1-2% of CRA bill payment timeouts. Please continue to use your biller reports to follow up with your members and have them resubmit their payments. We will provide an update by or before Friday, February 21, at 3 p.m. PT (6 p.m. ET).

investigating

Central 1’s technical teams are actively investigating and working to identify the root cause of the timeouts. This is impacting approximately 1-2% of CRA bill payments, please kindly keep using your biller reports to follow up with your members and have them resubmit their payments. We will provide an update by or before tomorrow, February 19, at 2 p.m. PT (5 p.m. ET).

investigating

We are currently experiencing intermittent timeouts with CRA Business Tax Forms, leading to next-day bill payment rejections. This issue is affecting MemberDirect, Forge, and API-only services. Our teams are actively investigating the matter. Action Required: • If you encounter a 'Missing Form' bill payment rejection on your daily "Bill Payment Rejected Transactions" report, please take the following steps: 1. Contact your customer or member. 2. Manually credit back the rejected amount to the customer's account for settlement. 3. Have your customer attempt the payment again. We appreciate your understanding and patience as we work to resolve this issue. An update will be provided by or before Wednesday February19th. Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878 Option 2

Report: "Resolved - INC196042 - OLT Application Unavailable"

Last update
resolved

The OLT application is now available. Please get in touch with Client Support if you encounter any further issues with OLT. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

identified

The bill payment Online Tracing (OLT) application is currently unavailable, users are encountering the error message “Service Temporarily Unavailable” when attempting to access the application. Central 1’s technical staff is investigating. An update will be sent out at 7:30 a.m. PT (10:30 am ET), or sooner if OLT becomes available. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

Report: "Resolved -INC195760 - Wires AML Compliance Delay"

Last update
resolved

Central 1’s vendor has restored the automated processing for reviewing outgoing and incoming domestic/foreign wires for anti-money laundering (AML) compliance. Wires will be processed by today’s wires system deadline. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

identified

Central 1’s vendor used for reviewing all outgoing and incoming domestic/foreign wires for anti-money laundering (AML) compliance continues to have system issues preventing the automated review of the wires. Central 1’s Teams are working diligently to manually review and release the wires for processing by today’s 2:15 p.m. PT (5:15 p.m. ET) cut off time. If you have an urgent wire with a pending status, please contact Central 1 to get the wire released. An update will be sent out at 1 p.m. PT (4 p.m. ET), or sooner if vendor service is restored. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

identified

Central 1’s vendor used for reviewing all outgoing and incoming domestic/foreign wires for anti-money laundering (AML) compliance has an incident today preventing the automated review of the wires. Central 1 has immediately began our disaster recovery plan for the manual review and release of the wires. If you have an urgent wire with a pending status, please contact the wire desk to get the wire released. An update will be sent out at 11 a.m. PT (2 p.m. ET), or sooner if vendor service is restored. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

Report: "Resolved - INC194303 - OLT Application Unavailable"

Last update
resolved

The OLT application is now available. Please contact Client Support if you encounter any further issues with OLT. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

identified

The bill payment Online Tracing (OLT) application is currently unavailable, users are encountering the error message “Service Temporarily Unavailable” when attempting to access the application. Central 1’s technical staff is investigating. An update will be sent out at 8 a.m. PT (11:00 am ET), or sooner if OLT becomes available Central 1 - Support@central1.com - 1.888.889.7878, press 1.

Report: "INC194054 - Servus Mastercard Payments Delayed - January 15 to 20, 2025"

Last update
resolved

Servus Mastercard has advised any payments made between January 15 to January 20 were delayed due to their system issues. All delayed payments have now been applied and backdated to the original date of payment. Central 1 - Support@central1.com - 1.888.889.7878, press 1

monitoring

Central 1 has been advised by Servus Mastercard that, due to technical issues, all payments made on January 16 have not yet been applied. They are actively investigating the matter. We will provide an update once Servus Mastercard has more information and/or has resolved their issues. Central 1 – Support@central1.com – 1.888.889.7878, press 1

Report: "INC193602 – Interac Alert: 3.5 e-Transfer degradation when updating contact information"

Last update
resolved

Interac has advised the incident is resolved as of 3:13pm PT / 6:13pm ET.

investigating

We are continuing to investigate this issue.

investigating

Interac has informed Central 1 that they are observing a degradation on their e-Transfer platform for all 3.5 Participants. An intermittent degradation was being observed on the e-Transfer platform for all 3.5 Participants. Customers using 3.5 may experience intermittent issues while trying to update contact information. Technical teams are actively investigating. Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878

Report: "INC193346 - Collabria Mastercard and Visa Payments Delayed - January 3 to 6"

Last update
resolved

Collabria has resolved their issue. All payments have been applied to the applicable cardholder accounts effective the original payment dates. Central 1 - Support@central1.com - 1.888.889.7878, press 1

monitoring

We have been notified by Collabria that the payments between January 3 and 6 are scheduled to process overnight tonight and will be applied to the accounts by tomorrow. Each payment will be posted to the accounts with an effective date of the original payment date. We will provide a final communication tomorrow. Central 1 – Support@central1.com – 1.888.889.7878, press 1

monitoring

Central 1 has been advise by Collabria that, due to issues with their internal system, all payments made to their Mastercard and Visa accounts between January 3 and January 6 have not yet been applied. Collabria is actively investigating the matter. Once their system issues have been resolved, all delayed payments will be applied to accounts effective the original payment dates. We will provide an update once Collabria has resolved their issues. Central 1 – Support@central1.com – 1.888.889.7878, press 1

Report: "INC191797 - Intermittent Login Issues on Digital Banking"

Last update
postmortem

**Postmortem: INC191797 - Intermittent login issues for 25 minutes | P2** On Sunday, December 8th, digital banking experience login degradation between 5:00 to 5:15 p.m. PT \(8:00 to 8:15 p.m. ET\) and then a second degradation from 5:22 to 5:32 \(8:22 to 8:32 p.m. ET\); total login outage/degradation of 25 minutes. Services recovered on their own resolving the incident. During the impact window member's may have encountered a "HOST Timed Out" error. Biometric login and credit unions on OIDC service \(OAuth replacement\) were not affected by this incident. **Point of failure**: A published AWS maintenance notification for a firewall device reboot was missed by Central 1. These types of notifications do not require any intervention because of redundancy however, the network equipment is a higher risk and should be monitored.  The C1 team will look at expanding notification and automating AWS Health Dashboard to filter out standard notices and notify on notifications that meet specific criteria so that important notices are not missed.  **Actions:** **PRB011459 – Root Cause Analysis** PTASK0010273 - Expand Distribution on AWS email Notifications - COMPLETED * Add checks and expand users to the AWS email notifications so that there are other team members aware of the notifications. * Reviewing emails notifications from AWS until automation can be put into place to filter out standard type notifications. PTASK0010276 - Firewall Monitoring Improvements - COMPLETED * Add alerting to all firewalls to ensure we get alerts to immediately identify when any firewall device goes down or becomes unavailable. PTASK0010274 - Automation and additional alerting of AWS Heath Check Dashboard Notifications – DUE Q1 2025 * Automate query to filter on specific type notifications that should issue alerts to ensure that the team reviews those alerts and do not treat them as standard routine notifications. PTASK0010275 - UCS Recovery Improvement – DUE Q1 2025 * Increase UCS heartbeat frequency so that UCS recovers faster in the event of any unplanned outages.  We sincerely apologize for the inconvenience caused by this login service degradation. We are committed to preventing future occurrences and ensuring the reliability of our services.  If you have any questions about this postmortem, please contact [digitalbanking\_support@central1.com](mailto:digitalbanking_support@central1.com).  Jason Seale, Director of Client Support Services - [jseale@central1.com](mailto:jseale@central1.com) | 778.558.5627

resolved

Central 1 is aware that OAuth Login has experienced a service degradation. Member's may have encountered a "HOST Timed Out" error between 5:00-5:32 p.m. PT (8:00-8:32 p.m. ET). Please note that biometric login is unaffected on mobile app, and clients that are not using our 2SV/OAuth service are unaffected by this incident. Services have remained stable and Central 1 is conducting a root cause analysis which will be shared in our postmortem. Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878

Report: "INC191106 - RESOLVED: Interac 3.4 service outage"

Last update
postmortem

**Postmortem: INC191106 - e-Transfer 3.4 outage | P2** On Wednesday, November 27th Central 1 experienced an e-Transfer 3.4 service outage between 2:45-3:45 a.m. PT \(5:45 to 6:45 a.m. ET\) – approximately 1 hour. During the outage users would have seen an error 910 or 914 in MemberDirect/Forge digital banking, or error 504 for API calls while sending, receiving, or accessing the Interac e-Transfer feature. Central 1 performed a server restart to restore service. Point of failure: The incident was triggered after a routine VMware patching change that took place. After VMware patching activity was completed successfully, monitoring alerts indicated one of our ActiveMQ servers went into an unhealthy state. The investigation revealed that the impacted server had not been restarted in over 500 days. Central 1 cannot say conclusively that this was the cause, but it may have been a contributing factor for the server going into this unhealthy state. Our high availability server failover did not occur automatically because the server was still available, however, in an unhealthy state that could not handle traffic. The fail-over process was never completed a full failover but rather the broker continued to re-attempt the connection towards the unhealthy server after the initial failover to an available server .  The team cited areas of improvements for High availability \(HA\) for these servers and added it to our actions below. **Actions:** [PRB011454](https://central1.service-now.com/problem.do?sys_id=1d9ad3be938ade1007b9bf027cba1008&sysparm_record_target=problem&sysparm_record_row=1&sysparm_record_rows=1&sysparm_record_list=parent%3D598d9ef693025e10faefbd4e1dba100f%5EORDERBYDESCsys_updated_on) – Root cause analysis * PTask - Scheduled ActiveMQ server restarts - COMPLETED * PTask - Review the HA logic - design improvements identified for ActiveMQ server High Availability \(2025 intiative\) We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements.  If you have any questions about this postmortem, please contact [digitalbanking\_support@central1.com](mailto:digitalbanking_support@central1.com).  Jason Seale | Director of Client Support Services | [jseale@central1.com](mailto:jseale@central1.com) | 778.558.5627

resolved

On Wednesday, Nov 27th, Central 1 experienced an Interac 3.4 e-Transfer service outage from 2:45 to 3:45 a.m. PT (5:45 to 6:45 a.m. ET). During the outage users would have seen an error 910 or 914 in MemberDirect/Forge digital banking, or error 504 for API calls while sending, receiving, or accessing the Interac e-Transfer feature. Our teams did an emergency server restart (ActiveMQ03) which restored service and assisted resolving the incident. Please contact Support if you experience any further issues. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "RESOLVED: INC192100 – Interac Alert: Advising of e-Transfer degradation being observed"

Last update
resolved

The incident has been resolved. Central 1 did not note any material impact during the reported degradation. Interac advised: “An intermittent degradation was observed on the e-Transfer platform for all Participants. Response times have stabilized. Participants might have experienced a minor delay in receiving and completing e-Transfers.” Degradation was observed today between 7:34 a.m. to 12:30 p.m. PT (10:34 a.m. to 3:30 p.m. ET).

monitoring

Interac has informed Central 1 that they are observing a degradation on their e-Transfer platform for all Participants “An intermittent degradation was being observed on the e-Transfer platform for all Participants. An improvement in response time has been observed and is anticipated to be back to normal operational limits in the next few hours. Technical teams are continuing to monitor.” Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878

Report: "INC190936 - EFM Login interruption"

Last update
resolved

The incident has been resolved by our vendor. Regular access to EFM has been restored.

investigating

We are currently experiencing a misconfiguration in the Enterprise Fraud Management (EFM) login workflow for Credit Union staff. When accessing the Featurespace site and clicking “Sign In”, the next page URL may include the word “authqa”. To resolve this, please manually remove “qa” from the URL. This adjustment should allow you to log in as usual. Thank you for your understanding. Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878

Report: "RESOLVED: INC192070 – Interac Alert: Advising of a minor degradation is being observed on the e-Transfer platform"

Last update
resolved

Interac confirmed their e-Transfer incident resolved. “An outage was observed on the e-Transfer platform for all Participants. Participants may have experienced time-outs connecting to e-Transfer service during impacted period. This outage was traced back to a maintenance activity and no further impact is anticipated.” The outage window was determined to be approximately 5:35 to 5:38 p.m. PT (8:35 to 8:38 p.m. ET).

investigating

Interac has informed Central 1 that they are observing a minor degradation on their e-Transfer platform for all Participants. Participants may experience time-outs as API calls are taking longer than expected to execute. Their technical teams are engaged. Central 1 will provide updates as they become available. Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878

Report: "Resolved - INC191388 - FX Drafts (Convera) - Delay in Processing of Drafts"

Last update
resolved

The FX Drafts processing issue has been resolved. If you continue to experience any issues, please contact Central 1 Support. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

investigating

Convera continues to investigate the slowness experienced by users when processing drafts. An update will be provided at 12:15 p.m. PT (3:15 p.m. ET) or sooner if the issue is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1

investigating

Convera has advised Central 1 that credit unions may be encountering issues when processing drafts. Some drafts are stuck in the processing status for 30-60 minutes after approval Convera’s technical team is currently investigating the issue and are actively working to resolve the issue. We will provide another update tomorrow by 7am PT (10am ET), or sooner if the incident is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

Report: "INC191258 - RESOLVED - Unable to Access Pending Requests in User Management (UM)"

Last update
resolved

The redirect of the User Management (UM) application has been completed successfully. The option of Manage Requests is now functioning. Please note that previously communicated enhancements are once again available in UM. Some users may need to clear their web browser cache to ensure you are getting to the current version of the UM. Thank you for your patience during this incident.

identified

The scheduled restart of the User Management application at 10 a.m. PT (1 p.m. ET) did not resolve the issue with accessing pending requests. At 10:45 a.m. PT (1:45 p.m. ET) we will be redirecting the application to a corrected server that includes the enhancements that were originally deployed on November 26 and subsequently rolled back. The incidents that triggered the rollback yesterday have been resolved. We will provide another update by 11:30 a.m. PT (2:30 p.m. ET). Thank you for your patience while we work on resolving this incident.

identified

Central 1 will be restarting our User Management (UM) application at 10 a.m. PT (1 p.m. ET) which will cause a 5-minute service outage. Please use CPS to complete any user administration activities during this time.

investigating

Central 1 is aware that pending requests in User Management (UM) are not accessible and have initiated an immediate investigation. Our experts are gathering data to determine the underlying cause. We will provide an update by 10 am PT (1 pm ET) or sooner if the incident is resolved by then. Central 1 - Support@central1.com - 1.888.889.7878

Report: "INC191216 - New Features and Enhancements in User Management (UM) Rollback (CHG153842)"

Last update
resolved

Due to various incidents with the User Management (UM) release on Tuesday, November 26, we have rolled back the release. Some users may need to clear their browser's cache to ensure you are reaching the current version of UM. We will try this release again after the Technical Team has had time to investigate and resolve. Once a date has been determined a notice will be posted. We apologize for any inconvenience caused. Central 1 - Support@central1.com - 1.888.889.7878

Report: "INC191125 - Intermittent Inter-Member transfers issue"

Last update
resolved

The incident has been resolved with deployment of the Urgent release. Please contact Support if you experience any further issues. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

identified

Central 1 has rolled back the ticket causing the incident in the lower environments and testing has been completed. The release has been scheduled for Production tonight, Thursday November 28, 2024 in Urgent Digital Banking Core Release 732. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

We are currently experiencing an intermittent issue with the inter-member transfer (Option 2) on our Digital Banking Retail and Small Business Desktop platform. When a user initiating a transfer to another member or customer, they will need to enter the recipient’s account number and branch number (if applicable) in the designated fields. Some users may encounter an issue at the final confirmation stage causing a blank screen. Please note that this issue does not affect the Mobile app or e-Transfers. As a work around, your users can use the Mobile App to complete the inter-member transfer or send an e-Transfer to the other member. We are actively investigating the issue and working to resolve it as quickly as possible. An update will be provided tomorrow Thursday Nov 28th, by 1 p.m. PT (4 p.m. ET) or sooner. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "INC191000 - Teamsite 20 Intermittent timeouts and slowness"

Last update
resolved

The incident has been resolved. Teamsite has recovered.

investigating

Our vendor is restarting Teamsite. This will cause a 2 minute full outage of the authoring/editing interface. This will have no impact on live sites (your customers or members will not be impacted).

investigating

Please be advised that we are currently experiencing a technical disruption with the Teamsite 20 CCpro and eStudio services. Your website content editors may notice that the system is extremely slow or show intermittent timeouts error 503. This incident does not affect online banking. OpenText and Central1 are actively investigating this issue and working to resolve it as our top priority. An update will be provided by 3p.m. PT (6 p.m. ET) or sooner. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "INC189931: Intermittent Inter-Member transfers issue"

Last update
resolved

Please note that the fix for this incident was successfully deployed last night with Digital Banking Core Release 730. Please let Support know if you have any further issues with the Online Banking Transfers feature. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

identified

Central 1 has rolled back the ticket causing the incident in the lower environments and is currently testing. The release is currently scheduled to go to Production on Thursday November 14, 2024 in Digital Banking Core Release 730. We will provide an update on or before November 14 if there are any changes. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

identified

Please be advised that this issue is also impacting regular transfers when the receiving account description contains a special character. For example, if the account description is “Alex’s Savings” the apostrophe will cause an error for users attempting the transfer. As a workaround, kindly ask your staff to assist users by removing the special character in the account description either through the banking system or, if available, through the “Rename Account” option in online banking. To resolve these issues, Central 1 teams are working to roll back the ticket causing the issue (MRR-34440). We are currently preparing an urgent release to resolve the issues. The roll back is currently in testing, and once complete we will be able to to share a release date. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

We are currently experiencing an intermittent issue with the inter-member transfer (Option 2) on our Digital Banking Retail and Small Business Desktop platform. When a user initiating a transfer to another member or customer, they will need to enter the recipient’s account number and branch number (if applicable) in the designated fields. Some users may encounter an issue at the final confirmation stage causing a blank screen. Please note that this issue does not affect the Mobile app or e-Transfers. As a work around, your users can use the Mobile App to complete the inter-member transfer or send an e-Transfer to the other member. We are actively investigating the issue and working to resolve it as quickly as possible. An update will be provided tomorrow Thursday Nov 7th, by 1 p.m. PT (4 p.m. ET) or sooner. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "INC189622 - Interac e-Transfer Service Degradation"

Last update
postmortem

**Postmortem: INC189622 - Interac e-Transfer Service Degradation** Please see Interac’s postmortem update for the November 1 service degradation. ------------------------------------------------------------------- **1 – Incident Summary** On Friday November 1st, 2024, the Interac® e-Transfer Service hit a record number of total transactions processed in a day; however, the high volume of requests resulted in a performance degradation of the service resulting in some failed Interac e-Transfer transactions and some email and SMS notification delays for our Participant’s customers. The duration of the incident lasted approximately 12 hours and 50 minutes. **2 – Incident Date** Start Date & Time: Friday 11/01/2024 – 09:40 ET End Date & Time: Friday 11/01/2024 – 22:30 ET **3 – Business Impact** Participants may have experienced failed transactions and delayed notifications: * Transactions Impact: On November 1st, 2024, while 7,119,931 transactions were processed successfully, 77,787 \(1.1%\) requests resulted in transaction failures. * Notifications Impact: During the incident, a total of 17,344,970 notifications were sent successfully. Out of the successful notification events, 14% \(2,174,848\) of events experienced a delay of over 30 minutes. **4 – Root Cause** A high volume of requests to the Interac e-Transfer Service resulted in an unexpected sustained and non-linear spike in queuing which caused the slow application response times, transaction failures and notification delays. The combination of underperforming database queries, with sub-optimal resource and connection management configurations have been identified as contributing factors to the spike in queuing and slow response times. **5 – Resolution** During the incident, Interac support teams implemented an emergency change by increasing application timeout settings to prevent transaction failures; however, sustained Interac e-Transfer load continued to delay notifications until the load volume began to subside and the backlog was processed. Out of an abundance of caution, Interac has paused changes to the Interac e-Transfer service until performance enhancements can be implemented in anticipation of the next forecasted High-Volume Day on November 29th, when 6-6.5M transactions are forecasted. The system continues to perform normally since the incident, and we continue to monitor with hyper-care vigilance. Ahead of November 29th, Interac will be implementing 9 performance improvements and will communicate change schedules as soon as available. **6 – Follow-up and Remediation** Problem investigation opened to determine root cause \(PRB0041806\) – In progress * Identify opportunities to improve Interac e-Transfer service in preparation for the next High-Volume Day \(HVD\) - Completed * Develop enhancements – Target Date - November 13th – In progress * Test enhancements – Target Date - November 14th – Not started * Apply candidate performance improvements – Target Date - November 18th – Not started

resolved

Interac confirmed that the incident was resolved last night at 8 p.m. PT (11 p.m. ET) and response times have now returned to normal. Additionally, the backlog of notifications has returned to normal. Interac technical teams will continue to further investigate the root cause and will provide a postmortem within the next few weeks.

monitoring

Interac is seeing response times improve but are still higher than expected. Notifications backlog is expected to clear in next two to three hours. Technical teams are continuing to investigate other solutions to restore the response time to normalcy. Until service degradation has been completely restored customers may still see transactions taking additional time to complete. Customers may experience a delay in sending and receiving transfers such as auto-deposits, Q&A, and money request transfers, as well as delays in email and SMS notifications. Central 1 will provide an update on Saturday, November 2 by 11 a.m. PT (2 p.m. ET).

investigating

Interac is still experiencing service degradation and is continuing to investigate with urgency. Central 1 will provide the next update by 8 p.m. PT (11 p.m. ET).

investigating

Interac has advised Central 1 that they are still observing e-Transfer impacts and are continuing to work on resolving their e-Transfer degradation. An update will be provided by 7 p.m. PT (10 p.m. ET) or sooner, should a change in status occur or additional information is identified.

investigating

Interac continues to work on resolving their e-Transfer degradation. An update will be provided by 3 p.m. PT (6 p.m. ET) or sooner, should a change in status occur or additional information is identified. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

Interac continues to work to resolve their e-Transfer degradation. We are still seeing elevated response times from Interac for all clients, affecting e-Transfer versions 3.4 and 3.5. An update will be provided by 1 p.m. PT (4 p.m. ET) or sooner. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

Interac at this time has not been able to resolve their e-Transfer degradation. We are seeing elevated response times from Interac for all clients, affecting e-Transfer versions 3.4 and 3.5. as there are some time-outs as API calls are taking longer than expected to execute. Your users may experience a delay in sending and receiving transfers such as auto-deposits, Q&A, and money request transfers. Technical teams are engaged and working to troubleshoot the issue Next update will be provided by 11 a.m. PT (2 p.m. ET) or sooner, should a change in status occur or additional information is identified. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

investigating

Central 1 has been informed by Interac that there is a intermittent service degradation on the e-Transfer platform, on both versions 3.4 and 3.5 . Interac Technical teams are actively investigating. Start time of the degradation: Friday, November 01 by 6:40 a.m. PT (9:40 a.m. ET). An update will be provided by 9 a.m. PT (12 p.m. ET) or sooner. Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2

Report: "INC190035 - Potential Canada Post Strike and Receiver General Cheques"

Last update
resolved

Please see below a notice from Public Services and Procurement Canada (PSPC) that was provided to Central 1 from Payments Canada. It provides information on the impact of Receiver General payments due to the potential Canada Post strike. Note the union needs to provide a 72-hour notice of any intent to start a labour disruption and at this time has not. They are in a legal position to strike as early as this Sunday, November 10. --------------------------------------------------------------------------------------------- In order to ensure the delivery of socio-economic payments to those in need, Canada Post have negotiated an agreement with the Canadian Union of Postal Workers for the once-a-month delivery of select Government of Canada payments during any labour disruption. In adherence with this agreement, Canada Post employees will deliver Canada Pension Plan (CPP), Old Age Security (OAS), and Canada Child Benefit (CCB) cheques to recipients. In the event of a full strike, all other Government of Canada cheques will be held for delivery until the situation is resolved. Delivery of the noted socio-economic payments would be on a monthly basis, in line with the CCB due date. This means that CPP and OAS cheques would be delivered to recipients prior to their due date; given the logistics of planning these deliveries, this will occur in November regardless of whether a labour disruption actually occurs. The Receiver General wishes to reassure our financial institution partners that any CPP and OAS payments redeemed prior to the date noted on the face of the cheque, will still be honoured. Direct deposit payments will be unaffected and deposited on the regular due date for each program.

Report: "Resolved - INC189736 - FX drafts printing issue"

Last update
resolved

There continue to be a couple of credit unions remaining that cannot print FX Drafts. Convera’s technical team continues to investigate the issue. FX Drafts can be printed using Google Chrome or Firefox. If your organization is having problems printing drafts, please contact Client Support at Central 1 so we can effectively troubleshoot the printing issue. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

monitoring

Convera continues working to resolve the FX Drafts printing issue. We will provide an update on Wednesday, November 6th, at 11 a.m. PT (2 p.m. ET) or sooner if the issue is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1.

monitoring

The FX Drafts printing issue happens when a user tries to print a draft using the Edge browser. Convera continues to work on a resolution for the printing issue. FX Drafts can be printed using Google Chrome or Firefox. We will provide an update on Tuesday, November 5th at 11 a.m. PT (2 p.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1.

monitoring

Convera has advised that they are still working on resolving the issues around the printing of drafts. We will provide another update by 1 p.m. PT (4 p.m. ET).

monitoring

We are continuing to monitor for any further issues.

monitoring

Convera's technical team continues to investigate the issue with printing Drafts when clients are using the FX Draft application. We will provide another update on Monday, November 4, 2024 by 8 a.m. PT (11 a.m. ET). Central 1 - Support@central1.com - 1.888.889.7878, press 1.

monitoring

Convera has advised Central 1 that some clients may still be experiencing issues when using FX Drafts. Their technical support continues to investigate, and they will share information with us as they have it. We will provide another update today by 4pm PT (7pm ET), or sooner if the incident is resolved.

investigating

Convera has notified Central 1 that credit unions may be encountering issues when printing Drafts using the FX Drafts application. Convera’s technical team is currently investigating the issue. An update will be sent out by 2 p.m. PT (5 p.m. ET), on Friday, November 1, 2024, or earlier if the incident is resolved. Central 1 - Support@central1.com - 1.888.889.7878, press 1.