Historical record of incidents for Sila
Report: "API 0.2.118 Release Deployment"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We will be undergoing scheduled maintenance during this time.
Report: "Maintenance Window"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We will be undergoing scheduled maintenance during this time.
Report: "Card Transacion processing is delayed in Prod"
Last updateThis incident has been resolved.
There is Card Transactions processing delay in Prod, we are currently investigating the issue.
Report: "Hotfix Deployment"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Partial Outage in the Sandbox Environment"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Evolve Service Outage"
Last updateThis incident has been resolved.
At 3:48 PM Thursday, July 18, Evolve reports that Microsoft is experiencing a global outage that impacts Evolve's Products. This outage impacts API and SFTP data interfaces, where data is received or distributed between Evolve and clients. Evolve is actively monitoring this incident and will provide updates as more information is made available.
Report: "Evolve ACH Origination Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
Evolve has identified the cause of the incident. The cause was an unprompted password change within our Core System Provider's infrastructure, which disrupted our system operations.
Evolve Service Disruption Affected Systems Components: ACH Origination | API, ACH Origination | File Transfer, ACH Receiving | File Transfer, ACH Settlement, Book Transfer | File Transfer, Card Issuance | API, Check Printing | File Transfer, Ledger | API, Online Banking, RDC | File Transfer, RPPS | File Transfer, RTP Receive | API, RTP Send | API, vAccounts | File Transfer, Wire | API, Wire | File Transfer, Wire Settlement Services: Interface - Online Banking, Processing - ACH, Processing - Book Transfer, Processing - Core Banking Functions, Processing - Wires (Domestic), Processing - Wires (International)
Report: "Evolve ACH Origination Outage"
Last updateThis incident has been resolved.
Evolve's ACH origination solution is down
Report: "KYC intermittent unavailability"
Last updateThis incident is now resolved. The API was unavailable for a total of 8 minutes between 10:56-10:58ET and 11:05-11:10ET.
Alloy has reported they are seeing intermittent latency and unavailability for their API. They have identified the source of poor performance. Please be aware this could impact performance of KYC checks.
Report: "Evolve network outage"
Last updateAs of 8:25 am June 12, Evolve has determined this incident has been resolved.
Evolve reports that as of 9:45 am, systems are operating as expected. Evolve will continue to monitor the situation and communicate updates as necessary.
As of 8:24 am PT today, June 11, Evolve is investigating a network connectivity issue that is impacting essential Evolve services. Evolve has recently discovered impacts to data extract delivery and are investigating further. Evolve is actively working on a resolution to restore normal operations as soon as possible.
Report: "Evolve Service Disruption"
Last updateUpon monitoring and review, this incident is resolved.
Evolve is continuing to work to ensure all clients are up to date in both transactions and file delivery. Evolve will continue to monitor processing and file delivery to clients.
Evolve is continuing to work through the reconciliation process from the incident date to ensure accuracy and consistency in data. Evolve will continue to monitor processing and file delivery to clients.
Evolve's connection to the data center has been re-established. Transactions and data extracts continue to process normally. Evolve will continue to monitor the incident to ensure all operations are functional.
Evolve complete maintenance over the weekend to stabilize network services at their Data Center on Core Service Provider Solutions. As of 6:35 am PT, June 3, Evolve is working diligently to get current in transaction processing and will continue to monitor processing and file delivery to clients.
Evolve continues to generate and deliver data extracts for all previously affected services to our clients including the Wire In/Out extracts. Transaction Processing: Evolve is current for all ACH and Domestic Wire transactions.
Evolve is still working to generate and deliver data extracts to all clients. Evolve will continue to monitor processing and file delivery.
As of 10:45 am PT, May 31, Evolve's End of Day processing has completed. Transaction processing has resumed. Evolve will continue to monitor processing and file delivery to clients.
Evolve’s Core System Provider is nearing completion of End of Day processing. Incoming transactions should begin processing shortly.
Additional update from Evolve - Evolve’s transaction processing is currently on hold due to an operational setback attributed to an issue with their core provider. This is due to the large transaction size of files during End of Day processing from the network outage. Root Cause Clarification: This issue stemmed from Evolve’s systems attempting to catch up from the previous day's transactions. Mitigation: Evolve is actively working with their Core System Provider to quickly finish End of Day processing.
At 7:22 am PT this morning, May 31, Evolve sent out a notification that due to the ongoing network issues affecting their data center, they needed to restart their End of Day processing. Consequently, all transactions will be temporarily paused to ensure data integrity and system stability. At 7:41 am PT, Evolve sent out another notification that they are working with their data center on fully restoring all services after overhauling the network. Evolve was current on all transactions as of approximately 4am ET. As of now, Evolve has paused the sending of data extracts for this final maintenance period. Evolve is able to process transactions but is currently unable to send out data extract files as they relate to transactions.
As of 2:17 pm central, May 30, Evolve reports that they are now operational and are working to process transactions and files that have been updated by the network outage.
At approximately 5:00 am CDT on May 29, Evolve lost internet connection to the Evolve Network. This loss of connection impacts all data extract files. As of 8:30 am CDT on May 30, Evolve continues to collaborate with their data center internet provider to address the ongoing internet network outage and expedite the restoration of all services. Evolve is working through the backlog to bring all transactions current.
Report: "Evolve ACH Processing Delay"
Last updateAs of 10:56 AM CT, processing has resumed and Evolve is not expecting any further delays. Evolve will continue to monitor processing and file delivery to clients.
As of 8:13 AM CT, Tuesday, April 30, 2024, Evolve reports ACH Wire and File processing is delayed. There is a processing issue with their Core System Provider. Evolve is investigating the cause.
Report: "ACH File Processing Evolve Outage"
Last updateAs of 5:30 CT, Evolve's internet service has been restored. Evolve will continue to monitor the incident to ensure all operations are functional.
Shortly after 3 pm Central Time today (04.23.2024) , Evolve lost internet connection from the Evolve Network. Evolve anticipates this to be a recurrence of yesterday's incident (04.22.2024). Impact: Same Day ACH originated files processed successfully within the Same Day window. Evolve is working toward a timely solution to avoid impacts to later delivery windows.
Report: "ACH File Processing Evolve Outage"
Last updateEvolve reports this incident has been resolved as of 5:30 AM PT. Evolve's internet service has been fully restored, and Evolve will continue to monitor the incident to ensure all operations are functional.
Update: Same Day ACH files did not make the same day window. As Evolve works through the network connectivity incident, Evolve aims to process the missed Same Day ACH transactions as Next Day ACH.
Shortly after 2 pm Central Time, Evolve lost internet connection from the Evolve Network. Evolve has actively been working on a resolution from the start of the incident. Impact: ACH file processing is expected to be impacted by this lapse in internet connectivity. Missed delivery windows will result in delayed ACH transaction delivery and posting. Please see https://status.getevolved.com for most recent updates.
Report: "ACH file transmittal was delayed, settlement time for some transactions will be delayed"
Last updateThis incident has been resolved.
A portion of the transactions from this event were found to have still been delayed. They are now expected to settle in the remote bank accounts later today by 18:00 ET 02/01.
The process responsible for the transmittal of ACH transactions to the ACH payment network experienced issues on the evening of 1/30/2024 through to 09:00 Pacific 1/31/2024. We have implemented a fix and are monitoring the process closely. * Same Day transactions from the first Same Day window at 05:15 Pacific on 01/31 that should have settled by 1:00pm ET will instead settle 4 hours later at 5:00pm ET. * Next Day transactions from 01/30 will be sent on 01/31, and should settle 24 hours later than expected.
Report: "We are experiencing an increase in the rate of R03 returns"
Last updateThis incident has been resolved.
Our upstream ACH provider has informed us that a fix for the underlying issue was implemented on 10/31. There is usually a two-day lag on returns, so it's expected that we would continue to see an elevated rate of R03 returns for a few days after the fix. We are now seeing that the rate of R03s for items is dropping back to the normal/historical levels. We will continue to monitor the R03 rates for another two business days, and will mark it as resolved if the R03 rate continues to match historical levels.
There has been a marked increase in the number of R03 (No bank account/unable to locate account) returns. This has been traced to our upstream ACH provider experiencing an issue with their software vendor regarding leading 0s being erroneously stripped from bank account numbers. We are constant contact with our ACH provider on this issue, and will update this incident when we are informed it has been resolved.
Report: "Standard ACH submission for November 1st may be delayed until November 2nd at 9:30 Pacific"
Last updateWe have verified with our ACH provider that all files were processed at the expected times, and that no customer transactions were impacted; all transactions were processed at the original expected times. This incident has been resolved.
Our ACH provider has informed us that this evening's ACH submissions have been accepted and forwarded on to the ACH network for processing within the normal window this evening.
We have successfully sent out a NACHA file for the day's Standard ACH transactions at 18:40 US/Pacific, but have very likely missed today's cutoff. In that case, we expect the transactions to be forwarded on to the ACH network early tomorrow morning. We will be updating this incident as we learn more.
We are currently investigating this issue.
Report: "Delay in Transactions processing"
Last updateThere is a fox has been deployed and transactions are processing normally now
There is delay in processing transactions on Prod. Team is currently investigating the issue
Report: "Maintenance to add Load balancer for Sandbox Servers"
Last updateThis incident has been resolved. All our sandbox traffic is now routing through Load Balancer, please make sure to clear cache If you see any loading issues on Sandbox
We are working on attaching Application Load Balancer for our Sandbox API servers to reduce latency in API requests
Report: "AWS email shutdown impacting services"
Last updateThis incident has been resolved and all operations performed normally.
The ability to send email from our API servers and AWS infrastructure (Lambdas, etc.) has been restored as of 19:30 PDT. We are currently monitoring all functionality that was affected by this service interruption, but are expecting no further impact.
Because we lost the ability to send Email temporarily, there will be degraded performance in team invitations, transactions for accounts with non-functional transaction webhooks, webhook email alerts (webhooks themselves are OK), sending statements, password reset, creating console logins. Working with AWS to get the ability to send emails back.
Report: "Processing delay in Transactions and KYC requests"
Last updateThis incident has been resolved, all operations on prod performed normally
There will be processing delay in prod KYC requests and transactions
Report: "Issue affecting all customers' ability to withdraw the full amount of a wallet's balance"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We have identified a bug which was introduced into production at 07-06-2023 13:30 PDT (Thursday afternoon) affecting all customers' ability to withdraw the full amount of a wallet's balance. This would result in transfer and redeem requests being refused as NSF when in fact there were sufficient funds. A fix has been deployed as of Friday morning 07-07-2023 10:45am PDT. Customers are reporting that transactions are now completing successfully as expected.
Report: "Issue affecting all customers' ability to withdraw the full amount of a wallet's balance"
Last updateWe have identified a bug which was introduced into production at 07-06-2023 13:30 PDT (Thursday afternoon) affecting all customers' ability to withdraw the full amount of a wallet's balance. This would result in transfer and redeem requests being refused as NSF when in fact there were sufficient funds. A fix has been deployed as of Friday morning 07-07-2023 10:45am PDT. Customers are reporting that transactions are now completing successfully as expected.
Report: "Upgrading Sandbox to Ubuntu 22.04"
Last updateWe are done with the upgrade and sandbox is operational now
We are working on upgrading sandbox to Ubuntu 22.04 version
Report: "Planned Maintenance"
Last updateThis incident has been resolved.
Maintenance is completed, we are back to operational
We are upgrading Instance types of prod server for better performance
Report: "Database version upgrade"
Last updateWe are done with Database upgrade. Prod API is operational now.
We are upgrading production database version upgrade. API will be unavailable until that’s done
Report: "Few Endpoints on Sandbox throwing 504s"
Last updateIncident has been resolved. We found out long running queries were the reason for timeout and killed those processes. Sandbox is operational now
After the database upgrade, we are seeing some of the endpoints throwing 504s. We are looking into the issue
Report: "Database Version Upgrade"
Last updateWe are done with upgrading database version. All operations should be performed normally now
We are upgrading our Sandbox Postgres Database version to 14.4
Report: "Developer Console Register and Login Unavailable"
Last updateWe are done with hotfix release and site is operational now
Releasing the fix that Enforce captcha checks on server-side for all requests that come from console pages that require Captcha pages
Report: "Instant ACH Delays"
Last updateThis incident has been resolved.
The issue has been resolved, we are continuing to monitor.
We are seeing delays in Instant ACH processing, and are currently investigating the issue.
Report: "Transaction and KYC degraded performance"
Last updateThis Incident has been Resolved. All transactions and KYC process normally now
There will be Transaction and KYC degraded performance on our end that is being caused by an Amazon AWS services degradation. We are monitoring the AWS Health Dashboard, and will update this status when AWS service availability has returned to normal levels.
Report: "Database Maintenance"
Last updateThis incident has been resolved and maintenance is completed. All operations perform normally
We are performing maintenance on our system between 6pm and 8pm (PST) tonight. Service will continue as normal, however you may see occasional intermittent delays on some requests of up to 30 seconds during this maintenance window.
Report: "Sandbox Maintenance"
Last updateThe maintenance has been completed. Sandbox is back to business as usual. Thank you for your patience.
[Sandbox Maintenance window 5pm-8pm PDT on 6/16/2022] Note: This will not impact our Production environment in any way. We are performing maintenance on our Sandbox environment to correct a bug introduced during a recent system upgrade. During this time, the Sandbox API interaction will continue to perform as normal, with the following exceptions: - Transactions in Sandbox will be delayed until after the maintenance has been completed. - KYC verifications in Sandbox will be delayed until after the maintenance has been completed. Wallet balances that were impacted by this bug will be correct after the maintenance has been completed.
Report: "Sandbox Transaction Processing Delay"
Last updateThis incident has been resolved. All transactions are processing normal
Processing of transactions in the sandbox is unusually slow due to an unusual volume of items to be processed. Expect for transaction requests in the sandbox to be delayed for upwards of 4 hours while our queues drain. We will update when processing returns to normal.
Report: "Degraded performance in Sandbox"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Sandbox outage"
Last updateSandbox API server had an outage. Investigation showed that Moesif configurations in settings.py caused the outage. Removed Moesif configs and restarted nginx/uwsgi to resolve.
Report: "Sandbox and Developer Console are responding slow"
Last updateThis Incident has been resolved.
We are currently investigating the issue
Report: "Security and availability enhancements on sandbox"
Last updateThis incident is resolved. Sandbox is operating normally
We are doing security and availability enhancements on Sandbox
Report: "Degraded Performance on sandbox and Prod APIs"
Last updateThis incident has been resolved 100%. Service is operating normally.
Things are fixed at AWS, we expect to be back to 100% in the next 30 minutes.
AWS reported InternetConnectivity operational issue on their end, which effected us in a way that some services lost connection to database, queues not processing the messages. Customers will see processing delay until AWS resolves this issue on their end
Report: "Website is down"
Last updateWebsite is backup and running.
Our website is down, we are currently investigating the issue.
Report: "Upgrading npm version on Prod Console"
Last updatenpm upgrade successful and console operating normally.
We are upgrading npm version on Prod console, this may take our server down for a bit, but we will try to backup soon.
Report: "Sandbox services and Console errors"
Last updateThis Incident has been Resolved. Sandbox and Developer Console Operates normally
We are currently investigating the issue
Report: "KYC Performance degraded on Production"
Last updateKYC should be operational now on Production.
We are moving some ethereum around from our Betalist authorized address which will cause some delay in KYC on Production
Report: "Performing Infrastructure maintenance tasks"
Last updateWe are done with the Maintenance work. Sandbox, API and Developer Console are backup and operational.
We are doing Infrastructure Maintenance tasks which will result in Prod api and Developer Console Unavailable and Sandbox api will have Partial outage. We will be back ASAP
Report: "Resizing the Sandbox Server"
Last updateThis incident has been Resolved. We are done with resizing, sandbox is operational now
We are Currently in the process of resizing our Sandbox Server.
Report: "Sandbox Issues"
Last updateThe incident has been resolved.
Services have been restored. We are continuing to monitor the situation while investigating the cause of the incident.
Our Sandbox is unavailable after a period of degraded performance. We are currently investigating the issue.
Report: "Delaying Prod Betalists"
Last updateThis Incident has resolved. Prod Betalistings should process normally now.
This issue has been identified and fix is being implemented.
We are currently Investigating the issue
Report: "Redeem, Transfer and Get Balance errors"
Last updateThis Incident has been resolved. All Sandbox Betalistings, transactions will be processed normally
We are continuing to monitor for any further issues.
A temporary fix has been implemented and we are monitoring the results while working on implementing a permanent solution.
Production transaction functionality has been restored. We are continuion to work on restoring Sandbox functionality.
We are currently investigating this issue.
Report: "Sandbox transactions failing to complete"
Last updateThis incident has been resolved, and all Sandbox transactions will be processed normally
An issue with KYC betalisting prevented new accounts from being fully betalisted, so transactions on new end-user accounts have been rejected. Betalisting is being completed, and subsequent transactions will complete successfully. This issue will be resolved shortly.
Some Sandbox transactions are failing to complete successfully. We are currently investigating the issue.
Report: "Sandbox services and Console errors"
Last updateThis incident has been resolved. Services are operational now
A fix has been implemented and we are monitoring the results.
Services have been restored, however we are continuing to investigate the root cause.
We are currently investigating the issue.
Report: "Unable to register new Apps in the Console"
Last updateThis incident has been resolved. Apps can now be created in the Console. All services are operational.
Console users were unable to register new Apps. The issue has been resolved.
Report: "Elevated API Errors"
Last update## Cause An update to our logging service in Sandbox caused requests with invalid or unexpected Content-Type headers to be rejected. ## Impact This only affected our Sandbox. Users who were not defining the expected Content-Type \(\`application/json\`\) experienced 500 responses on several endpoints. ## Resolution Release was rolled back within 1 hour to restore full functionality. Issue was patched prior to releasing that update again. ## Duration 10:50am PST - 11:35am PST
Several users reported experiencing an elevated level of API errors.