Sila

Is Sila Down Right Now? Check if there is a current outage ongoing.

Sila is currently Operational

Last checked from Sila's official status page

Historical record of incidents for Sila

Report: "API 0.2.118 Release Deployment"

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

We will be undergoing scheduled maintenance during this time.

Report: "Maintenance Window"

Last update
Completed

The scheduled maintenance has been completed.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

We will be undergoing scheduled maintenance during this time.

Report: "Card Transacion processing is delayed in Prod"

Last update
resolved

This incident has been resolved.

investigating

There is Card Transactions processing delay in Prod, we are currently investigating the issue.

Report: "Hotfix Deployment"

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

Report: "Partial Outage in the Sandbox Environment"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating this issue.

Report: "Evolve Service Outage"

Last update
resolved

This incident has been resolved.

identified

At 3:48 PM Thursday, July 18, Evolve reports that Microsoft is experiencing a global outage that impacts Evolve's Products. This outage impacts API and SFTP data interfaces, where data is received or distributed between Evolve and clients. Evolve is actively monitoring this incident and will provide updates as more information is made available.

Report: "Evolve ACH Origination Outage"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are continuing to work on a fix for this issue.

identified

We are continuing to work on a fix for this issue.

identified

Evolve has identified the cause of the incident. The cause was an unprompted password change within our Core System Provider's infrastructure, which disrupted our system operations.

investigating

Evolve Service Disruption Affected Systems Components: ACH Origination | API, ACH Origination | File Transfer, ACH Receiving | File Transfer, ACH Settlement, Book Transfer | File Transfer, Card Issuance | API, Check Printing | File Transfer, Ledger | API, Online Banking, RDC | File Transfer, RPPS | File Transfer, RTP Receive | API, RTP Send | API, vAccounts | File Transfer, Wire | API, Wire | File Transfer, Wire Settlement Services: Interface - Online Banking, Processing - ACH, Processing - Book Transfer, Processing - Core Banking Functions, Processing - Wires (Domestic), Processing - Wires (International)

Report: "Evolve ACH Origination Outage"

Last update
resolved

This incident has been resolved.

investigating

Evolve's ACH origination solution is down

Report: "KYC intermittent unavailability"

Last update
resolved

This incident is now resolved. The API was unavailable for a total of 8 minutes between 10:56-10:58ET and 11:05-11:10ET.

investigating

Alloy has reported they are seeing intermittent latency and unavailability for their API. They have identified the source of poor performance. Please be aware this could impact performance of KYC checks.

Report: "Evolve network outage"

Last update
resolved

As of 8:25 am June 12, Evolve has determined this incident has been resolved.

monitoring

Evolve reports that as of 9:45 am, systems are operating as expected. Evolve will continue to monitor the situation and communicate updates as necessary.

investigating

As of 8:24 am PT today, June 11, Evolve is investigating a network connectivity issue that is impacting essential Evolve services. Evolve has recently discovered impacts to data extract delivery and are investigating further. Evolve is actively working on a resolution to restore normal operations as soon as possible.

Report: "Evolve Service Disruption"

Last update
resolved

Upon monitoring and review, this incident is resolved.

monitoring

Evolve is continuing to work to ensure all clients are up to date in both transactions and file delivery. Evolve will continue to monitor processing and file delivery to clients.

monitoring

Evolve is continuing to work through the reconciliation process from the incident date to ensure accuracy and consistency in data. Evolve will continue to monitor processing and file delivery to clients.

monitoring

Evolve's connection to the data center has been re-established. Transactions and data extracts continue to process normally. Evolve will continue to monitor the incident to ensure all operations are functional.

monitoring

Evolve complete maintenance over the weekend to stabilize network services at their Data Center on Core Service Provider Solutions. As of 6:35 am PT, June 3, Evolve is working diligently to get current in transaction processing and will continue to monitor processing and file delivery to clients.

monitoring

Evolve continues to generate and deliver data extracts for all previously affected services to our clients including the Wire In/Out extracts. Transaction Processing: Evolve is current for all ACH and Domestic Wire transactions.

monitoring

Evolve is still working to generate and deliver data extracts to all clients. Evolve will continue to monitor processing and file delivery.

monitoring

As of 10:45 am PT, May 31, Evolve's End of Day processing has completed. Transaction processing has resumed. Evolve will continue to monitor processing and file delivery to clients.

identified

Evolve’s Core System Provider is nearing completion of End of Day processing. Incoming transactions should begin processing shortly.

identified

Additional update from Evolve - Evolve’s transaction processing is currently on hold due to an operational setback attributed to an issue with their core provider. This is due to the large transaction size of files during End of Day processing from the network outage. Root Cause Clarification:
 This issue stemmed from Evolve’s systems attempting to catch up from the previous day's transactions. Mitigation:
 Evolve is actively working with their Core System Provider to quickly finish End of Day processing.

identified

At 7:22 am PT this morning, May 31, Evolve sent out a notification that due to the ongoing network issues affecting their data center, they needed to restart their End of Day processing. Consequently, all transactions will be temporarily paused to ensure data integrity and system stability. At 7:41 am PT, Evolve sent out another notification that they are working with their data center on fully restoring all services after overhauling the network. Evolve was current on all transactions as of approximately 4am ET. As of now, Evolve has paused the sending of data extracts for this final maintenance period. Evolve is able to process transactions but is currently unable to send out data extract files as they relate to transactions.

monitoring

As of 2:17 pm central, May 30, Evolve reports that they are now operational and are working to process transactions and files that have been updated by the network outage.

identified

At approximately 5:00 am CDT on May 29, Evolve lost internet connection to the Evolve Network. This loss of connection impacts all data extract files. As of 8:30 am CDT on May 30, Evolve continues to collaborate with their data center internet provider to address the ongoing internet network outage and expedite the restoration of all services. Evolve is working through the backlog to bring all transactions current.

Report: "Evolve ACH Processing Delay"

Last update
resolved

As of 10:56 AM CT, processing has resumed and Evolve is not expecting any further delays. Evolve will continue to monitor processing and file delivery to clients.

investigating

As of 8:13 AM CT, Tuesday, April 30, 2024, Evolve reports ACH Wire and File processing is delayed. There is a processing issue with their Core System Provider. Evolve is investigating the cause.

Report: "ACH File Processing Evolve Outage"

Last update
resolved

As of 5:30 CT, Evolve's internet service has been restored. Evolve will continue to monitor the incident to ensure all operations are functional.

identified

Shortly after 3 pm Central Time today (04.23.2024) , Evolve lost internet connection from the Evolve Network. Evolve anticipates this to be a recurrence of yesterday's incident (04.22.2024). Impact: Same Day ACH originated files processed successfully within the Same Day window. Evolve is working toward a timely solution to avoid impacts to later delivery windows.

Report: "ACH File Processing Evolve Outage"

Last update
resolved

Evolve reports this incident has been resolved as of 5:30 AM PT. Evolve's internet service has been fully restored, and Evolve will continue to monitor the incident to ensure all operations are functional.

identified

Update: Same Day ACH files did not make the same day window. As Evolve works through the network connectivity incident, Evolve aims to process the missed Same Day ACH transactions as Next Day ACH.

identified

Shortly after 2 pm Central Time, Evolve lost internet connection from the Evolve Network. Evolve has actively been working on a resolution from the start of the incident. Impact: ACH file processing is expected to be impacted by this lapse in internet connectivity. Missed delivery windows will result in delayed ACH transaction delivery and posting. Please see https://status.getevolved.com for most recent updates.

Report: "ACH file transmittal was delayed, settlement time for some transactions will be delayed"

Last update
resolved

This incident has been resolved.

monitoring

A portion of the transactions from this event were found to have still been delayed. They are now expected to settle in the remote bank accounts later today by 18:00 ET 02/01.

monitoring

The process responsible for the transmittal of ACH transactions to the ACH payment network experienced issues on the evening of 1/30/2024 through to 09:00 Pacific 1/31/2024. We have implemented a fix and are monitoring the process closely. * Same Day transactions from the first Same Day window at 05:15 Pacific on 01/31 that should have settled by 1:00pm ET will instead settle 4 hours later at 5:00pm ET. * Next Day transactions from 01/30 will be sent on 01/31, and should settle 24 hours later than expected.

Report: "We are experiencing an increase in the rate of R03 returns"

Last update
resolved

This incident has been resolved.

monitoring

Our upstream ACH provider has informed us that a fix for the underlying issue was implemented on 10/31. There is usually a two-day lag on returns, so it's expected that we would continue to see an elevated rate of R03 returns for a few days after the fix. We are now seeing that the rate of R03s for items is dropping back to the normal/historical levels. We will continue to monitor the R03 rates for another two business days, and will mark it as resolved if the R03 rate continues to match historical levels.

monitoring

There has been a marked increase in the number of R03 (No bank account/unable to locate account) returns. This has been traced to our upstream ACH provider experiencing an issue with their software vendor regarding leading 0s being erroneously stripped from bank account numbers. We are constant contact with our ACH provider on this issue, and will update this incident when we are informed it has been resolved.

Report: "Standard ACH submission for November 1st may be delayed until November 2nd at 9:30 Pacific"

Last update
resolved

We have verified with our ACH provider that all files were processed at the expected times, and that no customer transactions were impacted; all transactions were processed at the original expected times. This incident has been resolved.

monitoring

Our ACH provider has informed us that this evening's ACH submissions have been accepted and forwarded on to the ACH network for processing within the normal window this evening.

monitoring

We have successfully sent out a NACHA file for the day's Standard ACH transactions at 18:40 US/Pacific, but have very likely missed today's cutoff. In that case, we expect the transactions to be forwarded on to the ACH network early tomorrow morning. We will be updating this incident as we learn more.

investigating

We are currently investigating this issue.

Report: "Delay in Transactions processing"

Last update
resolved

There is a fox has been deployed and transactions are processing normally now

investigating

There is delay in processing transactions on Prod. Team is currently investigating the issue

Report: "Maintenance to add Load balancer for Sandbox Servers"

Last update
resolved

This incident has been resolved. All our sandbox traffic is now routing through Load Balancer, please make sure to clear cache If you see any loading issues on Sandbox

identified

We are working on attaching Application Load Balancer for our Sandbox API servers to reduce latency in API requests

Report: "AWS email shutdown impacting services"

Last update
resolved

This incident has been resolved and all operations performed normally.

monitoring

The ability to send email from our API servers and AWS infrastructure (Lambdas, etc.) has been restored as of 19:30 PDT. We are currently monitoring all functionality that was affected by this service interruption, but are expecting no further impact.

identified

Because we lost the ability to send Email temporarily, there will be degraded performance in team invitations, transactions for accounts with non-functional transaction webhooks, webhook email alerts (webhooks themselves are OK), sending statements, password reset, creating console logins. Working with AWS to get the ability to send emails back.

Report: "Processing delay in Transactions and KYC requests"

Last update
resolved

This incident has been resolved, all operations on prod performed normally

identified

There will be processing delay in prod KYC requests and transactions

Report: "Issue affecting all customers' ability to withdraw the full amount of a wallet's balance"

Last update
resolved

This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

We have identified a bug which was introduced into production at 07-06-2023 13:30 PDT (Thursday afternoon) affecting all customers' ability to withdraw the full amount of a wallet's balance. This would result in transfer and redeem requests being refused as NSF when in fact there were sufficient funds. A fix has been deployed as of Friday morning 07-07-2023 10:45am PDT. Customers are reporting that transactions are now completing successfully as expected.

Report: "Issue affecting all customers' ability to withdraw the full amount of a wallet's balance"

Last update
resolved

We have identified a bug which was introduced into production at 07-06-2023 13:30 PDT (Thursday afternoon) affecting all customers' ability to withdraw the full amount of a wallet's balance. This would result in transfer and redeem requests being refused as NSF when in fact there were sufficient funds. A fix has been deployed as of Friday morning 07-07-2023 10:45am PDT. Customers are reporting that transactions are now completing successfully as expected.

Report: "Upgrading Sandbox to Ubuntu 22.04"

Last update
resolved

We are done with the upgrade and sandbox is operational now

identified

We are working on upgrading sandbox to Ubuntu 22.04 version

Report: "Planned Maintenance"

Last update
resolved

This incident has been resolved.

identified

Maintenance is completed, we are back to operational

identified

We are upgrading Instance types of prod server for better performance

Report: "Database version upgrade"

Last update
resolved

We are done with Database upgrade. Prod API is operational now.

identified

We are upgrading production database version upgrade. API will be unavailable until that’s done

Report: "Few Endpoints on Sandbox throwing 504s"

Last update
resolved

Incident has been resolved. We found out long running queries were the reason for timeout and killed those processes. Sandbox is operational now

investigating

After the database upgrade, we are seeing some of the endpoints throwing 504s. We are looking into the issue

Report: "Database Version Upgrade"

Last update
resolved

We are done with upgrading database version. All operations should be performed normally now

identified

We are upgrading our Sandbox Postgres Database version to 14.4

Report: "Developer Console Register and Login Unavailable"

Last update
resolved

We are done with hotfix release and site is operational now

identified

Releasing the fix that Enforce captcha checks on server-side for all requests that come from console pages that require Captcha pages

Report: "Instant ACH Delays"

Last update
resolved

This incident has been resolved.

investigating

The issue has been resolved, we are continuing to monitor.

investigating

We are seeing delays in Instant ACH processing, and are currently investigating the issue.

Report: "Transaction and KYC degraded performance"

Last update
resolved

This Incident has been Resolved. All transactions and KYC process normally now

identified

There will be Transaction and KYC degraded performance on our end that is being caused by an Amazon AWS services degradation. We are monitoring the AWS Health Dashboard, and will update this status when AWS service availability has returned to normal levels.

Report: "Database Maintenance"

Last update
resolved

This incident has been resolved and maintenance is completed. All operations perform normally

identified

We are performing maintenance on our system between 6pm and 8pm (PST) tonight. Service will continue as normal, however you may see occasional intermittent delays on some requests of up to 30 seconds during this maintenance window.

Report: "Sandbox Maintenance"

Last update
resolved

The maintenance has been completed. Sandbox is back to business as usual. Thank you for your patience.

monitoring

[Sandbox Maintenance window 5pm-8pm PDT on 6/16/2022] Note: This will not impact our Production environment in any way. We are performing maintenance on our Sandbox environment to correct a bug introduced during a recent system upgrade. During this time, the Sandbox API interaction will continue to perform as normal, with the following exceptions: - Transactions in Sandbox will be delayed until after the maintenance has been completed. - KYC verifications in Sandbox will be delayed until after the maintenance has been completed. Wallet balances that were impacted by this bug will be correct after the maintenance has been completed.

Report: "Sandbox Transaction Processing Delay"

Last update
resolved

This incident has been resolved. All transactions are processing normal

identified

Processing of transactions in the sandbox is unusually slow due to an unusual volume of items to be processed. Expect for transaction requests in the sandbox to be delayed for upwards of 4 hours while our queues drain. We will update when processing returns to normal.

Report: "Degraded performance in Sandbox"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Sandbox outage"

Last update
resolved

Sandbox API server had an outage. Investigation showed that Moesif configurations in settings.py caused the outage. Removed Moesif configs and restarted nginx/uwsgi to resolve.

Report: "Sandbox and Developer Console are responding slow"

Last update
resolved

This Incident has been resolved.

investigating

We are currently investigating the issue

Report: "Security and availability enhancements on sandbox"

Last update
resolved

This incident is resolved. Sandbox is operating normally

identified

We are doing security and availability enhancements on Sandbox

Report: "Degraded Performance on sandbox and Prod APIs"

Last update
resolved

This incident has been resolved 100%. Service is operating normally.

monitoring

Things are fixed at AWS, we expect to be back to 100% in the next 30 minutes.

identified

AWS reported InternetConnectivity operational issue on their end, which effected us in a way that some services lost connection to database, queues not processing the messages. Customers will see processing delay until AWS resolves this issue on their end

Report: "Website is down"

Last update
resolved

Website is backup and running.

investigating

Our website is down, we are currently investigating the issue.

Report: "Upgrading npm version on Prod Console"

Last update
resolved

npm upgrade successful and console operating normally.

identified

We are upgrading npm version on Prod console, this may take our server down for a bit, but we will try to backup soon.

Report: "Sandbox services and Console errors"

Last update
resolved

This Incident has been Resolved. Sandbox and Developer Console Operates normally

investigating

We are currently investigating the issue

Report: "KYC Performance degraded on Production"

Last update
resolved

KYC should be operational now on Production.

investigating

We are moving some ethereum around from our Betalist authorized address which will cause some delay in KYC on Production

Report: "Performing Infrastructure maintenance tasks"

Last update
resolved

We are done with the Maintenance work. Sandbox, API and Developer Console are backup and operational.

identified

We are doing Infrastructure Maintenance tasks which will result in Prod api and Developer Console Unavailable and Sandbox api will have Partial outage. We will be back ASAP

Report: "Resizing the Sandbox Server"

Last update
resolved

This incident has been Resolved. We are done with resizing, sandbox is operational now

identified

We are Currently in the process of resizing our Sandbox Server.

Report: "Sandbox Issues"

Last update
resolved

The incident has been resolved.

monitoring

Services have been restored. We are continuing to monitor the situation while investigating the cause of the incident.

investigating

Our Sandbox is unavailable after a period of degraded performance. We are currently investigating the issue.

Report: "Delaying Prod Betalists"

Last update
resolved

This Incident has resolved. Prod Betalistings should process normally now.

identified

This issue has been identified and fix is being implemented.

investigating

We are currently Investigating the issue

Report: "Redeem, Transfer and Get Balance errors"

Last update
resolved

This Incident has been resolved. All Sandbox Betalistings, transactions will be processed normally

monitoring

We are continuing to monitor for any further issues.

monitoring

A temporary fix has been implemented and we are monitoring the results while working on implementing a permanent solution.

investigating

Production transaction functionality has been restored. We are continuion to work on restoring Sandbox functionality.

investigating

We are currently investigating this issue.

Report: "Sandbox transactions failing to complete"

Last update
resolved

This incident has been resolved, and all Sandbox transactions will be processed normally

identified

An issue with KYC betalisting prevented new accounts from being fully betalisted, so transactions on new end-user accounts have been rejected. Betalisting is being completed, and subsequent transactions will complete successfully. This issue will be resolved shortly.

investigating

Some Sandbox transactions are failing to complete successfully. We are currently investigating the issue.

Report: "Sandbox services and Console errors"

Last update
resolved

This incident has been resolved. Services are operational now

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Services have been restored, however we are continuing to investigate the root cause.

investigating

We are currently investigating the issue.

Report: "Unable to register new Apps in the Console"

Last update
resolved

This incident has been resolved. Apps can now be created in the Console. All services are operational.

identified

Console users were unable to register new Apps. The issue has been resolved.

Report: "Elevated API Errors"

Last update
postmortem

## Cause An update to our logging service in Sandbox caused requests with invalid or unexpected Content-Type headers to be rejected. ## Impact This only affected our Sandbox. Users who were not defining the expected Content-Type \(\`application/json\`\) experienced 500 responses on several endpoints. ## Resolution Release was rolled back within 1 hour to restore full functionality. Issue was patched prior to releasing that update again. ## Duration 10:50am PST - 11:35am PST

resolved

Several users reported experiencing an elevated level of API errors.