Historical record of incidents for Floify
Report: "Google Cloud and Amazon Web Services Outages"
Last updateWe are receiving reports that various systems, including GCP and AWS, are experiencing outages, resulting in some services within Floify being impacted. We are reviewing these reports actively to identify resolution steps to restore full functionality. Encompass has also reported service issues, which could impact integrations to that system directly. For more information, you can review the ICE status center at the following link: https://emstatuscenter.elliemae.com/
Report: "eSign and Disclosures are showing as blank"
Last updateThis incident has been resolved.
We have implemented the fix to this issue and are seeing eSign functionality return to normal. We are monitoring the system to ensure everything is working as expected.
We have found an issue causing encompass disclosures and other eSign packages to appear as blank in the system. We are in the process of rolling back a system update made overnight to resolve this issue.
Report: "Twilio SMS Delivery"
Last updateThis incident has been resolved.
Our SMS Partner, Twilio, has informed us of known issues with SMS Delivery Failures to the Onvoy Spectrum Mobile Network in United States of America Via Subset of Shortcode and Toll-free Numbers. This can potentially impact the delivery of SMS messages out of our system. We are monitoring this issue and provide updates as they become available. While our system remains fully operational, if you experience any MFA issues, you may need to reset your configuration to be sent via email. Please reach out to support@floify.com if you have any questions or need assistance!
Report: "SendGrid Email Delivery"
Last updateThis incident has now been resolved.
Our email service provider, SendGrid, has informed us of a known issue with sending emails. This can potentially impact the delivery of emails out of our system. We are monitoring this issue and provide updates as they become available. While our system remains fully operational, if you experience any MFA issues, you may need to reset your configuration to be sent via text. Please reach out to support@floify.com if you have any questions or need assistance!
Report: "Issues Logging into Floify"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Our team has implemented a resolution and we will continue to monitor to ensure stability. Users should now be able to authenticate and access the system correctly.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue. We have identified the issue, and we are actively working on implementing a fix.
Report: "Floify System Outage"
Last updateWe identified and resolved an issue with various functionality in our system Saturday afternoon. We are working on restarted any failed system tasks, including LOS-integrated file creation. While functionality was returned to normal Saturday afternoon by our engineers, we are working to fix any specific loans impacted during the outage. If you need any specific support from our team around this issue, please feel free to reach out to us at support@floify.com
Report: "MFA Error + Home Dashboard View"
Last updateThis issue has been resolved. All Home Dashboards have been reverted to user-entered preferences with the fix going live into production.
We have identified an issue with MFA logins for some users. The issue is caused by having a home dashboard view different from the users home team. As a short term solution, we will be changing all users home dashboard under the user profile to the home team assigned to the account. Once the issue has been resolved, we will automatically change this configuration back to how it is currently configured. If you have any questions, difficulties, or need information on navigation in your account with this change, please reach out to us at support@floify.com
Report: "App Outages"
Last updateThis is resolved. We have implemented a fix to the root of the direct issue and are not seeing it occur further.
The application has been brought back online. We are monitoring the system to ensure stability.
We are investigating continued reports of our application going down and providing server errors.
Report: "Floify Application Down"
Last updateThis morning, the Floify Application was offline for about 21 minutes. We are investigating the root cause of this. The application was restored, but we are investigating the root of the issue to prevent this from occuring again.
Report: "Twilio SMS Delivery"
Last updateWe have found that the Twilio SMS issue has been resolved as of last night. We continued monitoring our systems and have confirmation that this has returned to normal.
Our SMS Partner, Twilio, has informed us of known issues with SMS delivery in connection to multiple large network providers, including AT&T and T-Mobile. This can potentially impact the delivery of SMS messages out of our system. We are monitoring this issue and provide updates as they become available. While our system remains fully operational, if you experience any MFA issues, you may need to reset your configuration to be sent via email. Please reach out to support@floify.com if you have any questions or need assistance!
Report: "Issue with milestone delivery and LOS Loan Assignment"
Last updateThis issue has been resolved in full.
We have implemented fixes for both reports and are monitoring the system.
We have performed a partial rollback of the impacted system changes, and milestone functionality should return to normal. We are still investigating the isolated incidents of LOS loan assignment for a small group of users.
We are investigating incidents of issue with milestone delivery and LOS loan assignment. Further updates will be available soon.
Report: "Loan Submission and Creation Issues"
Last updateThis issue has been resolved
Our investigation has ben concluded and we have implemented a preventative solution. We are continuing to monitor the solution to verify resolution.
A fix has been implemented and we are monitoring the system. Please reach out to us at support@floify.com if you experience this issue further.
We are investigating an issue with loan submission and creation and are working on resolution.
Report: "Loan Creation Application Submission, and Login Issues"
Last updateThis issue is resolved. We are putting further monitoring for similar issues into place and are implementing a system change for this. If you run into further issues, don't hesitate to reach out to our support team at support@floify.com
We have resolved the issue blocking loan creation and login issue. The system has returned to normal function and we are monitoring for further occurence.
We are currently investigating reports of issues with loan file creation and borrower application submission, as well as some login issues, and may perform intermittent system refreshing to return the system to normal functionality.
Report: "Loan Creation and Conversion Error Message"
Last updateThis issue has been resolved. We have restarted all automated tasks and loan conversion and creation is now accessible as normal. If you need any further support, please reach out to us!
A fix has been implemented and we are monitoring the system for recurrence. We have implemented additional automated monitoring. Manual loan creation and conversion is now operational. We are looking into restarting all automatic prospect conversion for files that were impacted by this incident and will post further information when it becomes available.
We will be performing an application refresh as part of the resolution, and will continue working toward resolution. The application will remain online.
We are currently investigating reports of failures to create loan flows via Start New Loan Flow or conversion from a prospect application. Users may be unable to convert or create a loan and will receive an error message when experiencing the issue.
Report: "Intermittent Performance Issues"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
With the fix implemented yesterday, this issue has been resolved. We will continue monitoring system performance.
A fix has been implemented and the application has been brought back online. We are continuing to monitoring system performance to ensure the issue is fully resolved.
We are currently testing a fix to resolve this issue. We have additionally engaged AWS to resolve this issue as quickly as possible. Another update will be posted here on our status page shortly.
We have taken the app offline to work on and implement a resolution for the identified issues. We will update this page again when we have brought it back online.
We've identified another issue and are working on a fix. Some users may notice degraded performance at this time.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented. We should be back online in 15 minutes.
We are taking the app offline for a short period of time to resolve the issue. We expect to be back online in 15 minutes.
We are currently experiencing intermittent performance issues following a recent database update. Our team is actively investigating and working to resolve this as quickly as possible. We appreciate your patience and will provide updates as we make progress. Thank you for your understanding.
Report: "Short ICE E-Close Delay in Borrower UI"
Last updateThis incident has been resolved.
The issue has been identified and we are currently hard at work at implementing a resolution for this delay. An update will be posted again once this issue has been resolved!
We are currently investigating the root of a short delay borrowers may experience when using the ICE E-Close functionality. The interface is still functional, but we are seeing reports that it is taking longer than usual to load the page.
Report: "Network Outages Impacting MFA"
Last updateThis issue seems to be resolved. At that, certain networks may continue to have further issue. If you need assistance with your multi-factor authentication, reach out to us at support@floify.com
We are receiving reports of various network carriers/providers experiencing delays and outages. While Floify remains operational and is sending out communications normally, some users may be experiencing delays with the Multi-Factor Authentication codes when sent via SMS. If you need assistance with your MFA setup, please connect with your admin to adjust your entry. We recommend configuring your account to have both email and SMS MFA setup. For questions, please reach out to support@floify.com!
Report: "Delayed Email Services"
Last updateWe implemented an alternate workflow for users without DKIM and SPF records in place in your DNS Host to have all emails sent out from info@floify.com. To set up your SPF and DKIM records, please reach out to support@floify.com or navigate to our help article for instructions at https://help.floify.com/hc/en-us/articles/360040773012-Stop-Emails-From-Going-to-Spam. Users utilizing their own mail servers or that have SPF and DKIM records in place in their DNS were unaffected by this.
This issue has been isolated to recipients with Google emails. We are working with Google to resolve this as quickly as possible.
We are currently investigating an issue with email services in Floify resulting in delivery delays. At this time, emails are being delivered, but potential delays are expected.
Report: "Merge Field Template Editing Issues"
Last updateThis incident has been resolved. For further support on any templates needing help reconciling merge fields, reach out to support@floify.com.
We have implemented a fix to allow for the editing of templates again. Any templates edited since August 28th, 2024 may have been impacted by this. Please review any changes you may have made to the following: - Email Templates - Milestone Templates - Email Signatures - Text Message Templates If you have any questions or need support in adjusting your templates, please reach out to support@floify.com!
We are currently investigating an issue with template editing where saving a template removes any merge fields. When saving a template, the merge fields are not be saved correctly the templates. Unedited templates are unaffected. This can impact, but is not limited to the following: - Email Templates - Milestone Templates - Email Signatures - Text Message Templates While this issue is being investigated and resolved, please refrain from editing your communication templates to avoid issue.
Report: "Encompass Document Upload Delays"
Last updateThis incident has been resolved. Documents are no longer being delayed and impacted document uploads to Encompass have been completed. We are implementing changes to prevent this from happening in the future.
We are receiving reports of Encompass document upload delays in some cases (both automatic and Stack and Send uploads). We are investigating this issue and will update once we have more information about the root cause and resolution.
Report: "Degraded Performance with SendGrid Mail Delivery"
Last updateThis issue was deemed after further review to only have had impact to our systems for a window in the morning. SendGrid resolved and monitored throughout the day and confirmed their systems are fully operational. This has been fixed!
The mail server used by the Floify application is experiencing delivery delays in some cases. We are monitoring their outage and will provide update on our status page upon resolution in their system. Some emails, including MFA emails, may experience delayed delivery.
Report: "LOS Loans are being hidden intermittently"
Last updateThis incident has been resolved.
We have implemented a fix for the issue by rolling back the code released previously causing the issue. We are monitoring for any further impacts on individual accounts, but have provided fix for reported loan files.
We are currently investigating an issue where loans are being hidden from the pipeline due to interactions with LOS systems. We are working to identify the root cause, temporary solution, and long term fix.
Report: "Floify Authentication Service"
Last updateLast night, our authentication service ran into issues, preventing users from logging in. Users who were already logged in were unaffected, but new login attempts were receiving an error message. This issue has been resolved. We are reviewing how we can improve the system framework around the authorization service to avoid these errors altogether in the future.
Report: "Custom Questionnaire Missing Questions"
Last updateThis incident has been fully resolved. Custom questionnaire PDFs generated between Tuesday, July 16th and Monday, July 22nd have been fixed to reflect the information in the application upon creation.
Floify will be regenerating all the incorrectly generated Custom Questions PDF with the correct version and will replace the existing PDF file in both Floify and the associated LOS. The time stamp of the PDF would not change in Floify and the LOS. Once completed, we will update the status here.
We have resolved an issue with the custom questionnaire where in some cases the document would come through blank or with missing questions. The custom questionnaire documents impacted were generated between Tuesday, July 16th at 3:00 AM MT and Monday, July 22nd at 5:30 PM MT . We have resolved the root issue, and are finalizing plans to correct any impacted documents.
Report: "Third party service impacts from CrowdStrike issues"
Last updateAs of July 20th 11:00 am MT, ICE has confirmed their services are operating as normal. In Floify, all Encompass related services are operating properly and any previously failed tasks have since been retried and processed.
ICE Mortgage has made significant progress with normalizing third party services. However, they are still working towards full stabilization. We will continue to monitor these services and provide updates as we have them.
While Floify is not directly impacted by the global IT outage caused by the Crowdstrike issue, some of our third party integrations and services are experiencing outages which impact certain functionality in Floify. We will continue to monitor these services and provide updates as we have them. 1) Byte services have been restored and are operational. 2) Encompass services continue to experience outages. For customers integrated with Encompass in Floify, these services may be unavailable due to the Encompass outage: - Disclosures - Milestones - Pre-approval letters - Start and edit loan flow - Stack and send - Document uploads 3) Twilio, our SMS service for MFA notifications, may experience delays.
Some Byte services are also experiencing intermittent latency and/or full outages due to the global IT outage. We will continue to monitor all third party services and provide updates as we have them.
ICE Mortgage has confirmed an outage to their Encompass services, which impacts Floify's integration with Encompass. As ICE provides more updates, we will update our status page. More information on this outage can be found at https://emstatuscenter.elliemae.com/.
Report: "Application outage"
Last updateThe issue has been resolved, and the application is now online. The outage was caused by a high volume of notifications being sent, which caused the server to overload.
We are currently investigating an outage and will provide updates as we have them.
Report: "Allowlist email server issue"
Last updateThis incident has been resolved.
The fix has been successfully implemented. Email and LOS files are now processing as expected. We will continue to monitor the logs to ensure ongoing stability.
We are currently working to resolve an issue with IP allowlisted addressed. We are currently implementing a fix to revert the IP address to the previous one. There is no action needed at this time and the issue will be resolved in around 1 hour.
Report: "Automatic deadline reminders are not being sent"
Last updateThis incident has been resolved and automatic deadline reminders resumed being sent as of 11:00am MT.
We are investigating an issue with automatic deadline reminders not be sent. Manual deadline reminders are still being sent. We will update this incident as we have more information.
Report: "LendingPad Service Outage"
Last updateLendingPad has resolved the issue, and loan file creation has been successful. We will continue to monitor the LendingPad loan creation process to ensure ongoing reliability.
LendingPad is actively working to resolve this issue. Currently, there are no additional updates available.
LendingPad is currently experiencing an outage, which is preventing loan files to be created from Floify. We will update our status page once LendingPad has resolved this issue.
Report: "Users unable to access loans"
Last updateThis incident has been resolved.
The issue has been identified and a fix is currently being implemented.
We are currently investigating an issue with users receiving a gray screen when accessing loans. We will update this page when we have further information.
Report: "T-Mobile MFA SMS Delay"
Last updateTwilio has resolved this issue.
Twilio, SMS service provider, has identified delays in SMS outgoing traffic for T-Mobile. We will update our status page once Twilio has resolved this issue. More details from Twilio can be found here: https://status.twilio.com/
Report: "Intermittent Outages"
Last updateThis issue has been resolved.
A fix has been implemented and we will continue to monitor the effectiveness of these fixes prior to updating this status to resolved.
We are continuing to work on a fix for this issue.
Our engineers have made significant progress towards pinpointing the root cause of the intermittent outages. To gather more granular data and confirm the exact cause, we'll be adding additional logging to the system during tomorrow's scheduled automatic refresh. We'll continue to provide updates on this status page as we gather more information. Thank you for your patience and understanding.
We are currently investigating intermittent outages that last around 2 minutes and occur once a day. Our team is actively working to resolve these issues as quickly as possible. We apologize for any inconvenience and appreciate your patience.
Report: "Third Party Icon Loading Delay"
Last updateThis incident has been resolved.
A third party service hosting certain icons in Floify, is experiencing an outage. As a result, some icons in Floify are taking longer than expected to load. Font Awesome is working on resolving this issue and we will update our status page once it has been resolved.
Report: "Encompass Integrated - Edit Loan Flow Issue"
Last updateThis incident has been resolved.
We've identified an issue when changes are made to borrower information in Encompass that are not updating in Edit Loan Flow in Floify. Our developers have identified the issue and are currently testing a fix. We will provide further updates on this page as soon as we have more information.
Report: "MFA SMS Delay"
Last updateThis incident has been resolved.
Twilio, SMS service provider, has identified delays in SMS outgoing traffic. We will update our status page once Twilio has resolved this issue. More details from Twilio can be found here: https://status.twilio.com/
Report: "Unable to download documents through API"
Last updateThis incident has been resolved.
We've identified an issue where documents cannot be opened when retrieved via API. We are actively working on a solution and will provide updates as soon as more information becomes available.
Report: "Floify Login Interruption"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The problem has been identified, and we are currently working on implementing a solution. The error has been isolated to situations where users click the home button or the logo image. Users should not encounter login issues except in these specific scenarios.The error message some users are receiving is unrelated to the identified issue, and there is no indication of unauthorized account access from another location. We will continue to update this page as we have more information.
We are currently investigating this issue.
Report: "Users are experiencing issues with submitting disclosure documents"
Last update**Issue Summary** From 3:00am to 12:07pm MT, Floify experienced a partial system outage, leading to users being unable to submit disclosures and encountering user access errors. The root cause was due to a code release which was scheduled in the daily system refresh. **Timeline \(all times in Mountain Time\)** * 3:00 AM - Code was deployed with the daily automatic system refresh. * 8:05 AM - Users reported errors when attempting to submit disclosures and inadvertently were logged out of the system. * 8:10 AM - Floify engineers were notified and began their investigation into the issue. * 11:40 AM - Floify engineers identified the root cause of the issue and began to rollback the code. * 12:07 PM - The rollback was completed and the issue was resolved. **Root Cause** The root cause was determined to be an issue with the code released at 3:00am. Due to this, user sessions were incorrectly handled and some users who were attempting to e-sign were incorrectly logged out. **Resolution and Recovery** This issue was resolved by following internal protocols by reverting to the application instance prior to the 3:00am refresh. **Corrective and Preventative Measures** The following are actions we are taking to address the underlying causes of the issue and to help prevent recurrence and improve response times: Short term: * Rollback application instance to prevent further issues \(completed\) Long term: * The quality assurance team will persist in developing automated test cases to conduct thorough testing before the release. Floify is dedicated to continuously adapting and enhancing our technology and processes to prevent service interruptions. We sincerely appreciate your patience and apologize for the impact of this outage. Your business and ongoing support are greatly valued. Sincerely, The Floify Team
This issue has been resolved and a root cause analysis will be posted shortly.
We are currently investigating an issue with submission of disclosure documents.
Report: "Disclosures unable to be downloaded"
Last updateThis incident has been resolved and all impacted disclosures have been processed.
We've identified an issue with downloading disclosure files. The root cause has been identified, and we are working to deploy the necessary changes to resolve the issue.
Report: "Interruptions with Disclosure Package Downloads"
Last update### Floify Service Interruption January 19, 2024 This incident report details the Floify interruption of service that occurred on January 19, 2024. **Issue Summary** From 11:30 AM MT Jan 19 to 7:30 AM MT Jan 20, Floify experienced a partial interruption of service impacting disclosures. Disclosures were not able to be downloaded and therefore could not be added to the loan pipeline and further, signed and completed. The root cause was due to a code release at 3:00 AM MT on Jan 19 that impacted the disclosure functionality. **Timeline \(all times in Mountain Time\)** * 3:00 AM Jan 19 – Floify code was deployed to Production * 11:30 AM Jan 19 – Errors specific to disclosure downloads were identified * 11:42 AM Jan 19 – Floify engineers were notified and began their investigation into the issue. * 5:30 PM Jan 19 - Floify engineers addressed the issue in the code, and the QA team initiated testing. * 6:13 AM Jan 20 – A code fix was deployed to Production. This addressed the disclosure download issues. * 6:34 AM Jan 20 - A database update was used to retry the failed disclosure downloads. **Root Cause** The root cause of this issue was a code fix that affected the functionality between the Floify App and the Floify eSign Service regarding co-mortgagors specifically when creating disclosure signing documents. **Resolution and Recovery** * The issue was resolved by completing a code release that resolved the issue. * After the code fix release and testing was completed, a database update was used to retry the failed disclosure downloads. **Corrective and Preventative Measures** The following are actions we are taking to address the underlying causes of the issue and to help prevent recurrence and improve response times: _Short term:_ * Database updates were used to correct failed disclosure downloads. _Long term:_ * The quality assurance team will persist in developing automated test cases to conduct thorough testing before the release. Floify is committed to continually adapting and improving our technology and process to prevent service interruptions. We appreciate your patience and again apologize for the impact of this interruption of service. We thank you for your business and continued support. Sincerely, The Floify Team
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've identified interruptions with disclosure packages being downloaded and our developers are currently testing a fix. We will update this page as we have more information.
Report: "Errors when attempting to start or convert loans"
Last updateAll impacted failed loans have successfully been converted and this issue is resolved.
Between 3:00pm to 4:30pm MT, some users were unable to start or convert loans. As of 4:30pm MT, the issue has been resolved and there have been no reports of loan start or converting instances. Our team is working on automatically converting failed loans that were impacted. This page will be updated once all failed loans have been converted.
Report: "Missing State in Start New Loan Flow"
Last updateThis incident has been resolved.
Our engineers have identified the issue and a fix is currently being tested.
We are currently investigating an issue impacting some users when starting a new loan flow. The states are not appearing in the dropdown. We will update this page as we have more information.
Report: "Delayed Email Notifications"
Last updateThis incident has been resolved and email notifications are no longer delayed.
The issue has been identified and our team is working on a resolution. Once the fix has been implemented, we will update this page to 'Resolved'.
We are currently investigating an issue with intermittent delays in notifications being sent.
Report: "Delay in email notifictations"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Floify eSign (mobile device version only) - Issue when attempting to complete text boxes"
Last updateThis incident has been resolved.
We've identified an issue when attempting to complete documents with text fields through Floify eSign on a mobile device (iOS + Android). Our team has identified the root cause and is working on a fix. This issue is only occurring on mobile devices and for documents needing text fields to be completed. During this time, please utilize Floify eSign desktop version for filling out documents.
Report: "Full Outage"
Last update### Floify Service Interruption August 28-29, 2023 This incident report details the Floify service interruptions that occurred on August 28th and August 29th, 2023. **All times in Mountain Time** **Issue Summary** From 9:39 AM on August 28th to 2:15 PM on August 29th, Floify experienced some intermittent errors that may have affected end-user activity on the site. In some cases, users received a 504 Gateway timeout error. The root cause was determined to be memory issues with a system job. **Timeline** August 28, 2023 * 9:39 AM - Pagers alerted the Engineering team about an outage * 9:39 AM - Investigation of the root cause began * 9:45 AM - The Floify application automatically restarted and service was restored * 9:45 AM - 6:15 PM Per standard processes, application refreshes were performed. * 5:30 PM - Additional memory was added to the Floify application August 29, 2023 * 8 AM - 12 PM - Engineering identified an issue with a system job that was using a large amount of memory under certain conditions. * 12:15 PM - The underlying data that was causing this inordinate memory usage was corrected. * 2:15 PM The system job successfully completed the workload and the memory issues were resolved. **Root Cause** A system job was processing a significant amount of records. This was causing the application to run low on memory, resulting in latency and timeouts. **Resolution and Recovery** At 9:39 AM on August 28, 2023, monitoring systems alerted our engineers who investigated and escalated the issue. For the rest of the day, Engineers were able to alleviate the effect of the issue by performing rolling refreshes both automatically and manually \(where needed\). At 2:15 PM on August 29, 2023, Engineers were able to apply a fix to address the root cause and access to the Floify service was restored. **Corrective and Preventative Measures** The following are actions we are taking to address the underlying causes of the issue and to help prevent recurrence and improve response times: Short term: * Increase system memory \(Completed\) * Perform Floify application refreshes to temporarily alleviate the app issues \(Completed\) * Correct the underlying data that was causing this inordinate memory usage \(Completed\) Longer-term strategies: * Add code changes to the system job to process workload in batches thereby limiting memory usage \(In Progress\) * Add code changes to enforce data validation and prevent this type of issue from occurring \(In Progress\) Floify is committed to continually adapting and improving our technology and processes to prevent service interruptions. We appreciate your patience and again apologize for the impact of this outage. We thank you for your business and continued support. Sincerely, The Floify Team
This incident has been resolved.
While we are investigating the root cause, users may experience latency while using the application.
The outage was resolved as of 9:42am MT. We are continuing to monitor the status and investigating the root cause.
We are investigating this issue.
Report: "Intermittent Degraded Performance"
Last update### Floify Service Interruption August 15, 2023 to August 16, 2023 This incident report details the intermittent issues with Floify that occurred August 15-16, 2023. **Issue Summary** From 1 PM MT on August 15 to 4:21 PM MT on August 16, Floify experienced intermittent database exceptions causing some loan creation and conversion interruptions. The root causes were determined to be a database query performance issue and an AWS database failover incident. **Timeline \(all times Mountain Time\)** August 15, 2023 * 1:00 PM Pagers alerted the Engineering team about abnormal database exceptions * 1:05 PM Hibernate exceptions were identified in the logs. * 1:10 PM Investigation of the root cause began in partnership with AWS. * 1:30 PM Engineering was able to identify and cancel a few blocking SQL queries to improve stabilization. * 3:30 PM Pagers alerted Engineering about abnormal database exceptions * 3:44 PM and 5:04 PM Rolling refreshes were performed as part of standard procedures. August 16, 2023 * 8 AM - 12 PM An issue was identified with the database locking mechanism used in one query. Code updates to fix the issue were made. AWS advised to update the database lock configuration parameter that would positively affect query performance and system stability. * 12 PM - 2 PM QA tested and certified the code patch. * 1:55 PM A rolling refresh performed with the code fix and the database parameter update. * 2:00 PM Engineering noticed dynamic database parameter update caused high CPU usage on the writer database instance. * 2:00 PM Engineering team partnered with AWS to address the issue. * 4:05 - 4:16 PM An automatic database failover was performed by AWS which caused loss of connectivity to the database and thereby intermittent issues in the Floify application. AWS confirmed their platform was the cause of this issue. * 4:21 PM A rolling refresh was performed on to refresh database connections. After monitoring database usage, the incident has been resolved. **Root Cause** A database SELECT FOR UPDATE mechanism was used in Floify code to synchronize changes to the loan custom field data. However, due to an issue with MySQL 5.x this query was holding a significant number of gap locks under certain load conditions. This locking caused concurrency issues with multiple transactions and led to database query timeouts. The first occurrence of this issue was detected on August 15, 2023. **Resolution and recovery** At 1:00 PM on August 15, 2023, monitoring systems alerted our engineers who investigated and escalated the issue. For the rest of the day Engineers were able to alleviate the effect of the issue by performing rolling refreshes of the Floify application and selectively terminating certain locked queries. The Engineering team quickly contacted AWS to troubleshoot the issue and develop a fix. At 4:21 PM on August 16, 2023, Engineers were able to deploy fixes to address the root cause and access to the Floify service was restored. **Corrective and Preventative Measures** The following are actions we are taking to address the underlying causes of the issue and to help prevent recurrence and improve response times: Short term: * Manually perform Floify application refreshes to temporarily resolve the database locking issues \(Completed\) * Deploy code changes and database configuration updates to resolved the issue \(Completed\) Longer term strategies: * Explore database strategies in partnership with AWS to identify hot spots and opportunities to scale to mitigate downtime \(In Progress\) Floify is committed to continually adapting and improving our technology and process to prevent service interruptions. We appreciate your patience and again apologize for the impact of this issue. We thank you for your business and continued support. Sincerely, The Floify Team
Monitoring is complete with no issues found since the fix was implementation on Aug 16th. This incident is resolved.
We will be implementing another fix shortly. During this time, user experience will not be impacted, however, some latency may occur.
A fix has been implemented and we're monitoring the results.
We are currently investigating this issue.
Report: "Floify eSign "Other" Signer Type Failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Floify eSign is failing to send when an additional signer with type "Other" is added to the document. The issue is isolated only when using the "Other" type. For documents impacted by this failure, an error would have prevented the document to sent and would need to be resent or recreated by the LO. These documents were not sent. Our developers have identified the issue which is currently being tested. We will update this status page once testing has concluded and when it will be moved to Production.
Report: "Interruption when starting a loan application"
Last updateThe root cause of the ongoing issue has been identified as a lock in the database. We have a short-term solution for any such occurrences. Our team is actively working on implementing a comprehensive solution to permanently resolve this problem. While we work towards a full resolution, we will continue to closely monitor the situation and address any occurrences with urgency.
We are continuing to monitor for any further issues.
The reported issue has been resolved. Nevertheless, our team is actively monitoring the situation to identify its root cause. Once we have determined and fixed the underlying issue, we will provide further updates on the status page.
We are currently investigating this issue.
Report: "System Logging Errors Detected"
Last updateThis incident has been resolved.
We have detected system logging errors and will be doing a rolling refresh to the application. During this time, user experience will not be impacted, however, some latency may occur.
Report: "System Logging Errors Detected"
Last updateThis incident has been resolved.
We have detected system logging errors and will be doing a rolling refresh to the application. During this time, user experience will not be impacted, however, some latency may occur.