Historical record of incidents for LoanMAPS
Report: "Issue When Saving Loan Application (with Co-Borrower)"
Last updateThis incident has been resolved.
A fix has been deployed and we are monitoring to ensure there are no further issues.
Deployment with the update is in progress
A fix has been implemented and we are preparing to deploy the change
This issue appears to be isolated to co-borrowers and page four data. We are working on a correction now.
We're investigating an issue with saving in the loan application interface
Report: "Icon Display Issues"
Last updateResolving, as we have routed all icon loading to our internal servers.
We've disabled loading icons from our icon provider and routed requests internally. Once our icon provider's service has been restored, we will switch back over to them.
We have identified an issue with icons not loading and confirmed it is an issue with our icon provider's service. They are currently investigating.
Report: "SSL Certificate Issue"
Last updateWe experienced an issue today where an expired SSL certificate was being used on one of our load balanced web servers. This was causing intermittent certificate expiration messages for end users. We monitor all load balanced web servers for expiring certificates, however we found today that one of the servers had an unused configuration section that was somehow being referenced when requesting the certificate for that server. By removing this unused configuration, the server started referencing the correct certificate from the correct configuration. Because we removed the unused configuration, this is not an issue we expect to happen again at the next certificate renewal date (in 2022).
Report: "Icon Display Issues"
Last updateWe've disabled loading icons from our icon provider and routed requests internally. Once our icon provider's service has been restored, we will switch back over to them.
We have identified an issue with icons not loading and confirmed it is an issue with our icon provider's service. They are currently investigating.
Report: "Degraded Performance: Database Server Under High Load"
Last updateThis issue was fixed in production at 3:00 PM. Monitoring for the past hour shows that it has been resolved.
We have finished updating the query and are deploying it into the production environment now.
We are currently working on testing a possible fix for this issue and will have more information to share shortly.
Clarification to the previous update: The query in question is causing overall high load on the database server, which is impacting other areas of the system.
We identified an issue with one of our database queries that loads the "Not Assigned" section of the "Pre-Close" tab on the dashboard. We are working to improve the performance of this query now.
We are aware of an issue causing our database server to be under high load. This is impacting the general performance of the system. This issue is being actively investigated.
Report: "Degraded Server Availability"
Last updateThis incident has been resolved.
One of the production servers we run on is currently experiencing issues. At this time, we have that server offline to prevent any further issues for our users. This may cause a slightly slower experience. We will be performing maintenance tonight to restore full functionality.
Report: "Server Infrastructure Outage"
Last updateThe infrastructure that we run on experienced an outage, taking down our production and monitoring systems. The infrastructure was down for approximately 30 minutes, and it took about 15 minutes to restore our production environment functionality, as all servers were started from a cold state.
Report: "Elevated Level of Memory"
Last updateThis incident has been resolved.
The elevated memory level issue has been resolved. We have not seen an elevated memory level today.
Memory has been holding stable this morning without any new incidents. Will continue to monitor for the rest of the day. Another update will be provided in the afternoon.
The update has been deployed, and we are now closely monitoring the application.
After experiencing some issues earlier in the day, the system has been stable this afternoon. We currently have a scheduled maintenance event tonight to address some of these issues, and will update this incident tomorrow based on the results of that maintenance.
We're experiencing an elevated level of memory usage on the primary web application. Our team is investigating this and updates will be provided as information is available. This elevated level of memory may cause intermittent instability within the application, and our team is closely monitoring the situation while a solution is being crafted.
Report: "Elevated Errors When Accessing LoanMAPS.com"
Last updateWe had an issue with elevated memory that caused some system instability. The issue was resolved quickly and stability has returned back to normal.
Report: "Elevated Errors When Accessing LoanMAPS.com"
Last updateThere was an elevated level of errors this afternoon caused by a database configuration issue. This issue was identified and solved, and scheduled maintenance has been scheduled to perform additional work related to this incident.
Report: "Elevated Errors When Accessing LoanMAPS.com"
Last updateThis morning, reports of users experiencing an elevated level of errors were received. The underlying cause of this was a disk capacity issue on our database server. This was quickly corrected, and we will be performing maintenance later tonight to improve the capacity issue.