Historical record of incidents for Truework
Report: "Elevated error rates during account authentication"
Last updateThis incident is resolved.
The fix has deployed and the problem appears to be resolved. We will continue to monitor the situation and make sure that it's resolved.
We believe we've identified the issue and it should resolve itself shortly.
We are currently investigating an issue occurring during account authentication. This incident impacts logins via SSO, OAuth, username/password, as well as encompass and empower transactions that authenticate on each request.
Report: "Elevated Error Rates"
Last updateWe've concluded our review and believe that all platform behavior should be back to normal.
The underlying issue with our infrastructure provider has been resolved and operations should be mostly back to normal. We will continue monitoring to make sure there aren't any remaining issues.
We are continuing to investigate this issue.
We are aware of an incident impacting various Truework features. This incident stems from an underlying issue with our infrastructure provider. We are monitoring the situation.
Report: "Elevated Error Rate / Increased Latency"
Last updateBetween approximately 16:15 and 16:19 Pacific Time, Truework experienced elevated error rates and increased request latency.
Report: "Elevated Error Rates / Temporary Outage"
Last updateBetween approximately 16:30 and 16:36 Pacific Standard Time, Truework experienced elevated error rates related to an issue with a primary database node. Automated failover took approximately 5 minutes to complete and service was restored.
Report: "Elevated Error Rate on www.truework.com"
Last updateOur hosting provider has resolved the issue and www.truework.com should be fully operational.
Our hosting provider appears to have resolved the underlying issue. We are continuing to monitor the situation to identify any remaining issues.
An issue with our hosting provider has been identified and they are working on a fix.
We are currently investigating an issue impacting www.truework.com
Report: "Elevated Error Rate / Increased Latency"
Last updateThe incident has been resolved.
We've identified the issue and it appears to be resolved. We will continue monitoring.
We're experiencing an elevated level of API errors and increased latency. We are currently looking into the issue.
Report: "Verification Requests not loading on user dashboard"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating the issue.
Report: "Problems affecting our SaaS Application"
Last updateThe incident has been resolved.
A fix has been implemented and has been deployed. We are monitoring to see if the issue has been resolved.
The issue has been identified and we are actively fixing it.
Truework is currently investigating the issue
Report: "Truework web services unavailable"
Last updateAvailability of Truework web services (app.truework.com and the Truework API) was degraded between 6:05am and 6:12am PDT (13:05-13:12 UTC) on Monday 19 October 2020. During this period, requests made by app.truework.com and to the Truework API may have experienced increased latency or failed with a server error.
Report: "API: Verification submission errors"
Last updateThis incident has been resolved.
The issue has been identified and currently prevents request submissions.
Report: "Degraded performance on all systems"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
The team is currently investigating an issue affecting all of our systems. Elevated response times and 504 errors may happen.
Report: "Degraded Service due to AWS outage"
Last updateThis incident has been resolved.
We're seeing recovery in our error rates, however, we'll continue monitoring until AWS closes their incident.
Due to an outage with our cloud services provider, certain operations, specifically the manipulation of PDFs, may be partially or completely unavailable. Other site functions remain operational.
Report: "Elevated Synchronous API Response Times and Time to Verify due to Partner and Cloud Services Outage"
Last updateAll components are functioning normally, and we are now operating at full capacity.
We have resolved issues with our payroll partner and are now waiting for further recovery from other service providers.
Our payroll partner is continuing to have issues, as are a number of other services we rely upon. The root cause of all issues is an outage in a major cloud services provider. We're continuing to monitor the issue and will announce when we have a resolution.
We have temporarily blocked requests to this partner, returning response times to normal, at the cost of reducing hit rate while the partner is down.
We have an issue with one of our payroll partners, causing synchronous requests to have elevated response times. We are working with the partner and monitoring this issue.
Report: "Elevated HTTP response latency"
Last updateThis incident has been resolved.
We are working with our provider to correct the issue and we are seeing recovery. We will continue to closely monitor the situation.
We are continuing our investigation into an infrastructure issue with one of our providers. We’ve mitigated most latency at this point, but are working to identify a root cause before we consider this issue resolved.
We're investigating an increase in latency to all HTTP endpoints.
Report: "Payment error - unable to submit verifications / manage payment methods through the portal"
Last updateOur payment partner saw elevated errors From 20:10–20:54 UTC. This is now resolved.
Our payment provider is seeing recovery. We are continuing to monitor the situation.
We have an issue with our payment provider, causing errors during submissions through the portal, as well as adding/changing payment methods. We are working with the partner and monitoring this issue.
Report: "Payment error - unable to manage payment methods through the portal"
Last updateBetween 1:35PM and 1:53 PM PDT, we had an issue with our payment provider, causing errors during adding/changing payment methods. Our payment provider has confirmed recovery and the Incident is now resolved.
Report: "Encompass customers are receiving an error message saying they have an outdated version of their loan file"
Last updateThis issue has been resolved.
We are continuing to work on a fix for this issue.
What happened: When a Truework verification is ordered, Encompass customers are receiving a server error saying they have an outdated version of their loan file. What caused it: Truework launched the ability to automatically write back the D1C Ref Number for D1C-eligible reports to the Truework GSE Vendor field in Encompass last week. As part of this, Truework writes to the loan object in Encompass when we update this information. How Truework had built this was to write back the entire loan object as is to Encompass and add the D1C Ref Number if present as an extra field. But when Encompass receives the loan object, it thinks that the entire loan file has been updated, even though the information Truework is writing back is the same as what already existed. This was an implementation mistake by Truework. Because Encompass thinks the loan file has changed, when customers save the loan file, they are seeing an error message that they have an outdated version of their loan file. The current workaround for the customer is to close and open Encompass, or log out and log back in to get rid of this error. The Truework order will be placed successfully. What we're doing to fix: Truework is in the process of deploying a change where we will not write to loan object if the report is not D1C-eligible, and will only write back to the specific Truework GSE Vendor field with the D1C Reference Number field if the verification report once completed is D1C-eligible. This will remove the server error that currently shows when a Truework report is ordered. There is nothing the customer needs to do to enable this. Timeframe: Fix will go in by end of day. Please let Truework know if this issue persists beyond end of day.
Report: "Failed Requests"
Last updateBetween 9:35 and 9:43am PDT we experienced some issues which may have caused requests to our API and application to fail. This issue has now been resolved.
Report: "Verifications are slow to resolve"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Elevated error rate"
Last updateOur cloud hosting provider has confirmed the power loss issue is resolved and Truework has confirmed recovery.
Our cloud hosting provider has identified the problem as a temporary loss of power, which has since been restored. We will continue to monitor closely.
Issue seems to have been resolved, we are closely monitoring for any further problems.
We are investigating an elevated error rate in all services.
Report: "Elevated Error Rate"
Last updateThis incident has been resolved. Downtime from 17:33-17:46 UTC.
We're experiencing an elevated level of API errors and are investigating the issue.
Report: "Elevated Error Rate"
Last updateWe were experiencing an elevated level of API errors from 00:08-00:13 UTC. This incident has been resolved.
Report: "Elevated HTTP response latency and errors"
Last updateWe've identified that this issue was caused by an infrastructure issue with one of our providers, which has been resolved, and have confirmed that our services have fully recovered.
We're investigating an increase in latency and error to some HTTP endpoints.
Report: "Verification Requests on Verifier dashboard failing"
Last updateThis Incident has been resolved.
We have identified an issue that may be affecting new verification submissions in the verifier dashboard. We are currently working on a fix.