Historical record of incidents for Total Expert
Report: "Journey access settings are not being reflected appropriately"
Last updateThis incident has been resolved.
We have implemented a fix in our Production environment, and will continue to monitor the situation over the weekend.
At this time, we are confident we have discovered the most likely cause. We are actively working on a resolution.
We have identified a likely cause, and will be implementing a fix.
We are investigating an issue with some users not being given access to Journeys when it looks like, according to access settings, the user should have access.
Report: "Journey access settings are not being reflected appropriately"
Last updateThis incident has been resolved.
We have implemented a fix in our Production environment, and will continue to monitor the situation over the weekend.
At this time, we are confident we have discovered the most likely cause. We are actively working on a resolution.
We have identified a likely cause, and will be implementing a fix.
We are investigating an issue with some users not being given access to Journeys when it looks like, according to access settings, the user should have access.
Report: "Errors Sending & Receiving SMS"
Last updateThe outage has been repaired at our 3rd party provider and we can confirm there are no more errors on our end. Marking this incident as resolved, there will be no further updates.
Portions of TotalExpert's SMS features rely on a 3rd party communications provider. This provider is currently experiencing an outage and it is impacting our ability to send and receive SMS. We will update this incident within the hour.
Report: "Meta (Facebook & Instagram) Integration Failures"
Last updateWe have not seen any further issues and are marking this incident resolved.
A fix has been implemented and we are no longer seeing failures. We will continue to monitor this for the next hour.
We are currently experiencing difficulties with our Meta (Facebook & Instagram) integration in the CRM. This includes sharing posts and connecting accounts. We will provide an update within an hour.
Report: "Errors when creating and updating contacts through the UI"
Last updateThis incident has been resolved.
A fix has been deployed. Performance appears to be back to normal. We are continuing to monitor. We'll provide an update in one hour.
We have identified the issue and are deploying a fix.
We are aware that some customers are experiencing issues with creating contacts via the UI. We are currently investigating this issue and will provide an update within an hour.
Report: "Users Unable to Access the Platform via SSO"
Last updateWe have confirmed no further issues and are closing this incident.
The issue has been resolved, and SSO login functionality has been fully restored. We are closely monitoring the system to ensure stability.
Report: "Users Unable to Access the Platform via SSO"
Last updateThe issue affecting Single Sign-On (SSO) has been fully resolved.
Our team has successfully implemented a fix, and we have verified that Single Sign-On (SSO) functionality has been restored. We will continue monitoring the system to ensure no further issues occur.
The issue has been identified and a fix is being implemented.
We are currently investigating reports of users being unable to log in using Single Sign-On (SSO). This only affects user using SSO.
Report: "Intermittent failure of Encompass imports"
Last updateThis incident has been resolved.
We are continuing to monitor progress of imports
We are continuing to monitor progress of imports
The Encompass API began processing imports again at 7:45AM ET and we are monitoring progress of imports. We are analyzing to determine if any data was missed and evaluating options for recovery.
The Encompass API is returning errors for a subset of organizations. Scheduled data imports are failing for these organizations.
Report: "Degraded Functionality: Focused Views are not loading"
Last updateWe have confirmed that both standard and custom focused views are now loading properly. Thank you for your patience while we worked to resolve this issue.
We have identified a resolution and are in the process of deploying a fix. We will provide an update within the hour once the issue has been confirmed as resolved.
We are currently experiencing an issue affecting the loading of both standard and custom focused views. Our team is actively investigating its root cause to develop a resolution.
Report: "Degraded Functionality: Journey Onboarding is delayed"
Last updateThe incident has been resolved and we anticipate any delays will be cleared within the next 2 to 3 hours.
We are currently experiencing a delay in onboarding journey campaigns for contacts. Our team is actively working to resolve the issue, and we will provide an update within the next hour as well as notify you once the issue has been resolved.
Report: "Occasional web errors appearing in the Platform"
Last updateThis incident has been resolved.
We have applied another fix and are updating the components to be operational. We will continue to monitor the situation and provide another update in 2 hours.
We have applied a change and we're monitoring the situation. We will provide another update in an hour.
We are aware of occasional "HTTP 500" errors appearing to some users in the platform. We have identified the issue and are working to resolve. We will provide an update in 15minutes.
Report: "Vanity site domains not rendering SPS/LCA sites correctly"
Last updateVanity site domains are now rendering correctly. This incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating an issue where customers using vanity domains for Lead Capture Apps and Single Property Sites may experience their sites not loading as expected.
Report: "Degraded Functionality: Delay in Automated Marketing Emails"
Last updateThe incident has been resolved, and automated marketing emails are now processing normally.
The issue has been identified and a fix is being implemented.
We are currently experiencing a delay in the delivery of Automated Marketing emails. Our team is aware of the issue and is working diligently to resolve it as soon as possible. We will provide an update within the next hour with the latest status or upon resolution.
Report: "Vanity site domains not rendering SPS/LCA sites correctly"
Last updateVanity site domains are now rendering correctly. This incident has been resolved.
We are currently investigating an issue where customers using vanity domains for Lead Capture Apps and Single Property Sites may experience their sites not loading as expected.
Report: "Delayed Processing of Data"
Last updateDelayed processing of contacts and other data created or updated between approximately 2:05pm and 2:45pm CDT
Report: "Inability to access saved media"
Last updateWe have confirmed the service and functionality have been restored.
We have identified the problem and are working on implementing a fix.
We are currently investigating an issue related to users not being able to access saved media items such as PDF files. This is also impacting the ability of users to generate and view new content.
Report: "Event Processing Delayed"
Last updateThis incident has been resolved.
The majority of our backlog has been processed. The rest will process over night and we will provide another update in the morning.
We are still working through a backlog of messages. We expect to have the majority of the backlog caught up within the next 2hours and we will provide another update at that time.
We are working through a backlog of messages and will provide another update in 2 hours.
We have identified an issue with our event processing. We are working towards a resolution and will update this incident in 2 hours.
Report: "Outbound Notifications Outage"
Last updateThe backlog of events have been processed and we are considering this resolved.
A fix has been implemented and we're processing outbound web hooks again. There is a small backlog that we are working through and we will monitor this until we have caught up.
We are testing a fix for the issue. We will provide another update within 3 hours.
Outbound web hooks are currently disabled. We have identified the issue and this only affects customer web hook subscriptions in our event processing. We will provide an update within 3 hours.
Report: "Delay in external data imports"
Last updateOur queued sync events have caught up and are no longer delayed. Marking this resolved.
We have implemented a fix and we expect the queue to finish later today. We will provide an update within 8 hours.
We have identified the issue and are working to resolve. We will provide an update within the hour.
We are investigating an issue in our sync events for external data imports. This is causing a delay in their processing.
Report: "Standard Reports Delayed Data"
Last updateThis incident has been resolved.
The Analytics Dashboards have been fully loaded. We are moving the incident to Monitoring while we finish the Standard Reports. The next update will be in 24hr.
We are in the final steps of the data load and close to resolution. We will have an update in 6 hours.
We are continuing to work on the reload of our data, and do not anticipate any issues. We will have the next update by 9am CDT.
We are continuing to work on the reload of our data. We will have another update in 4 hours.
Due to an issue, the Analytics Dashboards and Weekly Standard Reports will only contain data through midnight Friday, 3/8/23. We have identified the issue and are working to reload our data. We will have an update in 4 hours.
Report: "Degraded Performance: Event Processing"
Last updateWe see no remaining issues or concerns and are going to resolve this incident.
We are still monitoring the performance of our event processor. We will have another update within 3 hours.
Event processor has been repaired and the performance has recovered. We are monitoring the situation will have another update within the hour.
We are experiencing a performance issue with our primary event processor. We have identified the problem and are working to resolve. We expect minor delays while this is being repaired. We will have an update within the hour.
Report: "Degraded Functionality: Email Sender"
Last updateThis issue has been resolved.
We are currently experiencing an issue that is preventing the recipient list from loading properly when sending an email through the Email Sender page.
Report: "Degraded Functionality: Email lists are failing to load for come users"
Last updateThis issue has been resolved.
We've discovered an issue affecting some users who are experiencing difficulties in accessing email lists, particularly when trying to send emails or examine statistics of past emails. We've implemented a temporary workaround, which will be communicated via in-platform messaging. Our team is actively working to fix this issue permanently and we will keep you informed with any further updates.
Report: "Platform UI Issue"
Last updateFrom 2:27 PM to 2:40PM CT the Total Expert platform UI was unusable. This issue has been completely resolved and we're giving the all clear.
Report: "Degraded Functionality: New leads are not routing through dialer"
Last updateThis incident has been resolved.
Currently we are experiencing an issue where new leads are not routing through the dialer via Advanced Lead Management to available agents. We are working on resolving this as soon as possible and will update this page when the issue is resolved or as new updates are available.
Report: "Errors when creating leads and contacts"
Last updateTotal Expert has pushed a fix for this issue. Functionality around the creation, editing, and transfer of leads and contacts has been restored. This incident has been fully resolved.
We are continuing to monitor the issue as well as working on a permanent solution.
Functionality for lead generation in the platform has returned to normal operating levels. We have temporarily disabled the bulk change ownership feature until further investigation can be performed and feature performance can be improved. The time of impact was from 1:26pm CDT to 4:08pm CDT.
Total Expert has identified the primary cause of the issue. We have mitigated the issue and are continuing to work on a resolution.
We are continuing to investigate this issue.
Total Expert is in the process of investigating an issue with creating and updating leads and contacts.
Report: "Degraded Functionality: Users are receiving a 500 error when attempting to generate a PDF of print templates"
Last updateCurrently we are not experiencing issues with the PDF generation and are considering this incident resolved.
We are currently experiencing errors when users are attempting to generate a PDF of a published print template. The system returns a 500 error and the PDF is not generated. We are working on resolving this issue and will update you when we have more information and when this has been resolved.
Report: "Email sending delays"
Last updateThis incident has been completely resolved since 7:26 PM CT.
Delayed email sends are currently processing we are just under half way through the backlog. We expect all delayed emails to be sent within the next two hours. We will provide a final update once all delayed emails have been sent.
We have identified a problem with our email sending system that is delaying the delivery of emails. Emails began being delayed between approximately 10:42 AM CT and 2:42 PM CT. We're actively working to get the delayed emails processed and delivered and will update as soon as we get processing. Emails sent after 2:42 AM CT are unaffected and being delivered normally.
Report: "Degraded Functionality: Leads sent to TE that utilize lead parsing are not being created"
Last updateThis incident has been resolved. As this was an issue identified with the lead parsing only, any leads sent to TE during the outage were received and have been parsed and created as part of the resolution.
The issue has been identified and a fix is being implemented.
Currently we are experiencing an issue where all Lead Parse emails are not being parsed into new leads in the platform. We are working on this issue and will update the status as new information is available.
Report: "Degraded Functionality: Unsubscribe link in marketing emails is not updating email subscription preferences"
Last updateA fix has been put in place for the email subscription preferences. This incident has been resolved.
The customer preference center is currently unable to update email subscription preferences. This applies to all emails sent from all orgs.
Report: "Degraded Functionality: Email list page does not display for some users"
Last updateThis incident has been resolved.
The /email/list page is not displaying for some users, it loads for a long period of time then either returns a "500 An unexpected error has occurred." screen. Currently this isn't affecting all users.
Report: "Degraded Functionality: User activate/deactivate toggle grayed-out"
Last updateThis incident has been resolved.
Currently the toggle that allows an admin to activate/deactivate a user is grayed-out. Admins do not have the ability to activate a currently de-activated user when logged into the UI nor can they determine if a user is active or inactive.
Report: "Degraded Functionality: Users unable to invite new co-marketing partners"
Last updateThe incident has been resolved.
We are experiencing an issue with the create new co-marketing partner window. It is not allowing users to continue to the next stage in the process after the initial screen.
Report: "Degraded Functionality: Some users receiving multiple notifications for each new lead"
Last updateBetween the hours of 8:45AM CT and 12:45PM CT on 6/14/2023 some users received duplicate email notifications for new leads created via the API. The issue has been resolved.
Report: "Total Expert user interface styling issues"
Last updateThe issue with Total Expert's user interface has been resolved.
Users may be experiencing design and formatting issues with the Total Expert user interface. We have deployed a fix and are monitoring the situation.
Report: "Missing activity on activity stream"
Last updateTotal Expert has successfully resolved Monday’s issue with imports and email sends which resulted in unscheduled database maintenance and placing background processing on hold. The fix included the creation of a new activity streams table which substantially lowers any risk of recurrence. While investigating the issue, the team placed a proactive hold on processing, which resulted in queues for imports and emails for 24 hours. The hold on processing was lifted Monday evening and the queues were successfully processed in sequential order. We are continuing to process activity logs and tasks; this work could range from the next few hours up to two days. Please note that Contact list updates may be delayed until we complete processing the events backlog. Our team is tightly managing this final stage to mitigate impact on the overall performance of the platform. We apologize for the disruption and thank you for your patience as we worked through the issue. Our team will continue to post updates to the Total Expert status page.
All Functionally has been restored. Emergency database maintenance has now been completed for everything but Activity Logs and Tasks. Activity Logs and Tasks are being actively restored and will be appearing over the next few hours and days. Contact list updates may be delayed until we finish processing our backlog of events. We will continue to monitor the progress and provide an update in the morning.
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Paused services include journeys, imports, and email sending. Some pages may display stale data during this window. We are still on schedule to have paused services operational by 7pm Central.
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Paused services include journeys, imports, and email sending. Some pages may display stale data during this window. We expect to have paused services operational by 7pm Central. We will provide another update in 1 hour.
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Services impacted include journeys, imports, and email sending. Some pages may display stale data during this window. We will provide another update in 1 hour.
We are continuing to perform emergency database maintenance. We will provide another update in 1 hour.
We are currently performing emergency database maintenance.
We are preparing to perform emergency database maintenance in order to resolve the reported issue.
Some users may be experiencing an issue where system activities are not populating in the user's activity stream.
Report: "Difficulties Logging into TotalExpert"
Last updateThis incident has been full resolved as of 11:27 AM CT.
Some users are reporting difficulties with logging into the platform and navigating to various areas.
Report: "Difficulties Logging into TotalExpert"
Last updateThis incident has been full resolved as of 10:46 AM CT.
Some users are reporting difficulties with logging into the platform and navigating to various areas.
Report: "Total Expert access issues for certain geographic areas"
Last updateOur upstream provider has resolved the incident. We can confirm there are no longer issues connecting to Total Expert.
We are receiving reports of access being restored. We will continue to monitor the situation.
We are continuing to work with our upstream providers to monitor this situation. We will update this page with more information as we learn more.
Users in certain geographic areas and using certain internet providers may experience issues accessing Total Expert. This appears to be a widespread internet issue. All automation is unaffected and will continue to operate as expected.
Report: "Print Media generation failures"
Last updateThis incident has been resolved since 11:05 CT. We're giving the all clear.
We have identified the issue and implemented a fix. We have confirmed all print media is rendering properly. We will continue to monitor the recovery and give an all clear soon.
We are still actively investigating this issue. We'll update as soon as we know more.
We are currently investigating an issue with print media generation. Next update in 30 minutes.
Report: "Difficulties Logging into TotalExpert"
Last updateThis incident has been full resolved since 9:32 PM CT
AWS has resolved the issue. Response times have returned to normal. We're going to continue to monitor the issue before we give a final all clear.
We're continuing to work with our hosting partner AWS on the increased latency issue. We will provide another update once we have more information.
We have found an issue with AWS latency issues. We are working with AWS to gather more details.
Some users are reporting difficulties with logging into the platform and navigating to various areas.
Report: "Degraded Database Performance"
Last updateThis maintenance has been completed and we have confirmed performance has been restored. Marking this as resolved.
Emergency maintenance has been successful, we are going to monitor the database for the next 24hours.
Services have been restored, and we are taking final steps in our emergency database maintenance. We will provide another update within 30 minutes.
We are performing emergency database maintenance.
Report: "Static Content was missing"
Last updateStatic Content was missing for 15 minutes and this has been resolved.
Report: "Delayed Contact List updates in the Web Application"
Last updateThis incident has been resolved.
We are continuing to monitor the delayed contact list updates and working on a resolution.
We are continuing to monitor the delayed contact list updates and working through a resolution.
For some customers contact list updates are being delayed. We are actively monitoring the situation and will provide further updates throughout the day.
Report: "Platform Unresponsive due to AWS us-east-2 downtime"
Last updateThis incident has been resolved. Our platform experienced a service outage from 12:08 pm CT to 12:24 pm CT, since 12:25 pm CT Total Expert platform has returned to a fully functional state.
Our service provider Amazon Web Services (AWS) experienced an outage around 12 PM Central due to a power outage at a data center. Amazon has started to resolve the issue. We are actively monitoring the Total Expert application to ensure functionality is fully restored and will provide an update in the next hour.
AWS is investigating network connectivity issues for some instances and increased error rates and latencies for the EC2 APIs within the US-EAST-2 Region.
Report: "Delayed Contact List updates in the Web Application"
Last updateThis incident has been resolved.
We are continuing to monitor the delayed contact list updates. The processing time is continuing to decrease.
We are still monitoring the delayed contact list updates; however, processing time has decreased from 50mins to 20mins.
For some customers contact list updates are delayed due to the increased Holiday traffic. We are actively monitoring the situation and will provide further updates throughout the day.
Report: "Delayed Event Processing"
Last updateWe are resolving this incident. We have fully delivered our backlog of events, and there are no longer any outstanding Journey delays from the platform.
We have implemented our solution and are monitoring our backlog. Our next update will be within 24 hours.
We are still tracking some event processing delays while we catch up on our backlog of messages. We will provide another update within the next 2 hours.
We are tracking and are seeing a few hours of delay. We have identified the issue and will provide an update within the next 2 hours.
Report: "Unable to create and edit emails, email templates and static media"
Last updateWe have identified the issue and taken steps to resolve it. The ability to create and edit emails, email templates and static media has been restored. We are closing this incident.
We are tracking an issue impacting the ability to create and edit emails, email templates and static media. We are working to resolve this issue and will provide an update within the next 2 hours or when it is resolved.
Report: "Delayed Event Processing"
Last updateWe are resolving this incident. We have fully delivered our backlog of events, and there are no longer any outstanding Journey delays from the platform.
We have applied a fix to our delayed Journeys and are monitoring the backlog. We will provide another update within the next 2 hours.
We are tracking a large volume of events impacting Journeys and are seeing a few hours of delay. We have identified the issue and will provide an update within the next 2 hours.
Report: "Delayed Event Processing"
Last updateWe are resolving this incident. We have fully delivered our backlog of events, and there are no longer any outstanding Journey delays from the platform.
We have processed the majority of our Journey backlog, but we still have some events remaining. Our next update will be within 12 hours.
We are still tracking some event processing delays within our Journeys. We will provide another update within the next 3 hours.
We are still tracking some event processing delays while we catch up on our backlog of messages. We will provide another update within the next 3 hours
We are tracking a large volume of events due to the upcoming holiday. We are closely monitoring the situation and will provide an update within the next 3 hours.
Report: "Inaccurate Email Delivery Statistics"
Last updateThis incident has been resolved
We are starting to see resolution of this issue and will continue to monitor it. We will provide another update in 24 hours as we verify that this is fully resolved.
We are continuing to work with our upstream email statistics provider on a solution to this issue. We will provide another update by Monday morning or as soon as we have more information. We appreciate your patience as we await the resolution of this issue.
We are still working with our upstream email statistics provider on a solution. We will provide another update within 24hours.
We are continuing to work with our upstream provider on a solution. We will provide another update before 9am CDT.
Customers may experience inaccurate email statistics regarding clicks, opens, sends, etc. Email delivery is not impacted. We are working with our upstream email provider and will provide more updates as we receive more information. We will provide another update within 2 hours.