Total Expert

Is Total Expert Down Right Now? Check if there is a current outage ongoing.

Total Expert is currently Operational

Last checked from Total Expert's official status page

Historical record of incidents for Total Expert

Report: "Journey access settings are not being reflected appropriately"

Last update
resolved

This incident has been resolved.

monitoring

We have implemented a fix in our Production environment, and will continue to monitor the situation over the weekend.

identified

At this time, we are confident we have discovered the most likely cause. We are actively working on a resolution.

identified

We have identified a likely cause, and will be implementing a fix.

investigating

We are investigating an issue with some users not being given access to Journeys when it looks like, according to access settings, the user should have access.

Report: "Journey access settings are not being reflected appropriately"

Last update
Resolved

This incident has been resolved.

Monitoring

We have implemented a fix in our Production environment, and will continue to monitor the situation over the weekend.

Update

At this time, we are confident we have discovered the most likely cause. We are actively working on a resolution.

Identified

We have identified a likely cause, and will be implementing a fix.

Investigating

We are investigating an issue with some users not being given access to Journeys when it looks like, according to access settings, the user should have access.

Report: "Errors Sending & Receiving SMS"

Last update
resolved

The outage has been repaired at our 3rd party provider and we can confirm there are no more errors on our end. Marking this incident as resolved, there will be no further updates.

identified

Portions of TotalExpert's SMS features rely on a 3rd party communications provider. This provider is currently experiencing an outage and it is impacting our ability to send and receive SMS. We will update this incident within the hour.

Report: "Meta (Facebook & Instagram) Integration Failures"

Last update
resolved

We have not seen any further issues and are marking this incident resolved.

monitoring

A fix has been implemented and we are no longer seeing failures. We will continue to monitor this for the next hour.

identified

We are currently experiencing difficulties with our Meta (Facebook & Instagram) integration in the CRM. This includes sharing posts and connecting accounts. We will provide an update within an hour.

Report: "Errors when creating and updating contacts through the UI"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been deployed. Performance appears to be back to normal. We are continuing to monitor. We'll provide an update in one hour.

identified

We have identified the issue and are deploying a fix.

investigating

We are aware that some customers are experiencing issues with creating contacts via the UI. We are currently investigating this issue and will provide an update within an hour.

Report: "Users Unable to Access the Platform via SSO"

Last update
resolved

We have confirmed no further issues and are closing this incident.

monitoring

The issue has been resolved, and SSO login functionality has been fully restored. We are closely monitoring the system to ensure stability.

Report: "Users Unable to Access the Platform via SSO"

Last update
resolved

The issue affecting Single Sign-On (SSO) has been fully resolved.

monitoring

Our team has successfully implemented a fix, and we have verified that Single Sign-On (SSO) functionality has been restored. We will continue monitoring the system to ensure no further issues occur.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating reports of users being unable to log in using Single Sign-On (SSO). This only affects user using SSO.

Report: "Intermittent failure of Encompass imports"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor progress of imports

monitoring

We are continuing to monitor progress of imports

monitoring

The Encompass API began processing imports again at 7:45AM ET and we are monitoring progress of imports. We are analyzing to determine if any data was missed and evaluating options for recovery.

identified

The Encompass API is returning errors for a subset of organizations. Scheduled data imports are failing for these organizations.

Report: "Degraded Functionality: Focused Views are not loading"

Last update
resolved

We have confirmed that both standard and custom focused views are now loading properly. Thank you for your patience while we worked to resolve this issue.

monitoring

We have identified a resolution and are in the process of deploying a fix. We will provide an update within the hour once the issue has been confirmed as resolved.

investigating

We are currently experiencing an issue affecting the loading of both standard and custom focused views. Our team is actively investigating its root cause to develop a resolution.

Report: "Degraded Functionality: Journey Onboarding is delayed"

Last update
resolved

The incident has been resolved and we anticipate any delays will be cleared within the next 2 to 3 hours.

monitoring

We are currently experiencing a delay in onboarding journey campaigns for contacts. Our team is actively working to resolve the issue, and we will provide an update within the next hour as well as notify you once the issue has been resolved.

Report: "Occasional web errors appearing in the Platform"

Last update
resolved

This incident has been resolved.

monitoring

We have applied another fix and are updating the components to be operational. We will continue to monitor the situation and provide another update in 2 hours.

monitoring

We have applied a change and we're monitoring the situation. We will provide another update in an hour.

identified

We are aware of occasional "HTTP 500" errors appearing to some users in the platform. We have identified the issue and are working to resolve. We will provide an update in 15minutes.

Report: "Vanity site domains not rendering SPS/LCA sites correctly"

Last update
resolved

Vanity site domains are now rendering correctly. This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating an issue where customers using vanity domains for Lead Capture Apps and Single Property Sites may experience their sites not loading as expected.

Report: "Degraded Functionality: Delay in Automated Marketing Emails"

Last update
resolved

The incident has been resolved, and automated marketing emails are now processing normally.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently experiencing a delay in the delivery of Automated Marketing emails. Our team is aware of the issue and is working diligently to resolve it as soon as possible. We will provide an update within the next hour with the latest status or upon resolution.

Report: "Vanity site domains not rendering SPS/LCA sites correctly"

Last update
resolved

Vanity site domains are now rendering correctly. This incident has been resolved.

investigating

We are currently investigating an issue where customers using vanity domains for Lead Capture Apps and Single Property Sites may experience their sites not loading as expected.

Report: "Delayed Processing of Data"

Last update
resolved

Delayed processing of contacts and other data created or updated between approximately 2:05pm and 2:45pm CDT

Report: "Inability to access saved media"

Last update
resolved

We have confirmed the service and functionality have been restored.

identified

We have identified the problem and are working on implementing a fix.

investigating

We are currently investigating an issue related to users not being able to access saved media items such as PDF files. This is also impacting the ability of users to generate and view new content.

Report: "Event Processing Delayed"

Last update
resolved

This incident has been resolved.

monitoring

The majority of our backlog has been processed. The rest will process over night and we will provide another update in the morning.

identified

We are still working through a backlog of messages. We expect to have the majority of the backlog caught up within the next 2hours and we will provide another update at that time.

identified

We are working through a backlog of messages and will provide another update in 2 hours.

identified

We have identified an issue with our event processing. We are working towards a resolution and will update this incident in 2 hours.

Report: "Outbound Notifications Outage"

Last update
resolved

The backlog of events have been processed and we are considering this resolved.

monitoring

A fix has been implemented and we're processing outbound web hooks again. There is a small backlog that we are working through and we will monitor this until we have caught up.

identified

We are testing a fix for the issue. We will provide another update within 3 hours.

identified

Outbound web hooks are currently disabled. We have identified the issue and this only affects customer web hook subscriptions in our event processing. We will provide an update within 3 hours.

Report: "Delay in external data imports"

Last update
resolved

Our queued sync events have caught up and are no longer delayed. Marking this resolved.

monitoring

We have implemented a fix and we expect the queue to finish later today. We will provide an update within 8 hours.

identified

We have identified the issue and are working to resolve. We will provide an update within the hour.

investigating

We are investigating an issue in our sync events for external data imports. This is causing a delay in their processing.

Report: "Standard Reports Delayed Data"

Last update
resolved

This incident has been resolved.

monitoring

The Analytics Dashboards have been fully loaded. We are moving the incident to Monitoring while we finish the Standard Reports. The next update will be in 24hr.

identified

We are in the final steps of the data load and close to resolution. We will have an update in 6 hours.

identified

We are continuing to work on the reload of our data, and do not anticipate any issues. We will have the next update by 9am CDT.

identified

We are continuing to work on the reload of our data. We will have another update in 4 hours.

identified

Due to an issue, the Analytics Dashboards and Weekly Standard Reports will only contain data through midnight Friday, 3/8/23. We have identified the issue and are working to reload our data. We will have an update in 4 hours.

Report: "Degraded Performance: Event Processing"

Last update
resolved

We see no remaining issues or concerns and are going to resolve this incident.

monitoring

We are still monitoring the performance of our event processor. We will have another update within 3 hours.

monitoring

Event processor has been repaired and the performance has recovered. We are monitoring the situation will have another update within the hour.

identified

We are experiencing a performance issue with our primary event processor. We have identified the problem and are working to resolve. We expect minor delays while this is being repaired. We will have an update within the hour.

Report: "Degraded Functionality: Email Sender"

Last update
resolved

This issue has been resolved.

investigating

We are currently experiencing an issue that is preventing the recipient list from loading properly when sending an email through the Email Sender page.

Report: "Degraded Functionality: Email lists are failing to load for come users"

Last update
resolved

This issue has been resolved.

investigating

We've discovered an issue affecting some users who are experiencing difficulties in accessing email lists, particularly when trying to send emails or examine statistics of past emails. We've implemented a temporary workaround, which will be communicated via in-platform messaging. Our team is actively working to fix this issue permanently and we will keep you informed with any further updates.

Report: "Platform UI Issue"

Last update
resolved

From 2:27 PM to 2:40PM CT the Total Expert platform UI was unusable. This issue has been completely resolved and we're giving the all clear.

Report: "Degraded Functionality: New leads are not routing through dialer"

Last update
resolved

This incident has been resolved.

investigating

Currently we are experiencing an issue where new leads are not routing through the dialer via Advanced Lead Management to available agents. We are working on resolving this as soon as possible and will update this page when the issue is resolved or as new updates are available.

Report: "Errors when creating leads and contacts"

Last update
resolved

Total Expert has pushed a fix for this issue. Functionality around the creation, editing, and transfer of leads and contacts has been restored. This incident has been fully resolved.

monitoring

We are continuing to monitor the issue as well as working on a permanent solution.

monitoring

Functionality for lead generation in the platform has returned to normal operating levels. We have temporarily disabled the bulk change ownership feature until further investigation can be performed and feature performance can be improved. The time of impact was from 1:26pm CDT to 4:08pm CDT.

identified

Total Expert has identified the primary cause of the issue. We have mitigated the issue and are continuing to work on a resolution.

investigating

We are continuing to investigate this issue.

investigating

Total Expert is in the process of investigating an issue with creating and updating leads and contacts.

Report: "Degraded Functionality: Users are receiving a 500 error when attempting to generate a PDF of print templates"

Last update
resolved

Currently we are not experiencing issues with the PDF generation and are considering this incident resolved.

investigating

We are currently experiencing errors when users are attempting to generate a PDF of a published print template. The system returns a 500 error and the PDF is not generated. We are working on resolving this issue and will update you when we have more information and when this has been resolved.

Report: "Email sending delays"

Last update
resolved

This incident has been completely resolved since 7:26 PM CT.

monitoring

Delayed email sends are currently processing we are just under half way through the backlog. We expect all delayed emails to be sent within the next two hours. We will provide a final update once all delayed emails have been sent.

identified

We have identified a problem with our email sending system that is delaying the delivery of emails. Emails began being delayed between approximately 10:42 AM CT and 2:42 PM CT. We're actively working to get the delayed emails processed and delivered and will update as soon as we get processing. Emails sent after 2:42 AM CT are unaffected and being delivered normally.

Report: "Degraded Functionality: Leads sent to TE that utilize lead parsing are not being created"

Last update
resolved

This incident has been resolved. As this was an issue identified with the lead parsing only, any leads sent to TE during the outage were received and have been parsed and created as part of the resolution.

identified

The issue has been identified and a fix is being implemented.

investigating

Currently we are experiencing an issue where all Lead Parse emails are not being parsed into new leads in the platform. We are working on this issue and will update the status as new information is available.

Report: "Degraded Functionality: Unsubscribe link in marketing emails is not updating email subscription preferences"

Last update
resolved

A fix has been put in place for the email subscription preferences. This incident has been resolved.

investigating

The customer preference center is currently unable to update email subscription preferences. This applies to all emails sent from all orgs.

Report: "Degraded Functionality: Email list page does not display for some users"

Last update
resolved

This incident has been resolved.

investigating

The /email/list page is not displaying for some users, it loads for a long period of time then either returns a "500 An unexpected error has occurred." screen. Currently this isn't affecting all users.

Report: "Degraded Functionality: User activate/deactivate toggle grayed-out"

Last update
resolved

This incident has been resolved.

investigating

Currently the toggle that allows an admin to activate/deactivate a user is grayed-out. Admins do not have the ability to activate a currently de-activated user when logged into the UI nor can they determine if a user is active or inactive.

Report: "Degraded Functionality: Users unable to invite new co-marketing partners"

Last update
resolved

The incident has been resolved.

investigating

We are experiencing an issue with the create new co-marketing partner window. It is not allowing users to continue to the next stage in the process after the initial screen.

Report: "Degraded Functionality: Some users receiving multiple notifications for each new lead"

Last update
resolved

Between the hours of 8:45AM CT and 12:45PM CT on 6/14/2023 some users received duplicate email notifications for new leads created via the API. The issue has been resolved.

Report: "Total Expert user interface styling issues"

Last update
resolved

The issue with Total Expert's user interface has been resolved.

monitoring

Users may be experiencing design and formatting issues with the Total Expert user interface. We have deployed a fix and are monitoring the situation.

Report: "Missing activity on activity stream"

Last update
resolved

Total Expert has successfully resolved Monday’s issue with imports and email sends which resulted in unscheduled database maintenance and placing background processing on hold. The fix included the creation of a new activity streams table which substantially lowers any risk of recurrence.  While investigating the issue, the team placed a proactive hold on processing, which resulted in queues for imports and emails for 24 hours. The hold on processing was lifted Monday evening and the queues were successfully processed in sequential order. We are continuing to process activity logs and tasks; this work could range from the next few hours up to two days. Please note that Contact list updates may be delayed until we complete processing the events backlog. Our team is tightly managing this final stage to mitigate impact on the overall performance of the platform.   We apologize for the disruption and thank you for your patience as we worked through the issue. Our team will continue to post updates to the Total Expert status page.

monitoring

All Functionally has been restored. Emergency database maintenance has now been completed for everything but Activity Logs and Tasks. Activity Logs and Tasks are being actively restored and will be appearing over the next few hours and days. Contact list updates may be delayed until we finish processing our backlog of events. We will continue to monitor the progress and provide an update in the morning.

identified

We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Paused services include journeys, imports, and email sending. Some pages may display stale data during this window. We are still on schedule to have paused services operational by 7pm Central.

identified

We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Paused services include journeys, imports, and email sending. Some pages may display stale data during this window. We expect to have paused services operational by 7pm Central. We will provide another update in 1 hour.

identified

We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Services impacted include journeys, imports, and email sending. Some pages may display stale data during this window. We will provide another update in 1 hour.

identified

We are continuing to perform emergency database maintenance. We will provide another update in 1 hour.

identified

We are currently performing emergency database maintenance.

investigating

We are preparing to perform emergency database maintenance in order to resolve the reported issue.

investigating

Some users may be experiencing an issue where system activities are not populating in the user's activity stream.

Report: "Difficulties Logging into TotalExpert"

Last update
resolved

This incident has been full resolved as of 11:27 AM CT.

investigating

Some users are reporting difficulties with logging into the platform and navigating to various areas.

Report: "Difficulties Logging into TotalExpert"

Last update
resolved

This incident has been full resolved as of 10:46 AM CT.

investigating

Some users are reporting difficulties with logging into the platform and navigating to various areas.

Report: "Total Expert access issues for certain geographic areas"

Last update
resolved

Our upstream provider has resolved the incident. We can confirm there are no longer issues connecting to Total Expert.

monitoring

We are receiving reports of access being restored. We will continue to monitor the situation.

identified

We are continuing to work with our upstream providers to monitor this situation. We will update this page with more information as we learn more.

identified

Users in certain geographic areas and using certain internet providers may experience issues accessing Total Expert. This appears to be a widespread internet issue. All automation is unaffected and will continue to operate as expected.

Report: "Print Media generation failures"

Last update
resolved

This incident has been resolved since 11:05 CT. We're giving the all clear.

monitoring

We have identified the issue and implemented a fix. We have confirmed all print media is rendering properly. We will continue to monitor the recovery and give an all clear soon.

investigating

We are still actively investigating this issue. We'll update as soon as we know more.

investigating

We are currently investigating an issue with print media generation. Next update in 30 minutes.

Report: "Difficulties Logging into TotalExpert"

Last update
resolved

This incident has been full resolved since 9:32 PM CT

monitoring

AWS has resolved the issue. Response times have returned to normal. We're going to continue to monitor the issue before we give a final all clear.

identified

We're continuing to work with our hosting partner AWS on the increased latency issue. We will provide another update once we have more information.

identified

We have found an issue with AWS latency issues. We are working with AWS to gather more details.

investigating

Some users are reporting difficulties with logging into the platform and navigating to various areas.

Report: "Degraded Database Performance"

Last update
resolved

This maintenance has been completed and we have confirmed performance has been restored. Marking this as resolved.

monitoring

Emergency maintenance has been successful, we are going to monitor the database for the next 24hours.

identified

Services have been restored, and we are taking final steps in our emergency database maintenance. We will provide another update within 30 minutes.

identified

We are performing emergency database maintenance.

Report: "Static Content was missing"

Last update
resolved

Static Content was missing for 15 minutes and this has been resolved.

Report: "Delayed Contact List updates in the Web Application"

Last update
resolved

This incident has been resolved.

identified

We are continuing to monitor the delayed contact list updates and working on a resolution.

identified

We are continuing to monitor the delayed contact list updates and working through a resolution.

identified

For some customers contact list updates are being delayed. We are actively monitoring the situation and will provide further updates throughout the day.

Report: "Platform Unresponsive due to AWS us-east-2 downtime"

Last update
resolved

This incident has been resolved. Our platform experienced a service outage from 12:08 pm CT to 12:24 pm CT, since 12:25 pm CT Total Expert platform has returned to a fully functional state.

monitoring

Our service provider Amazon Web Services (AWS) experienced an outage around 12 PM Central due to a power outage at a data center. Amazon has started to resolve the issue. We are actively monitoring the Total Expert application to ensure functionality is fully restored and will provide an update in the next hour.

identified

AWS is investigating network connectivity issues for some instances and increased error rates and latencies for the EC2 APIs within the US-EAST-2 Region.

Report: "Delayed Contact List updates in the Web Application"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor the delayed contact list updates. The processing time is continuing to decrease.

monitoring

We are still monitoring the delayed contact list updates; however, processing time has decreased from 50mins to 20mins.

investigating

For some customers contact list updates are delayed due to the increased Holiday traffic. We are actively monitoring the situation and will provide further updates throughout the day.

Report: "Delayed Event Processing"

Last update
resolved

We are resolving this incident. We have fully delivered our backlog of events, and there are no longer any outstanding Journey delays from the platform.

monitoring

We have implemented our solution and are monitoring our backlog. Our next update will be within 24 hours.

identified

We are still tracking some event processing delays while we catch up on our backlog of messages. We will provide another update within the next 2 hours.

identified

We are tracking and are seeing a few hours of delay. We have identified the issue and will provide an update within the next 2 hours.

Report: "Unable to create and edit emails, email templates and static media"

Last update
resolved

We have identified the issue and taken steps to resolve it. The ability to create and edit emails, email templates and static media has been restored. We are closing this incident.

investigating

We are tracking an issue impacting the ability to create and edit emails, email templates and static media. We are working to resolve this issue and will provide an update within the next 2 hours or when it is resolved.

Report: "Delayed Event Processing"

Last update
resolved

We are resolving this incident. We have fully delivered our backlog of events, and there are no longer any outstanding Journey delays from the platform.

monitoring

We have applied a fix to our delayed Journeys and are monitoring the backlog. We will provide another update within the next 2 hours.

identified

We are tracking a large volume of events impacting Journeys and are seeing a few hours of delay. We have identified the issue and will provide an update within the next 2 hours.

Report: "Delayed Event Processing"

Last update
resolved

We are resolving this incident. We have fully delivered our backlog of events, and there are no longer any outstanding Journey delays from the platform.

monitoring

We have processed the majority of our Journey backlog, but we still have some events remaining. Our next update will be within 12 hours.

monitoring

We are still tracking some event processing delays within our Journeys. We will provide another update within the next 3 hours.

monitoring

We are still tracking some event processing delays while we catch up on our backlog of messages. We will provide another update within the next 3 hours

monitoring

We are tracking a large volume of events due to the upcoming holiday. We are closely monitoring the situation and will provide an update within the next 3 hours.

Report: "Inaccurate Email Delivery Statistics"

Last update
resolved

This incident has been resolved

monitoring

We are starting to see resolution of this issue and will continue to monitor it. We will provide another update in 24 hours as we verify that this is fully resolved.

identified

We are continuing to work with our upstream email statistics provider on a solution to this issue. We will provide another update by Monday morning or as soon as we have more information. We appreciate your patience as we await the resolution of this issue.

identified

We are still working with our upstream email statistics provider on a solution. We will provide another update within 24hours.

identified

We are continuing to work with our upstream provider on a solution. We will provide another update before 9am CDT.

identified

Customers may experience inaccurate email statistics regarding clicks, opens, sends, etc. Email delivery is not impacted. We are working with our upstream email provider and will provide more updates as we receive more information. We will provide another update within 2 hours.