Historical record of incidents for CallTrackingMetrics
Report: "Intermittent issues with voice services."
Last updateThe incident has been resolved. We will continue to monitor performance. Please reach out if you continue to experience issues with receiving or placing voice calls.
A fix has been implemented and we are monitoring the results.
Our upstream service provider has identified the problem causing the issues with CTM. They are working on a solution. Visit https://status.twilio.com/incidents/5mbfvb8mzhcn for updates.
Our technical partners are diligently working to identify and address the issue. Please visit https://status.twilio.com for more information. We will update you as we have more information.
We are continuing to investigate this issue.
We are aware of intermittent errors affecting your ability to answer calls. Our incident response team is investigating.
Report: "Investigating Potential Issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've identified the cause, and have implemented a fix towards full recovery.
We are aware of intermittent errors occurring on the service. Our incident response team is investigating.
Report: "Difficulty connecting to app.calltrackingmetrics.com"
Last updateAt approximately 1:19 pm EST on November 4, 2024 a brief period of very isolated issues connecting to https://app.calltrackingmetrics.com was experienced by a small subset of our users. Remediation actions were swift and the site recovered to normal operation at 1:26 pm EST.
Report: "Investigating Potential Issue"
Last updateThis incident has been resolved.
We have been working with our upstream provider and they have deployed a resolution to the issue. The issue is being tracked here by Twilio: https://status.twilio.com/ We will continue to post updates as we monitor the recovery.
Our upstream provider has identified the issue and is working on a fix. We are seeing restoration.
We have identified the issue is related to device authentication from our upstream voice provider. It is only affecting softphone and chat connections. We have found that the issue is intermittent and with jwt access tokens and that that refreshing and retrying can get through. We are actively working on a resolution with our provider.
We are currently investigating intermittent reports of customers having difficulty accessing our softphone. Some clients are able to authenticate but others are being affected by an issue with our upstream voice provider. We are actively working with them now
We are currently investigating intermittent reports of customers having difficulty accessing our softphone. This is not affecting all regions. We will report back shortly.
Report: "Investigating Potential Issue"
Last updateThis incident has been resolved.
All affected systems are operational.
Session data is mostly restored. Session data is still delayed in the US-West areas.
We are seeing recovery in most of the affected areas and continue to monitor our systems. Session data is still delayed.
Our Salesforce integration has recovered. Delays in transcriptions, and email and text notifications are delayed but improving. Session data recovery continues.
Our Salesforce integration is delayed but improving. Transcriptions are delayed Email and text notification are delayed but improving. Session Data recovery still continues.
The issue has been identified and a fix is being implemented.
We are aware of issues currently affecting the indexing of tracking session data that might affect call routing that depends on the presence of that data. There will also be delays in this information showing in the call log and delayed indexing of the calls. Our technical parter is working on identifying the cause of the issue.
Report: "Investigating Potential Issue"
Last updateAll issues affecting the normal operation of inbound and outbound calls have been identified and resolved.
Currently, the baseline call functionality has been restored and is operating normally. Users should experience no issues with standard voice calls. Our team is working diligently to resolve any residual issues to minimize any disruption to our users. We will provide another update in 30 minutes or as soon as more information becomes available.
Our technical partners continue investigating a service interruption affecting Twilio Voice services, impacting both inbound and outbound calls, including calls with Flex Services. Customers may experience degraded audio quality or difficulties connecting new calls. Their engineering team is actively working on identifying and resolving the cause. We aim to provide another update within 1 hour or as soon as more information becomes available. - We were seeing some recovery, but want to reflect the true status of the issue.
We have observed increased normal operation of our the softphone and are monitoring performance.
Our technical partners continue to investigate the cause of the issue affecting inbound and outbound calls within CTM.
Our upstream provider has made us aware of issue that are affecting the proper operation of CTM. They have assured us that all available resources are working to resolve the issue in a priority manner. Currently, there is no timeframe for resolution.
Report: "Investigating Potential Issue"
Last updateThis incident has been resolved.
A small number of requests returned 500 errors. The cause for the errors has been identified and stopped. We're currently monitoring and working through a solution to prevent future errors.
We are aware of intermittent errors occurring on the service. Our incident response team is investigating.
Report: "Elevated 500's with Reports"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Investigating Potential Issue with Delayed Transcriptions"
Last updateThis incident has been resolved.
We are aware of intermittent errors occurring with transcriptions. Your patience is appreciated while we look into the cause. We do not anticipate any loss of data; however the completion of the transcriptions might be delayed.
Report: "An issue with some CTM tracking numbers exists."
Last updateThis incident has been resolved.
Our technical partner has identified the cause and deployed a repair. Normal operation of voice calls is being reported. We will continue to monitor our systems.
We are aware of issues with a small subset of CTM tracking numbers in which a caller receives a busy signal and cannot connect. This cause of this issue is with systems that neither CTM nor our Direct Technical Partners control. We ask for your patience while this issue on the broader telecom network is being addressed. We will update via this channel as we receive new information.
Report: "Investigating Potential Issue affecting Logins and Access to Features"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are aware of intermittent errors occurring that affect login and access to features. Our incident response team is investigating.
Report: "Issue with purchasing numbers"
Last updateTracking numbers can be purchased using the inventory look up once again.
We are aware of issues affecting the ability to purchase numbers within our platform. We are working on a resolution that will return the ability to purchase numbers to you. In the interim, you may request a number by visiting https://app.calltrackingmetrics.com/tracking_numbers/buy_new.
Report: "Investigating Issue with Transcriptions"
Last updateThe issues with transcriptions have been resolved. Please report any issues to our Technical Support team at https://www.calltrackingmetrics.com/support.
A fix has been put in place and we are monitoring performance. Transcriptions that did not happen are being retried and should be populated over the course of today.
We are aware of intermittent errors occurring on the service. Our incident response team is working with our transcription provider to solve the issue as quickly as possible. Additional updates will be posted as information is received.
Report: "Issue affecting AskAI"
Last updateServices have been restored and operations are back to normal. Thank you for your patience.
Our technology partner has identified the issue affecting AskAI and has implemented a fix. Normal performance should be recovering if it has not already done so. Please visit https://status.openai.com/ for progress.
We are aware of an issue that may affect AskAI within CTM. Our technology partner is experiencing an outage and is investigating the cause. Please visit https://status.openai.com/ for progress.
Report: "Investigating Issue with our Transcription Service"
Last updateWe observed elevated errors for approximately 1 hour . A process is running to transcribe all impacted recordings and workflows
We are aware of problems with our transcription service. Our incident response team is investigating.
Report: "Intermittent issues marking Smart Dialer calls as completed"
Last updateThis incident has been resolved.
Between 11:34 and 11:42 ET we experienced intermittent failures when marking Smart Dialer calls as completed. This was due to a networking issue with a caching server. The issue is resolved and we are monitoring.
The issue has been identified and a fix has been implemented.
Report: "Delays with recording availability"
Last updateAll Intelligence and Transcriptions that were recoverable have been recovered. Programmable Voice Recording and Intelligence services are operating normally at this time.
New Voice recordings are now operating normally; our upstream carrier is still investigating impact to recordings & transcriptions during the impact window.
Our upstream provider has identified an issue causing some delays in recording availability. They are working to identify the cause.
Report: "CallTrackingMetrics Marketing Site Outage"
Last updateOur vendor has identified and resolved the network issue affecting our marketing site. Our CallTrackingMetrics platform was never affected.
Our vendor is working on a networking issue. Our CallTrackingMetrics platform remains unaffected. To login you can go directly to https://app.calltrackingmetrics.com/
Our marketing site is experiencing an outage. Our CallTrackingMetrics platform is unaffected. To login you can go directly to https://app.calltrackingmetrics.com/
Report: "Investigating Potential Issue with Elevated 502 errors."
Last updateServices have been restored and operations are back to normal. Thank you for your patience.
Root cause has been identified and mitigation steps have been implemented. We are still working on a full resolution and teams are monitoring error rates as they continue to improve.
We are aware of intermittent errors occurring on the service. Our incident response team is investigating.
Report: "Investigating Potential Issue - Elevated 500 errors"
Last updateServices have been restored and operations are back to normal. Thank you for your patience.
Root cause has been identified and mitigation steps have been implemented. We are still working on a full resolution and teams are monitoring error rates as they continue to improve.
We are aware of intermittent errors occurring on the service. Our incident response team is investigating.
Report: "Issue with queued calls"
Last updateOur technical partner has resolved the issue with the queueing of calls.
Our Technical Partner has informed us of an issue which is causing calls to terminate abruptly within 10 seconds when routed through a queue. "Our monitoring systems have detected a potential issue with dequeue calls. Our engineering team has been alerted and is actively investigating. We will update as soon as we have more information."
Report: "Investigating Potential Issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring progress. We're currently processing jobs for reporting and events.
The issue has been identified and a fix is being implemented.
We are aware of intermittent errors occurring on the service. Our incident response team is investigating.
Report: "Browser Softphone"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are seeing partial recovery
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Investigating Potential Issue"
Last updateThis incident has been resolved.
We have identified and taken corrective action. We are monitoring performance.
We are aware of intermittent errors occurring on the service. Our incident response team is investigating.
Report: "Elevated 502 Errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Intermittent 502 errors occurring for white-labeled hosts"
Last updateServices have been restored and operations are back to normal. Thank you for your patience.
This is affecting a small subset of users. We are still working on the issue and you may connect to https://app.calltrackngmetrics.com.
You may connect to our main site https://app.calltrackingmetrics.com without errors.
We are continuing to investigate this issue.
We are aware of intermittent 502 errors occurring for white-labeled accounts. Our incident response team is investigating.
Report: "Intermittent issues searching for and purchasing phone numbers"
Last updateServices have been restored and operations are back to normal. Thank you for your patience.
Root cause has been identified and mitigation steps have been implemented. We are still working on a full resolution and teams are monitoring error rates as they continue to improve.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are aware of intermittent issues searching for and purchasing phone numbers. Our incident response team is investigating.
Report: "Access to the CTM Service"
Last updateWe experienced a brief period of elevated 500 errors between 10:04 and 10:12
Report: "Elevated 500/502 errors"
Last updateStarting at 11:36 and ending at 12:08 we experienced elevated 502 errors. This was due to a recovery process that was putting too much load on our databases related to the incident on Thursday. Our team is working on expanding our footprint to support increased load spikes like this now. The issue is resolved and we will not repeat this.
Report: "Issues with connecting to CTM"
Last updateThis incident has been resolved.
Our system is currently stable. Activities that were received during the incident will eventually appear in the log. New activities will appear in real time. We will continue to monitor performance.
We are seeing recovery and are monitoring performance. We will update again at 5:00 pm EST.
We are continuing to work on a fix for this issue.
At the moment, our issues have returned. Our team is working on the resolution. The next scheduled update is at 4:30 pm EST.
Site performance has improved, we continue to monitor. Our next update will be at 4:30 pm EST.
We continue to monitor site performance. Thank you for your patience. Our next update will be at 3:30 pm EST.
We are seeing recovery and continue to monitor performance. We will update again at 3:00 pm EST.
Site availability has been restored. Our team continues to monitor as a regional outage in AWS infrastructure caused a database failure.
We appreciate your patience as we continue to investigate the cause of the issue. We will update again at 2:30 pm EST.
We are continuing to investigate the issue.
We are currently investigating the issue.
Report: "Google Call Extensions/Call Only Ads"
Last updateFrom 5:27pm EST to 8:04pm EST retrieval of Google Click data was limited. Our teams were alerted to the issue and worked on a solution. During this time call delivery was not impacted but retrieval of the additional Click data was not available. You may notice phone calls during this period missing attribution data.
Report: "Users experienced 500 errors when using CallTrackingMetrics"
Last updateWe received reports of 500 errors when trying to perform various functions within CTM. The incident began at 12:11 p.m and was resolved at 12:39 pm. All times Eastern.
Report: "SMS Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Messages with a trust profile are not impacted by this issue. A work around is in progress to restore delivery
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Issue with Voice Calls - Inbound and Outbound"
Last updateAll affected services have fully recovered.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We have been informed of recovery and monitoring status by our upstream provider. Normal performance is being reported. We continue to monitor.
We are seeing lag in handling of call event processing. This is primarily due to our upstream provider having a critical incident with webhook infrastructure.
We continue to see improved performance with CallTrackingMetrics. We will continue to monitor and will update this page within the next 30 minutes.
We have experience an issue with inbound and outbound calls and have taken corrective action. We are monitoring the performance and will update this page within 30 minutes.
Report: "Degraded performance with Voice Activities"
Last updateThe issues with the ability to answer and place calls has been resolved. Please contact our Help Desk if you continue to experience issues.
Voice activities are operating normally. We continue to monitor performance.
The issue causing the inability to answer and place calls continues to be investigated with the highest priority. We will update as soon as we have more information.
We are currently experiencing an incident that is negatively affecting the ability to answer and place calls within CTM. This issue lies with the upstream voice network and is being investigated with the highest priority. We will update as soon as we have more information.
Report: "API Logs"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are looking into an issue impacting logging of API activity. Temporarily logs for third party API data will be suspended.
Report: "US Toll Free and Subset of US Long Code Numbers Experiencing Inbound and Outbound SMS and MMS Delivery Delays to/from US and Canada Networks"
Last updateThis incident has been resolved.
While we've observed noticeable recovery after the implementation of the workaround, we continue to see intermittent delivery delays sending and receiving Toll Free SMS and MMS messages in US and Canada, along with a small subset of T-mobile traffic on US long codes. The issue is also causing delivery receipts to be delayed, with delivery status stuck at "Sent". Our engineering team is continuing to work with our carrier partner to fully resolve the issue. We will provide another update in 1 hour or as soon as more information becomes available.
A fix has been implemented and we are monitoring the results.
We are now observing some recovery in which some SMS and MMS to/from US and Canada networks from US Toll-Free and US long codes are being delivered. Our engineers have identified the issue and implemented a workaround for older messages. However, message DLRs will continue to be delayed even while messages are getting delivered. Our engineers are continuing to work with our carrier partner to resolve the issue. We will provide another update in 4 hours or as soon as more information becomes available. (from upstream)
The issue has been identified and a fix is being implemented.
We are continuing our investigation and have discovered inbound traffic to Twilio Toll-Free numbers from US and Canada networks is also affected and experiencing delivery delays.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing our investigation and have discovered inbound traffic to Twilio Toll-Free numbers from US and Canada networks is also affected and experiencing delivery delays. In addition, we have discovered a small portion of US long code numbers are affected and also experiencing SMS and MMS delivery delays for inbound and outbound traffic to/from US and Canada networks. Our team continues working with our carrier partner to resolve the issue. We will provide another update in 2 hours or as soon as more information becomes available.
Report: "Elevated Error Rates"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Intermittent Errors"
Last updateWe encountered increase site issues starting at around 11:14pm EST until 11:32pm EST. We continue to monitor.
Report: "Elevated Error Rates"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have managed to mostly isolate the impacts of the issue from a single availability zone in AWS however due to many APIs failing to respond and observing intermittent network availability from within the zone as instances lose connectivity we may experience elevated timeout errors.
The issue appears to be subsiding and possibly related to an outage in one region within AWS. We see we had multiple failover events from multiple systems at the same time all originating from a single region and availability zone within AWS.
We are currently investigating this issue.
Report: "Issues Sending SMS"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
This issue may also be impacting some call flows as well as degrading phone number availability for some phone numbers.
We are working with our upstream providers on an issue related to sending text messages.
Report: "Issue affecting Call Log activities"
Last updateThis incident has been resolved. Normal CTM call log functionality is restored.
We are continue to see recovery of the issues and seeing increased normal performance. We are monitoring the situation.
Beginning 3:15 PM EST, our upstream provider started experiencing API degradation causing latency and timeout errors for a myriad of services impacting our Calls, Texts and Chat communications. Their Engineers have worked on mitigating the issue and we are seeing recovery of product and services since 4PM EST. The next update will be in 30 minutes.
Our carrier network has identified the issue causing the degraded performance and are taking steps to mitigate the issue.
We are continuing to monitor for any further issues.
We are monitoring the issue with our upstream and seeing nearly full recovery now
We are receiving notification from our upstream carrier network of issues that are affecting the normal operation of CTM. The issue is being investigated.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Elevated Error Rates From AWS Lambda"
Last updateThis incident has been resolved.
We are seeing fewer errors from AWS Lambda however AWS still reports recovering on some services so we will continue to monitor
All delays are resolved, we are still seeing elevated errors from AWS
Job processing delays related to AWS outage is causing some delays in delivery of text messages outbound and additionally we are still tracking some delays in updating the total active count of calls in the log. We are continuing to monitor status from AWS for a resolution.
We are starting to see delays in some post processing tasks related to slow response times from aws dependent services. In some cases providers we work with that are currently impacted by AWS is impacting the time to process some events in the call log such as total active call counts etc.
AWS has posted status on this issue. We are waiting on a resolution. All delayed lambda function invocations will be replayed once the issues are cleared.
The issues continue but they do appear limited to at least AWS Lambda, any customers not using Lambda do not appear to be impacted.
We are observing elevated error rates from AWS Lambda. This could cause issues with custom call scripts, triggers and routing rules that rely customers have created to run in AWS. We are looking into migrating some of these failing functions to a different region.
Report: "Issue with Voice Calls - Inbound and Outbound"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We have identified the cause of the issue and are implementing a solution.
Report: "Chat Issues Reported"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating issues with Chat.
Report: "Delays in processing Facebook conversions"
Last updateThis incident has been resolved.
We will continue to monitor https://status.fb.com/adsmanager
We are holding conversions until Facebook's API's recover from their outage today. Once API access is fully restored we will begin the process of replaying failed conversion uploads.
Report: "Reports of outbound call errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Saving Lambda Function Issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating reports of issues saving Lambda functions. The issue appears to be related to a resource contention issue with AWS
Report: "Elevated errors sending push notifications via Google"
Last updateThis incident has been resolved.
We are tracking the issue now with Firebase: https://status.firebase.google.com/
We are currently investigating elevated 502 errors when sending push notifications via Google push notification service. This would potentially impact notifications for agents using Chat and Native app Android users receiving incoming phone calls.
Report: "Temporary issues loading activity reports"
Last updateThis incident has been resolved.
We are currently investigating this issue.