Historical record of incidents for CallRail
Report: "Text messages not appearing in the timeline"
Last updateThere is currently an issue with text conversations not loading in the timeline. We are investigating the issue.
Report: "Google Ads disruption of service"
Last updateGoogle Ads is reporting disruptions to its Ads Manager service, which includes the APIs used by our integration. Customers using CallRail's Google Ads Integration may experience degraded service or intermittent failures due to issues with the upstream provider, including delayed conversion reporting data and missing or incorrect CPL, keyword, and campaign data. Customers may also experience issues configuring or authorizing the Google Ads Integration on CallRail's platform. We are closely monitoring the fix and will report when service is restored for all customers.
Report: "Google Ads disruption of service"
Last updateWe are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
Google Ads is reporting disruptions to its Ads Manager service, which includes the APIs used by our integration. Customers using CallRail's Google Ads Integration may experience degraded service or intermittent failures due to issues with the upstream provider, including delayed conversion reporting data and missing or incorrect CPL, keyword, and campaign data. Customers may also experience issues configuring or authorizing the Google Ads Integration on CallRail's platform. We are closely monitoring the fix and will report when service is restored for all customers.
Report: "Cloud Console outage"
Last updateWe have identified an issue with an upstream vendor and are actively working with them to mitigate the situation. This issue is causing the following impacts:<ul> <li> <li> </ul> We sincerely apologize for any disruption this may cause.
Report: "Facebook Integration disruption of service"
Last updateThis incident has been resolved.
Our monitoring indicates that the previously reported errors are now clearing. We are continuing to observe the situation closely to confirm full resolution.
Our monitoring indicates disruption to the APIs used by our Facebook Integration. Customers using CallRail's Facebook Integration may experience degraded service or intermittent failures due to issues with the upstream provider, including delayed conversion reporting data and missing or incorrect CPL, keyword, and campaign data. Customers may also experience issues configuring or authorizing the Facebook Integration on CallRail's platform. We are closely monitoring the fix and will report when service is restored for all customers.
Report: "Facebook Integration disruption of service"
Last updateThis incident has been resolved.
Our monitoring indicates that the previously reported errors are now clearing. We are continuing to observe the situation closely to confirm full resolution.
Our monitoring indicates disruption to the APIs used by our Facebook Integration.Customers using CallRail's Facebook Integration may experience degraded service or intermittent failures due to issues with the upstream provider, including delayed conversion reporting data and missing or incorrect CPL, keyword, and campaign data. Customers may also experience issues configuring or authorizing the Facebook Integration on CallRail's platform.We are closely monitoring the fix and will report when service is restored for all customers.
Report: "Google Ads disruption of service"
Last updateThe incident has been resolved and the Google Ads integration is functioning normally.
Google Ads is reporting disruptions to its Ads Manager service, which includes the APIs used by our integration. https://ads.google.com/status/publisher/incidents/kUAodsf3fRfrx2Gu7wLs Customers using CallRail's Google Ads Integration may experience degraded service or intermittent failures due to issues with the upstream provider, including delayed conversion reporting data and missing or incorrect CPL, keyword, and campaign data. Customers may also experience issues configuring or authorizing the Google Ads Integration on CallRail's platform. We are closely monitoring the fix and will report when service is restored for all customers.
Report: "Events passed in Google Analytics integration are missing attribution data"
Last updateThe issue is now resolved. Note that due to GA processing times, you may need to wait until tomorrow (5/14) to see this reflected in your reports.
We have been made aware of a change in format for the session cookie issued by Google Analytics (GA4) that is causing events to be passed without standard GA4 attribution data. This impacts calls and texts to website pools as well as form submissions that are sent to GA4 via our integration. These events will show in report with attribution data of "not set" and default channel grouping of "undefined". In-app reporting and other integrations are not impacted.
Report: "Investigating reports of API users recieving 500 Internal Server errors"
Last updateThis incident has been resolved. All functionality has been restored.
We are investigating reports of API users recieving 500 Internal Server errors. No other functionality should be impacted. Calls are continuing to route.
Report: "Ongoing report exporting delays"
Last updateThis incident has been resolved.
Our engineering team identified the cause of the slow report exports and have added additional capacity to mitigate the export issues. We are monitoring the system and will continue to make updates as they become available.
We are investigating delays with report exports completing. The reports are loading and are viewable in the application.
Report: "iOS CallRail app loading delays"
Last updateThis incident has been resolved.
We are aware of an issue causing the iOS app to remain in a loading state for some users. Reinstalling the app has been effective in resolving these cases. Our team is actively monitoring the situation to ensure a complete resolution.
Report: "Caller ID not available with call"
Last updateThe issue with our downstream provider has been resolved and caller ID information is no longer delayed. Current/in-progress calls will display the expected information. Prior calls that were received while the incident may still show as "Unavailable", but we will backfill any missing data.
We have identified that this issue is caused by a downstream provider and we are working with them to resolve this issue. In the meantime, all calls are routing correctly although caller identification data may be missing. We will continue to monitor and work with our provider to fix this issue as quickly as possible.
We are currently investigating an issue that is causing the receipt of caller ID information to be delayed. Caller names may display as "Unavailable" in Lead Center, the call and text activity logs, and in reports. Call routing is not impacted.
Report: "Google Ads Call Details Forwarding Issue"
Last updateThis issue has been resolved.
Google Ads Call Details Forwarding calls should now be returning to normal. Though we will continue to monitor this to ensure the issue has been fully resolved.
We have identified the root cause of this issue and are working to resolve it.
We are investigating reports of some dropped calls for Google Ads Call Details Forwarding to CallRail trackers. Other calls are routing correctly and reporting and access to CallRail are unaffected.
Report: "Call Log Export Issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results. We appreciate your continued patience.
Our team is diligently pursuing solutions for the reported issue. We will continue to provide regular updates on our progress.
We've identified the cause of the Call Log export and our engineers are working on a resolution. We will continue to make updates as they become available.
We're aware of an issue with exporting the Call Log. Our engineers are working diligently to identify the issue. We will make updates as they become available. Thank you for your patience.
Report: "Investigating Isolated Call Routing Issues"
Last updateThis incident has been resolved.
A fix has been implemented and we're observing normal call routing performance.
We have identified an issue with an upstream vendor that is impacting call routing for some customers. We are actively working with them to mitigate the situation. We sincerely apologize for any disruption this may cause.
We're currently investigating call routing failures affecting a small portion of calls. Our team is actively working to identify the root cause and restore full functionality. We will provide updates as we learn more.
Report: "SMS Messaging May be Delayed"
Last updateIssues with messaging and webhooks have now been resolved. All delayed messages should have been delivered and new messages are being delivered without delay.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We've identified an issue with SMS, causing some messaging to be delayed. Integration processing for SMS events, including Webhook events, is also delayed. We are currently implementing a resolution.
Report: "Caller ID information is delayed"
Last updateAll missing caller ID information has been backfilled.
We are still working to recover caller name information for calls received between approximately 9:12am ET and 11:02am ET. Calls received since 11:02am are processing normally and have the expected caller ID information.
The issue is resolved and caller ID information is no longer delayed. Current/in-progress calls will display the expected information. Prior calls that were received while the incident may still show as "Unavailable", but we expect to backfill missing data shortly.
We are continuing to investigate this issue.
We are currently investigating an issue that is causing the receipt of caller ID information to be delayed. Caller names may display as "Unavailable" in Lead Center, the call and text activity logs, and in reports. Call routing is not impacted.
Report: "Dashboard Unavailable"
Last updateFull operational performance has been restored.
We have reverted a recent deployment, and are currently monitoring to ensure full performance has been restored.
We are currently investigating a problem with application dashboard availability, and believe to have identified the source of the issue. More details will be posted as soon as possible.
Report: "Google Ads disruption + delayed event updates"
Last updateGoogle Ads has resolved their incident. We've re-enabled the integration and data should be flowing as normal.
We've identified a disruption in service in the APIs used by our Google Ads integration. We have temporarily disabled conversion reporting to Google Ads as well as data retrieval of keywords and campaign data while we wait for the Google team to resolve the issue. Users may be unable to activate new Google Ads integrations. This issue is causing delays to some events, especially events triggered post-call. Some actions such as user management and email notifications may take additional time to process. Call routing is not impacted.
Report: ""Call back" option not displaying in the Analytics section"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue that is causing the "call back" button to not display in various pages of the Analytics section, including the Activity logs and the Lead timeline page. Call routing and reporting are not impacted.
Report: "Call summaries not accessible to some users"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Call summaries are not accessible in the report view. They appear to be working fine in the Timeline view.
Report: "Google Call Context data missing"
Last updateThe issue has been resolved. Calls from Google Ad Extensions that are configured for Call Details Forwarding are now receiving additional details like gclid and Campaign ID.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue with our Google Call Details Forwarding integration. You may be missing keywords, campaign, and gclid information in your reports for Google Ad Extension calls as early as October 15th. This issue does not affect call routing or conversion uploads to Google Ads.
Report: "Convert assist insights may be slow to generate"
Last updateThis incident has been resolved.
A fix has been deployed, and we are now monitoring.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "CallRail and Lead Center Intermittent Access Delays"
Last updateThis incident has been resolved.
We are seeing systems begin to recover and performance has been restored. We will continue to watch performance closely to ensure no further action is necessary.
We are seeing additional reports of errors when accessing the application. We are investigating and will continue to provide updates.
We identified an incident at our infrastructure provider that caused some traffic to fail to route properly through our load balancers. We took steps to reinitialize our load balancers there and our monitoring tools are indicating performance has been restored. We'll continue to watch performance closely to ensure no further action is necessary.
We've identified an issue at our infrastructure provider that appears related to this incident. We're watching that incident, and working to expedite a full resolution for our affected systems.
Some customers are continuing to experience issues accessing the CallRail web app and mobile app. Using the web app incurs several requests each time you navigate, we are seeing a small percentage of these failing. We are aware of these issues and are taking rapid action to resolve the situation. Calls and Dynamic Number Insertion are working normally, though Lead Center may not be accessible for some.
Report: "Service Provider Performance Issues Impacting Plan Changes & Invoice Viewing"
Last updatePerformance issues have been resolved and functionality has been restored.
We are continuing to monitor. As a reminder, the following actions are unavailable during this time: - Plan changes - Invoice viewing - New trial sign up with credit card - Adding/updating credit card to existing accounts
We are continuing to monitor.
Due to performance issues with one of our service providers, we have temporarily disabled the following: - Plan changes - Invoice viewing - New trial sign up with credit card
Report: "HubSpot outage impacting integration setup and data flow"
Last updateThis incident has been resolved.
HubSpot is fully operational. Quick check done to confirm.
Per HubSpot status page, HubSpot has addressed the initial outage issue and are still in monitoring mode of all their systems.
HubSpot is currently experiencing an outage: https://status.hubspot.com/ Until it is resolved, it may impact receipt of data in integrated HubSpot accounts and customers may be unable to set up new HubSpot integrations. We are closely monitoring the HubSpot status page.
Report: "Degraded Application Performance"
Last updateThis incident has been resolved
We have identified a product update that was negatively impacting performance, and have reverted that change. Performance is recovering back to expected levels.
We are continuing to investigate this issue.
We are investigating delays with reports and dashboards loading. This may result in notifications and data flowing via webhooks being delayed as well. We are actively working to resolve this issue. We apologize for any inconvenience this may cause.
Report: "Delayed event updates"
Last updateDelays in Google Ads APIs have cleared, and we've re-enabled conversion reporting.
Delays in Google Ads APIs continue, so conversion reporting remains disabled. All other post-call events are processing normally. We'll continue to monitor Google Ads API health and restore conversion reporting when it recovers.
We've identified a delay in the APIs used by our Google Ads integration. We have temporarily disabled conversion reporting to Google Ads as well as data retrieval of keywords and campaign data. Users may be unable to activate new Google Ads integrations. Overall CallRail event queue health is improving and we will continue to monitor Google's metrics. We'll re-enable all processes and post any delayed conversions as soon as the APIs have recovered.
We've identified the issue and are working on a fix.
We are continuing to investigate this issue.
Due to some processing queue backups, we are experiencing delays to some events, especially events triggered post-call. Some actions such as user management may take additional time to process.
Report: "Sign-on issues"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are receiving reports of errors when logging in from a small subset of users. We are actively working on a resolution and apologize for any disruptions you are experiencing related to this issue.
Report: "Lead Center mobile app isn't showing notifications"
Last updateThis incident has been resolved.
The issue was caused by a required certificate update that mobile app users were previously notified of. Impacted users must log out and log back into the app on their mobile device to continue receiving calls in the Lead Center mobile app.
Some customers are reporting not receiving notifications in Lead Center on mobile devices. We are currently investigating.
Report: "Some events may not have sent to Google Analytics 4"
Last updateDue to an issue that has since been resolved, some call, form, and text events failed to send via our Google Analytics 4 (GA4) integration between 12:08-12:11pm EST and 12:31-12:47pm EST. Only a subset of events that occurred during those time frames were impacted, and events occurring outside of those timeframes were sent successfully.
Report: "Call Failures for Some Tracking Numbers"
Last updateAll systems are operating normally at this time.
All internal metrics indicate call functionality has been fully restored. We continue to monitor updates from our upstream carrier.
Our carrier is reporting a partial recovery, and our internal metrics indicate call activity has returned to normal. We continue to monitor the situation, but believe calls are routing as expected at this time.
We are currently experiencing a partial outage affecting incoming and outgoing calls on about 25% of tracking numbers. Calls handled within Lead Center, and calls with Advanced Call Flow steps are more likely to be impacted. Our team has identified an outage at an upstream carrier as the root cause, and we are working with that carrier towards a resolution. No ETA is currently known, but both teams are actively working to fully restore service. We'll post an update as more information becomes available.
We are continuing to work on a fix for this issue.
We have identified an issue with an upstream vendor and are actively working with them to mitigate the situation. This issue is causing the following impacts: One-way audio, calls ending abruptly, or calls not connecting on some call tracking numbers One-way audio, calls ending abruptly, or calls not connecting on some Lead Center calls Issues with Advanced Call Flows We sincerely apologize for any disruption this may cause.
Report: "Application Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our monitoring has revealed increased delays in some internal systems. Some functions in the web application may be disrupted, but call routing is unaffected.
Report: "Call Log intermittently failing to populate"
Last updateThis incident has been resolved.
We have identified an issue with the Call Log intermittently failing to populate or loading slowly. Calls are still being tracked normally. We apologize for any inconvenience this may cause.
Report: "Issues with Lead Touchpoint Creation"
Last updateOur engineers have finished each of the tasks associated with this and the issue is now resolved.
The fix has resolved the issue and tasks to backfill missing data are continuing to run. We will post a final update when these tasks are complete.
Earlier this afternoon we discovered an issue related to attribution data (source, medium, campaign, keywords, landing page, etc). Impacts of issue include * Email notifications (for calls, texts, and forms) being sent without attribution data * Integration events being sent without attribution data or in some cases, not sent at all * Attribution data not sent in the responses for API calls * Attribution data not showing or incomplete in the app dashboard - home page, caller timeline, activity logs, reports * Lead attribution data not showing in Lead Center when handling calls, forms, and texts Our engineers have implemented a fix and are now standing by as they continue to monitor for any latent concerns.
We've identified the cause of the issue and are working to implement a fix.
We are investigating possible interruption in the creation of Lead Touchpoints are investigating the cause.
Report: "Sign-in Issues on CallRail Web Application"
Last updateThis incident has been resolved.
We are receiving reports of errors when logging in from a small subset of users. We are actively working on a resolution and apologize for any disruptions you are experiencing related to this issue. In the meantime, deleting your cookies for the CallRail application should allow you to sign in.
Report: "Google Ads disruption of service"
Last updateGoogle has marked the incident as resolved.
Google Ads is reporting disruptions to its Ads Manager service, which includes the APIs used by our integration. https://ads.google.com/status/publisher/incidents/Qv2RhZ21A54UJTPVDodp Customers using CallRail's Google Ads Integration may experience degraded service or intermittent failures due to issues with the upstream provider, including delayed conversion reporting data and missing or incorrect CPL, keyword, and campaign data. Customers may also experience issues configuring or authorizing the Google Ads Integration on CallRail's platform. We are closely monitoring the fix and will report when service is restored for all customers.
Report: "Event Queue Backup"
Last updateThis incident has been resolved.
The event queue is operating normally and we are continuing to monitor.
Event queues have returned to normal. We are continuing to investigate and have identified an issue with a third party provider.
Due to some processing queue backups, we are experiencing delays to some events, especially events triggered post-call. Some actions such as user management may take additional time to process. We are investigating the issue.
Report: ""Sign In With Google" Errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are aware of customers who use the "Sign In With Google" feature to access the web application encountering errors when logging in, and are investigating.
Report: "HubSpot disruption of service"
Last updateThis incident has been resolved.
HubSpot is reporting disruptions to several of its services: https://status.hubspot.com/ Customers using CallRail's HubSpot Integration may experience degraded service or intermittent failures due to issues with the upstream provider. Customers may also experience issues configuring or authorizing the HubSpot Integration on CallRail's platform. We are closely monitoring the fix and will report when service is restored.
Report: "Meta/Facebook Disruption of Services"
Last updateMeta/Facebook has resolved any impacted functionality: https://metastatus.com/
Meta/Facebook are reporting disruptions to several of their services: https://metastatus.com/?ref=dfc Customers using CallRail's Facebook Integration may experience degraded service or intermittent failures due to issues with the upstream provider. Customers may also experience issues configuring or authorizing the Facebook Integration on CallRail's platform. We are closely monitoring the fix and will report when service is restored.
Report: "Issue with HubSpot integration"
Last updateThe incident is resolved. We were able to restore functionality and set the state to Active for the majority of impacted integrations. However, we do recommend logging in to your CallRail dashboard to verify the current status of your integration. You may need to manually upload or backfill records for calls, texts, and form submissions that occurred while the issue was live.
A fix has been implemented and we are monitoring the results. Impacted customers are able to re-authorize to restore their failed HubSpot integrations.
We are aware of an issue impacting some customers using our HubSpot integration, causing it to fail. The issue has been identified and a fix is underway.
Report: "Sign-in Issues on CallRail Web Application"
Last updateThe earlier incident regarding CallRail Web Application sign-in issues is now resolved. We appreciate your patience.
We are actively monitoring reports of sign in issues. If you continue to have issues please visit https://app.callrail.com/authenticate/sign_out before trying to login again.
We are receiving reports from a small subset of users with login issues. We are actively working on a resolution and apologize for any disruptions you are experiencing related to this issue. In the meantime, visit https://app.callrail.com/authenticate/sign_out before trying to login again.
Report: "Issue with agent status in Lead Center"
Last updateThe earlier incident regarding stuck Lead Center calls is now resolved. We appreciate your patience.
We have implemented a fix for the Lead Center status issue. We are continuing to monitor the situation to ensure the issue is resolved and agents are no longer stuck in “busy” status. We sincerely apologize for any disruption this issue may have caused.
We are receiving scattered reports of Lead Center agents being stuck in “busy” status and being unable to hang up. Your ability to place and receive calls is of critical importance to us at CallRail. We are actively working on a resolution and apologize for any disruptions you are experiencing related to this issue.
Report: "Issues with report exports"
Last updateThe earlier incident regarding report generation is now resolved. We appreciate your patience.
Exports of CallRail reports are now loading normally for those accessing them via both browser and email. We are actively monitoring the issue to ensure report exports continue to load normally.
We have identified an issue with exports of CallRail reports failing for those accessing them via both browser and email. We are actively working to resolve the issue. We apologize for any inconvenience this may cause.
Report: "Systemwide Emergency Maintenance"
Last updateAs of 1:36am EST, error rates have returned to normal and all calls are routing normally. This issue has been resolved.
We are continuing to monitor systemwide performance. Performance continues to improve however we are still experiencing elevated error rates for call routing.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Degraded Call Transcript Availability"
Last updateOur transcript provider has confirmed that transcripts are being processed at normal times.
Internal monitoring indicates transcript availability has recovered and should be operating as expected. We will continue to monitor until we receive confirmation from the upstream provider.
Due to an issue with one of our upstream providers, we are currently experiencing delays in transcriptions and related Conversation Intelligence features. You may see delays in retrieving transcriptions in the app UI. Call notifications, webhooks, and other integrations that typically send transcription data may be sent without that data until the issue is resolved
Report: "Upstream Provider Issue - Delay in Call Recordings and Transcriptions"
Last updateThis incident has been resolved.
Due to an issue with one of our upstream providers, we are currently experiencing delays in call recordings, transcriptions, and related Conversation Intelligence features. You may see delays in retrieving recordings and transcriptions in the app UI. Call notifications, webhooks, and other integrations that typically send recordings and transcriptions may be sent without that data until the issue is resolved
Report: "Delay in processing call events"
Last updateFrom 9:30 pm to 10:00 pm Eastern time, planned maintenance activity resulted in an unanticipated delay in processing call events. During this time, access to call recordings and notifications was delayed, Lead Center users did not receive full caller context data, and some manual changes to call records (via the web application or the API) failed and returned errors. Processing of those events has now caught up, and call recordings and notifications for the impacted calls have been delivered.
Report: "Issue with starting trials"
Last updateThe issue from this morning has been resolved. Thank you for your patience.
A fix has been implemented for the earlier Trial Signup issue and we are now monitoring.
We have identified an issue with customers starting trials. We are actively working to resolve the issue. We apologize for any inconvenience this may cause. We encourage you to reach out to support or your account manager so that they can help and circle back with you once the issue is resolved.
Report: "Call Attribution Interruption"
Last updateThis incident has been resolved. All source attribution and UTM data is now available.
We are aware of and addressing an issue delaying or interrupting availability of website visit data to derive call attribution. We apologize for the delay, during which source attribution and UTM data for new first time calls to website trackers may be unavailable. We will update as the delayed data is reprocessed.
Report: "Delay in seeing changes to number swap"
Last updateThis incident has been resolved. All number and form creations and edits should now be in effect.
A fix has been implemented and we are monitoring the results.
There is a delay in reflecting changes to number swaps: new numbers and edited numbers will not start swapping immediately, and edits to swapping rules for existing numbers will not be immediately in effect. New custom forms will also not be visible immediately. Existing numbers are still swapping normally at this time and are not experiencing delays. Existing custom forms are still tracking normally. We are actively working to resolve this issue. We apologize for any inconvenience this may cause.
Report: "Delay in Caller ID information in Lead Center"
Last updateThis incident has been resolved.
Our third-party provider's fix was successful and we are no longer seeing delays in receiving caller ID information. However, we continue to actively monitor the situation. We apologize for any inconvenience this incident may have caused.
Our third-party provider has implemented a fix. We are still seeing some delays in caller ID information and are actively monitoring the situation. We apologize for any inconvenience this may cause.
We are currently experiencing a delay in receiving caller ID information from our 3rd party identity resolution provider. Only Lead Center customers are affected at this time. Calls are unaffected and routing as expected.
Report: "Temporary SMS availability issues for fully registered customers"
Last updateThis incident has been resolved.
Outbound SMS communications may be temporarily delayed or unavailable for a short time for customers fully registered with the Campaign Registry due to maintenance associated with 10DLC regulations
Report: "Call routing inconsistencies"
Last updateThe issue has been resolved. Calls are continuing to route as expected. Call recordings and call statuses are now up to date.
A fix has been implemented and we are monitoring the results.
The issue has been identified. Calls are now routing as expected. Some call recordings and call status' will be resolved once call maintenance is complete.
We have redirected all call handling to our backup data center. The majority of calls are routing correctly. A small number of LeadCenter agents may still be unable to receive calls, and our team continues to investigate this issue.
We are continuing to investigate this issue.
We’re aware that a portion of call are not routing and the team is investigating.