Historical record of incidents for Cytracom
Report: "ControlOne: US-NorthEast (EWR) Partial Outage"
Last updateWe have identified and rectified an issue affecting a subset ControlOne customers using US-NorthEast (EWR) region.
Report: "UCaaS Fax Completion Outage"
Last updateWe have identified an issue impacting Cytracom UCaaS customers utilizing fax completion. This is a failure within a third-party component powered by GCP, which is experiencing a widespread outage. We will provide updates as they become available.
Report: "ControlOne US-SouthEast (ATL) Region Infrastructure Improvements"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Engineers will be performing maintenance on connectivity components in the US-SouthEast (ATL) Region. This is disruptive maintenance. A subset of Bridges and Agents may be intermittently unable to connect to resources in US-SouthEast (ATL) Region during this maintenance window.If you have any questions or need assistance, our support team is available 24/7 at support@cytracom.com or 877-411-2987.
Report: "ControlOne: US-NorthEast (EWR) Partial Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
A fix for this issue is beginning to roll out to impacted customers.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We have identified an issue impacting ControlOne customers utilizing the ControlOne Agent within US-NorthEast (EWR) tegion. Our engineers are actively working on a fix.
Report: "UCaaS Cytracom Desktop Call Completion"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified an issue impacting incoming calls to the Cytracom Desktop application. Our team is working on a fix.
Report: "Cytracom Desktop Degraded Performance"
Last updateThis incident has been resolved.
A fix has been deployed for Cytracom Desktop users, and engineers are actively monitoring application performance. Users should refresh their session to ensure the latest version of Cytracom Desktop is installed. If you have any questions or need assistance, our team is available at support@cytracom.com or 877-411-2987.
The source of crashes associated with Cytracom Desktop has been identified and engineers are working on a fix. The user experience at this time has returned to normal, however, once the fix is implemented the status will be updated.
Engineers are investigating reports of call failures and application crashes when using the Cytracom Desktop application to place and receive calls.
Report: "Cytracom Desktop Degraded Performance"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified an issue causing a degraded experience for users of the Cytracom Desktop application. Our engineers are implementing a fix.
Report: "ControlOne Partial Service Degradation in US-NorthEast (EWR)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We have identified an issue impacting ControlOne customers utilizing the ControlOne Agent and Bridge in our US-NorthEast(EWR) region. We have implemented a fix to route around upstream carrier instability and are monitoring the results.
Report: "ControlOne Management Portal - Config Jobs Slow to Apply"
Last updateThis incident has been resolved.
Customers should expect to continue to see delays in configuration submissions. Engineers have identified the cause and are implementing a fix.
All fixes have been implemented and config jobs have returned to processing normally. We will continue to monitor the results.
The root cause has been identified and a fix has been applied.
We are currently investigating an issue with the ControlOne introducing delays in applying configurations
Report: "ControlOne US-Northeast (EWR) Region: ISP Connectivity Issues"
Last updateDamage to internet backbone fiber persists in the Southeastern United States, affecting overall internet performance on the broader eastern seaboard. Our engineering team executed mitigations and has been monitoring since the last update. We have seen a return to steady performance over the past 24 hours.
We are continuing to monitor the situation on the ground. All customer connections have been successfully rerouted through alternative ingress pathways; at this time you may see increased latency or variability in latency as traffic takes non-ideal detours, but packet delivery rates have returned to nominal and the service should be broadly usable as normal. We will update you as the situation progresses from our vantage point.
We are continuing to monitor the overall situation and the Cytracom NOC continues optimizing network pathways for current Internet conditions. We will update you as the situation progresses from our vantage point.
We have identified and are currently monitoring widespread connectivity issues affecting multiple end-user ISPs in the US-Northeast (EWR) region. Users may experience intermittent connection problems, packet loss, or increased latency when connecting to ControlOne services. Our Network Operations Center (NOC) is actively working to mitigate these issues by: - Implementing dynamic routing adjustments - Prioritizing critical traffic paths - Coordinating with affected ISPs to identify resolution timelines If your organization is experiencing severe disruption, please reach out to our support team to discuss available emergency alternatives. Our support team is available 24/7 at support@cytracom.com or 877-411-2987.
Report: "Partial Service Degradation in the US-SouthEast (ATL) Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue impacting ControlOne customers using our US-SouthEast (ATL) Region and have implemented a fix.
Report: "ControlOne Agent and Bridge: Partial Service Degradation in US-SouthEast (ATL) Region"
Last updateThis incident has been resolved.
A fix has been implemented and all impacted services are now operational. We will continue to monitor.
We are continuing to work on a fix for this issue.
We have identified and mitigated an issue within the US-SouthEast (ATL) region. We have implemented a fix and we are monitoring the results.
Report: "Delays or unable to login to PBX Management + Partner Portal + Cytracom Desktop + UC Mobile"
Last updateThis incident has been resolved.
Login to PBX, Partner Portal and UC Applications remain fully operational. Business messaging load times have returned to normal. We will continue to monitor
We are continuing to work on a fix for this issue.
A fix has been implemented. Logins to PBX, Partner Portal and UC Applications are fully operational.
We are currently investigating an issue preventing users from logging in to Cytracom Management UI, Partner portal, UC Mobile, UC Desktop.
Report: "Management + Partner Portal + Cytracom Desktop"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Impact to Call Center, Management Portal and Partner Portal have been resolved. Monitors continue to see impact to Desktop. Engineers continue to work to identify and resolve the source of these issues.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating an issue preventing the partner portal from loading after login. New Cytracom Desktop sessions may fail to load. Outbound calls from Cytracom Desktop may experience a delay. Inbound Calls are unaffected.
Report: "Bridge: Partial Service Degradation US-MidWest (ORD) Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The majority of affected ControlOne customers are online and passing traffic. Configuration changes may be delayed.
We have identified an issue impacting a subset of ControlOne customers using a ControlOne Bridge within the US-Midwest (ORD) Region. ControlOne Agents online and passing traffic. Our engineers are actively working on a fix.
Report: "Partial Service Degradation for Agents in the US-Midwest (ORD) Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue impacting ControlOne Agents within the US-Midwest (ORD) region. Our engineers are actively working on a fix.
Report: "Partial Service Degradation for Agents in the US-Midwest (ORD) Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results. At this time Agents are fully operational.
The issue has been identified and a fix is being implemented.
We have identified an issue impacting ControlOne Agents within the US-Midwest (ORD) region. Our engineers are actively working on a fix.
Report: "Control Agent: Partial Service Degradation US-SouthCentral (DFW) Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue impacting ControlOne customers utilizing the ControlOne Agent within our US-SouthCentral (DFW) region. Our engineers are actively working on a fix.
Report: "Partial Service Degradation US-SouthEast (ATL) Region"
Last updateThis incident has been resolved.
We have identified an issue impacting ControlOne customers using our US-SouthEast (ATL) Region and have implemented a fix.
Report: "Bridge: Partial Service Degradation US-SouthEast (ATL) Region"
Last updateAll routing services in the US-SouthEast (ATL) Region are operating nominally
All CBRs should be operational. Our engineers are continuing to monitor the situation.
We have identified an issue impacting ControlOne customers utilizing the ControlOne Agent within our US-SouthEast (ATL) Region. Our engineers are actively working on a fix.
Report: "Connectivity disruption in the ControlOne US-MidWest (ORD) region"
Last updateThis incident has been resolved.
Connectivity has been restored, at this time Agents and Bridges are fully operational.
We have identified and corrected an issue with upstream connectivity from the ControlOne midwest region.
Report: "UCaaS Partial Service Outage"
Last updateThis incident has been resolved.
Engineers have successfully implemented a fix to address the call completion and call delay issues. We will continue to monitor the platform.
The issue has been identified and engineers are implementing a fix. A subset of customers may experience delays in calling until the fix is fully implemented.
We are currently investigating service failure for a subset of customers
Report: "UCaaS Call Quality Impact"
Last updateFrom 6A to 8:50A CT 2/24/25, all customers experienced call delay, intermittent disruption during call setup, and poor call quality. This issue was resolved at 8:50 CT.
Report: "UCaaS Partial Outage"
Last updateThis incident has now been resolved. All systems are fully operational.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Some customers may experience delays when making or receiving calls. Call Center for affected customers will require the agent user to log out and back into the queue.
We are continuing to work on a fix for this issue.
We have identified the cause of a short interruption to UCaaS services and are investigating.
Report: "ControlOne IPSec connectivity degradation"
Last updateThis incident has been resolved.
Configuration changes to ControlOne tenants with an IPSec connector may be delayed in being applied and, in some cases, cause the IPSec session to disconnect. A fix has been implemented and we are monitoring as affected IPSec connectors re-initialize with their peers.
Report: "Partial Service Degradation in the US-NorthEast (EWR) Region"
Last updateThis incident has been resolved.
Engineers have fully implemented the fix. Bridges and Agents are now fully operational.
Engineers continue to implement the fix to restore services. At this time Agents and Bridges are passing traffic. There is still performance degradation for Bridges and Agents
At this time ALL ControlOne agents connected to US-NorthEast (EWR) Region are impacted and ControlOne Bridges in the region are partially impacted.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We have identified an issue impacting ControlOne customers within our US-NorthEast (EWR) region. Our engineers are actively working on a fix.
Report: "Bridge: Partial Service Degradation in US-SouthEast (ATL) Region"
Last updateThis incident has been resolved.
The US-SouthEast (ATL) is operational. This was due to regional network transit issue unrelated to Cytracom or the ControlOne platform. We are continuing to monitor the situation.
We have identified a regional backbone provider impacting a large number of customers in the Atlanta region. This will impact both tunneled and non-tunneled connectivity to the Internet. We will continue to provide details as they become available.
We are investigating an issue impacting ControlOne customers utilizing the ControlOne Agent within our US-SouthEast (ATL) region. Our engineers are actively working on a fix.
Report: "ControlOne Service Outage US-Southcentral (DFW)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue with ControlOne customers utilizing the ControlOne Agent within our US-Southcentral (DFW) region. Our engineers are actively working on a fix.
Report: "ControlOne Service Outage US-Northwest (SEA) and US-Southwest (LAX)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We have identified an issue impacting ControlOne customers utilizing the ControlOne Agent within our US-Northwest (SEA) and US-Southwest (LAX). Our engineers are actively working on a fix.
Report: "ControlOne US-Southcentral Increased Latency and Impaired Routing"
Last updateThis incident has been resolved.
Active monitoring systems show ControlOne Agent and Bridge connection rates are stable. We will continue to monitor carrier routing paths and update status if there is a change in service.
At approximately 12:20 PM CT, instability was detected on the primary carrier servicing the US-Southcentral ControlOne region. Automatic remediation was performed to route around the primary carrier, which is converging on the broader internet. Agents may show a retrying status while alternate routing comes online. We are continuing to monitor the situation and will provide updates.
Report: "Service Outage: US-Northeast (EWR)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We have identified an issue impacting ControlOne customers utilizing the ControlOne Agent and Bridge within our US-Northeast (EWR) Region. Our engineers are actively working on a fix.
Report: "Phone Registration Failures in DFW Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an increase in phone registration failures in the DFW Region.
Report: "Login failure on secure.cytracom.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
This issue has been identified and a fix put in place. Authentication are now processing normally
We are investigating issues with new logins to secure.cytracom.com. Existing authenticated users are not impacted.
Report: "Cytracom Desktop Partial Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue with audio establishing on Cytracom Desktop and are currently investigating.
Report: "Partner Portal Outage"
Last updateThis incident has been resolved.
The Cytracom engineering team has resolved the issue, and we are now monitoring the solution.
Report: "Partial service degradation in US-SouthEast (ATL) Region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue impacting ControlOne customers utilizing the ControlOne Agent within our US-SouthEast (ATL) Region. Our engineers are actively working on a fix.
Report: "ControlOne Management Portal - Configuration and Status Delays, Intermittent Connectivity Interruptions"
Last updateThis incident is resolved. We have restored functionality for all bridges as well as configuration changes in the ControlOne management portal.
A fix has been implemented. All customers should see no further connectivity interruptions. We are continuing to monitor to ensure reliable configuration changes via the ControlOne portal.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a resolution for this issue. Customers in the EWR, ORD, and LAX regions should see fully restored operation.
We are continuing to implement a fix for the issue.
We have implemented a potential fix for this issue and are actively monitoring the results.
We are continuing to work on a fix for this issue.
Configuration changes made in the ControlOne Management Portal and status information are currently experiencing delays populating throughout our platform. A subset of customers using a ControlOne Bridge are experiencing intermittent or degraded connectivity issues. We have identified the cause of the issue and are working on a fix.
Report: "ControlOne - Partial service degradation in US-NorthEast (EWR)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are implementing additional fixes for this issue.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
A subset of ControlOne customers in the US-NorthEast (EWR) are experiencing a service interruption.
Report: "ControlOne - Partial service degradation in US-NorthEast (EWR)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our engineers are continuing to implement the required fixes within the US-NorthEast environment. Customers should begin seeing restored connectivity.
We are continuing to implement a fix for this issue.
The issue has been identified and the team is working on a fix.
A subset of ControlOne customers in the US-NorthEast (EWR) are experiencing a service interruption.
Report: "Fax Portal Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified an issue with our upstream fax provider. Inbound and outbound fax services are operational, but users are unable to access the cFax portal. We appreciate your continued patience while we worked to resolve this issue.
Report: "Voice Services Partial Degradation"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified an issue affecting call completion, call quality, and call center functionality for a subset of our customers. The issue has been identified and we are working on a fix.
Report: "ControlOne - Partial connectivity degradation in US-NorthEast (EWR)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we're monitoring the results
We have implemented a temporary mitigation. All customers on the ControlOne Bridge and Agent can access public resources, but will unable to access private resources on the ControlOne platform.
The issue has been identified and a fix is being implemented.
A subset of customers utilizing the ControlOne Bridge are also experiencing connectivity disruptions. Our team is investigating to identify root cause and remediate.
A subset of ControlOne agents are experiencing intermittent connectivity disruptions.
Report: "ControlOne - Partial degradation in agent connectivity in US-SouthCentral (DFW)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
A subset of ControlOne agents are experiencing intermittent connectivity disruptions. A fix is being implemented.
Report: "ControlOne - Partial degradation in payment processing functionality"
Last updateThe root cause of the billing services issue has been resolved by our upstream provider. At this time, all billing-related functions appear to be operating normally.
Our billing system provider is actively working on the issue. We will keep a close watch on the situation and provide updates as new information becomes available. Please reach out to your partner account manager for any urgent ControlOne orders.
We are continuing to experience service degradation due to issues with our billing system provider. We will continue to monitor the situation closely and provide further updates as we receive more information.
An upstream provider is currently experiencing a service degradation. Cytracom ControlOne customers may see issues managing licenses or processing payments.
Report: "UCaaS Call Quality Degradation"
Last updateThis incident is resolved.
We are continuing to work with our infrastructure providers to mitigate the impact.
Degraded interconnects between Internet backbone providers are causing degraded call quality for a subset of customers in and around southern California. We are working with our transit providers to mitigate the impact.
Report: "Degraded Performance in US-SouthCentral (DFW)"
Last updateAll operations returned to normal levels.
A fix has been implemented. We are monitoring the results.
The issue has been identified and a fix is being implemented
We are investigating degraded performance in US-SouthCentral (DFW) Region.
Report: "Voice Services Partial Degradation"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified an issue affecting call completion for a subset of our customers using Cytracom Desktop. Cytracom Engineering is working to resolve this and will provide updates here.
Report: "ControlOne US-SouthCentral (DFW) Production Routing Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to implement a fix for this issue. Systems are coming back online.
The issue has been identified and a fix is being implemented.
We are investigating issues with ControlOne routing in the US-SouthCentral (DFW) region.
Report: "Fax Services Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results
We have identified an issue with our upstream fax provider. We appreciate your continued patience while we worked to resolve this issue.
Report: "UCaaS Degraded Performance"
Last updateThis incident has been mitigated and resolved.
We have identified and mitigated an issue impacting the UCaaS product. We are monitoring the results.
Report: "Voice Services Outage"
Last updateOn April 10 and 11, CPU, IO, and network contention within a shared tenancy environment led to service degradation. The interruption was in connection to a data center upgrade project that has been underway for several months and is now complete. In April, the final phase of migration away from legacy architecture was initiated. The disruptions on April 10 and 11 were the result of the temporary use of a third-party cloud provider—a maintenance step during the infrastructure upgrade that was required in the process of moving to new architecture. That provider experienced a service interruption that directly impacted Cytracom voice customers. The project is now complete, and all VoIP users have been migrated to the new data center architecture as intended.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
A fix has been implemented and we are monitoring the results.
We have identified an issue affecting call completion for a subset of our customers. Cytracom Engineering is working to resolve this and will provide updates here.
Report: "Voice Services Degradation"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue affecting call completion for a subset of our customers. Cytracom Engineering is working to resolve this and will provide updates here.
Report: "Voice Services Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue affecting call completion for a subset of our customers. Cytracom Engineering is working to resolve this and will provide updates here.