Is Voxtelesys Down Right Now? Check if there is a current outage ongoing.

We have received reports of intermittent dropped connections for customers in/around the West Coast. Voxtelesys network engineering is actively investigating the issue, with all available resources dedicated to identifying the scope and resolving the impairment to the network as quickly as possible. Please note this impacts certain customer whose ISP is using the handoff exchange that is dropping the packets. Our leadership team is fully engaged, with all network events automatically brought to their attention. We recognize the critical importance of reliable communication, and restoring full service is our top priority.

Power upgrades were performed successfully. All alarms have cleared, and no service interruptions were experienced.An additional power upgrade will be scheduled in the near future.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

In order to increase capacity for the years to come, we will be undergoing power upgrades in our Dallas–Fort Worth datacenter outside of normal business hours in the Central US.This work involves a controlled transfer of power from the old Power Distribution Units (PDUs) to the new ones, one power leg at a time. Local data center operations will proceed with the second leg only after all alarms have cleared.Should any issues arise, the work will be halted, reverted to previous configuration and rescheduled to a later date.While we are taking every precaution to prevent any disruption, power maintenance inherently carries some risk. We will be closely monitoring our Dallas–Fort Worth DC to ensure quick recovery and will provide updates if any unexpected issues occur.

Power upgrades were performed successfully. All alarms have cleared, and no service interruptions were experienced.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

This is a previously scheduled event https://status.voxtelesys.com/incidents/kk7c1sfccbfsIn order to increase capacity for the years to come, we will be undergoing power upgrades in our Salt Lake City datacenter outside of normal business hours in the Western US.This work involves a controlled transfer of power from the old Power Distribution Units (PDUs) to the new ones, one power leg at a time. Local data center operations will proceed with the second leg only after all alarms have cleared.Should any issues arise, the work will be halted, reverted to previous configuration and rescheduled to a later date.While we are taking every precaution to prevent any disruption, power maintenance inherently carries some risk. We will be closely monitoring our Salt Lake City DC to ensure quick recovery and will provide updates if any unexpected issues occur.

Monitoring continues to show stability. This incident has been resolved.

Our Operations team identified an elevated failure rate affecting messaging MO and DLR webhook deliveries. Upon detection, our Engineering team was promptly engaged and implemented a fix. Webhook deliveries are now completing successfully. We will continue to monitor the situation to ensure continued stability.

Monitoring continues to show stability. This incident has been resolved.

Our Operations team identified an elevated failure rate affecting messaging MO and DLR webhook deliveries. Upon detection, our Engineering team was promptly engaged and implemented a fix.Webhook deliveries are now completing successfully. We will continue to monitor the situation to ensure continued stability.

This power migration will be rescheduled to a later date. This is to ensure all necessary on-site equipment compatibility, addressing an issue identified with plug types.We're working with our vendors to sort out the logistics, which includes getting the right components shipped.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

In order to increase capacity for the years to come, we will be undergoing power upgrades in our Salt Lake City datacenter outside of normal business hours in the Western US.This work involves a controlled transfer of power from the old Power Distribution Units (PDUs) to the new ones, one power leg at a time. Local data center operations will proceed with the second leg only after all alarms have cleared.Should any issues arise, the work will be halted, reverted to previous configuration and rescheduled to a later date.While we are taking every precaution to prevent any disruption, power maintenance inherently carries some risk. We will be closely monitoring Salt Lake City to ensure quick recovery and will provide updates if any unexpected issues occur.

The incident has been resolved, and successful eFax delivery rates returned to normal. Thank you for your patience as we worked to address the issue.

We've identified an issue that may have impacted faxes sent through our eFax service between at 9:38 AM to 12:44 PM CST today. Some faxes may have been returned with the reason “Failed to Deliver - Error: Internal Failure”. A fix has been successfully implemented. If your faxes were impacted, please resend them at your convenience. We will continue to monitor. Thank you for your patience and understanding.

Further investigation showed a fiber cut which has now been spliced.

Voxtelesys NOC was alerted of a downed circuit affecting our DFW datacenter. The issue was isolated to a single Tier-1 upstream transit provider, which was automatically withdrawn from routing. Within 30 seconds, alternative IP transit providers successfully rerouted the affected traffic. We have an active trouble ticket with the implicated IP transit provider who is suspecting a potential fiber cut in Mckinney, TX. We are continuing to monitor.

At 14:16 CST, DNS Records that had been published to Cloudflare for several months were mysteriously removed, as indicated by the logs provided by Cloudflare. At 14:17:22 CST, Voxtelesys monitoring systems detected Origin Errors being reported by Cloudflare across various API endpoints and portal.voxtelesys.net. At 14:33 CST, Voxtelesys engineering pushed a mass DNS record change to the voxtelesys.net domain. By 14:36 CST, the DNS change had propagated to mainstream public DNS resolvers. This entire event lasted 20 minutes. We have engaged Cloudflare's Enterprise Support and are awaiting a much-needed explanation to the logs provided to us. We apologize for the inconvenience and will continue to monitor closely for any further issues.

A temporary fix has been implemented. If you are still seeing issues, please flush any DNS resolver cache. If the issue persists please contact support@voxtelesys.com.

Voxtelesys NOC has received reports of intermittent issues with SMS and reaching the portal. Voxtelesys network engineering is currently investigating.

All 3CX VM images were live migrated successfully and no downtime was observed.

We have identified an issue on one of the physical machine's in the hosting clusters. In order to minimize any disruption to service, we will be migrating the 3CX's on that node to a healthy physical machine while we perform maintenance. In order to minimize downtime, we will attempt to perform live migrations in all possible cases. A live migration would result in no downtime, but minor performance decreases in disk I/O and a second or less of packet loss as the network is switched over to the new physical host. In the event that we are not able to perform a live migration of a 3CX, we will perform an offline migration during which the 3CX will be powered off and migrated offline during the maintenance window. We apologize for any inconvenience and will be happy to answer any questions or concerns – simply call 701-929-9797 or open a support ticket through your portal account.

Further investigation showed a fiber cut which has now been spliced.

Voxtelesys NOC was alerted of packetloss affecting our DFW datacenter. The issue was isolated to a single Tier-1 upstream transit provider, which we immediately withdrew from our BGP routing. We are continuing to monitor, but at this point the packetloss has cleared up.

This incident has been resolved.

At this time, the service impact event has been remediated by the soft reset of the upstream core router at our SLC facility. If any customer continues to experience issues after re-testing their service availability post 2:15 pm CST, they should contact Voxtelesys Support with an update to their service ticket.

We are continuing to monitor for any further issues.

Between 12:30-12:45PM, we observed intermittent congestion and packet loss at our SLC datacenter. At this time, congestion and packet loss have stabilized. We will continue to monitor the issue.

Starting around 2PM CDT, we were alerted of unusual inbound Toll Free traffic patterns affecting numbers provisioned with AT&T. We immediately removed AT&T by activating our disaster recovery templates that exclude AT&T. The migration took approximately 30-45 minutes for all traffic to be off of AT&T. After removing AT&T from routing, we were informed that AT&T experienced a failure at the core of their network. Once AT&T verifies a fix has been implemented, we will confirm via in-house tests and metrics before adding AT&T back to our Toll Free templates.

This incident has been resolved.

Voxtelesys was alerted of packetloss affecting our SLC datacenter. After investigation, the issue was isolated to a single ISP which was immediately removed from our BGP routing. At this point the packetloss has cleared up. We will continue to monitor.

All services are fully restored. We are continuing to monitor.

Status Update: All issues are resolved with the exception of one DID provider failing to deliver calls. We are on the phone with them now and hope for a speedy resolution.

***Urgent Notification for STATIC IP AUTHENTICATION!! The Data Center's Core IP network is still failing. Because of this extended outage, Voxtelesys has moved all services to a alternate data center. 1. Registration authentication customers should not be impacted. 2. Customers using Static IP Authentication will need to point to our new Static IP. Please change to NEW STATIC IP AUTHENTICATION to 192.41.56.156 for both outbound and inbound calls.

We identified a problem with the Data Center's core router. Engineers are on-site and working on the issue. Please stand by for updates.