Historical record of incidents for Flowroute
Report: "Failed SMS/MMS Handoffs to 3CX Webhooks"
Last updateThis incident has been resolved.
We are receiving reports of completing messages through 3CX. We are continuing to work with them to ensure the issue is resolved.
We are still actively working on the issues with failed SMS/MMS handoffs to 3CX Webhooks. Our in-house development team has been notified and is currently investigating the matter further. Additionally, we have provided updates to 3CX, as other customers not using 3CX PBX systems are not experiencing any issues with SMS/MMS handoffs. At this time, it appears the issue is only affecting 3CX PBX systems. We will provide an update as soon as we have more information.
We have reached out to the 3CX Team for investigation and will update this event as new information becomes available. At this time only 3CX webhooks are experiencing failures.
We are looking into reports of inbound SMS/MMS that are being rejected when handed off to 3CX webhooks.
Report: "Call Failures - Canadian Peering Networks"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Peering networks connecting traffic to local Canadian services have reported an outage with their network. They are aware of the issue and have deployed resources to investigate.
Report: "Intermittent Inbound Call Failures Impacting AT&T Customers in Southeast Florida"
Last updateThis incident has been resolved.
A fix has been implemented by the upstream carrier, and we are currently monitoring the results.
Our peer networks have identified the issue and are working on a fix.
We are continuing to work with our peer networks to address this issue.
We are continuing to work with our peer networks to address this issue.
We are continuing to work with our peer networks to address this issue.
We are actively collaborating with our peer networks to resolve the intermittent inbound call failures impacting AT&T customers, particularly in Southeast Florida, including the Miami and Melbourne areas. This issue appears to be localized to this region. We will continue to monitor the situation closely and provide updates as new information becomes available.
Report: "Intermittant Inbound Call Failures AT&T"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our peer networks have identified the issue and are working on a fix.
We are continuing to work with our peer networks to address this issue.
We are actively working with our peer networks to address the intermittent inbound call failures affecting AT&T customers, specifically in Southeast Florida, including the Miami and Melbourne areas. This issue appears to be localized to that region. We will continue to monitor the situation and provide updates as we receive them.
Report: "Inbound call failure in the Dallas area"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work with our peer networks to investigate this issue.
We are continuing to work with our peer networks to investigate this issue.
We are investigating intermittent inbound call failures in the Dallas TX LATA 552.
Report: "Flowroute Manage Portal Errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue with the Flowroute Manage Portal providing 500 errors.
Report: "Issues with receiving email to support@flowroute.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue with receiving email to support@flowroute.com.
Report: "Flowroute Support number 18553569768 busy signal"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are working with internal teams to identify an issue with the Flowroute support number 18553569768 that is providing a busy signal on inbound calls.
Report: "E911 Profile Provisioning Failures"
Last updateThis incident has been resolved.
Emergency Service Partners are currently investigating an issue that is preventing E911 profile requests from provisioning/updating. This does NOT impact 911 call routing.
Report: "Issue with Flowroute ticketing system"
Last updateThis incident has been resolved.
Email sending from our ticketing system has been restored. We will continue to monitor the situation.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue with our ticketing system and are unable to send outbound email.
Report: "Inbound calls failing from AT&T in Florida"
Last updateThis incident has been resolved.
We have been made aware of inbound calls failing to reach our network from AT&T due to a fiber cut in the Florida area. We are receiving updates from peering networks that are monitoring the situation and will provide an update here once we see movement on the matter.
Report: "Auto-replenishment Charges"
Last updateOur team has reinstated Auto-Replenishments and reverted the erroneous charges for applicable accounts. This incident did NOT affect all accounts. Email notifications have been sent to those accounts that will be receiving refunds. Please allow 5-7 business days for those transactions to be reflected on bank statements.
Our team is still investigating the full scope of this event. We expect to be reverting associated auto-replenishment charges that were triggered within the next 48 hours. Those reverted charges may take several business days for the reversal of those transactions to be visible on credit card reports. We thank you for your patience with this as we look to correct the issue that was experienced.
We are reviewing the incident from yesterday where multiple Auto-Replenishments had been triggered and are working to understand the full scope of the event. We will provide further updates as we collect more information.
We are reviewing the incident from yesterday where multiple Auto-Replenishments had been triggered and are working to understand the full scope of the event. We will provide further updates as we collect more information.
Report: "Auto-replenishments"
Last updateOperations have returned to normal.
Our team has implemented a fix to prevent further auto-replenishment notification triggers. We are monitoring and will provide more updates as they become available.
We are currently investigating an issue where duplicate auto-replenishment emails are being triggered.
Report: "CDR Exports"
Last updateServices are restored and CDRs are now available for export.
Our team is aware of an issue that is currently preventing CDR Exports from being generated. Resources are investigating and we'll provide updates as those become available.
Report: "E911 Provisioning for Canadian Addresses"
Last updateOperations are back to normal and this incident has been resolved.
The 3rd-party vendor experiencing issues has implemented a fix and services are being monitored.
Emergency Service Partners are investigating an issue with address provisioning for Canadian addresses due to issues with a 3rd-party vendor. Emergency calls are still routing appropriately.
Report: "Static/Choppy Audio on Toll Free call"
Last updateThis incident has been resolved.
An IP network was identified in the Southern part of the US East Coast to have issues which have sense cleared and traffic appears normal. We will continue to monitor to the EOB today.
Our Toll Free peer is reporting they are working with the underlying carrier on an issue causing static or choppy audio on Toll Free calls. This is centered on the Southern part of the US East Coast.
Report: "SMS/MMS Delivery issues to T-Mobile and Verizon"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Reports from messaging partners indicates that the issue may be clearing at this time with normal delivery observed to Verizon and T-Mobile. We continue to work with messaging peers to identify the root cause of this issue. Note: This was impacting traffic in the 10DLC ecosystem itself and not isolated to Flowroute services or messaging partners.
Our messaging partners have informed us of possible SMS/MMS delivery issues to T-Mobile and Verizon numbers. This is currently under investigation. We are monitoring and will keep you informed on further updates.
Report: "Delays when Exporting CDRs"
Last updateThis incident has been resolved.
Our team is monitoring an issue with vendors that may cause delays with CDR Exports while pulling the requested information. We will update this event as we receive further word from our team.
Report: "Issues with AT&T Calling Flowroute OK Numbers"
Last updateThis incident has been resolved.
We have been informed that AT&T has repaired the circuit. Our peer is monitoring it. We are waiting for the official word that it is resolved. Thank you for your patience.
Our peer has informed us that AT&T is actively working on the affected service. There is currently no ETR. We will provide updates as we receive them. Thank you!
Our peer is continuing to work with AT&T on restoring calling to Oklahoma. We will continue to post updates as we receive them. Thank you.
Our peer has let us know that they are working with AT&T to restore calling to Oklahoma. We will updates as we receive them. Thank you.
Our peer continues to work with their upstream carrier to investigate the issue with AT&T. We will provide an update as soon as we have one. Thank you.
Our peer has let us know that there is a known issue with AT&T numbers unable to connect to Flowroute numbers in Oklahoma. Our peer is working with AT&T on this. We will provide updates as soon as we receive them. Thank you for your patience in this matter.
Report: "E911 Provisioning"
Last updateThis incident has been resolved.
Emergency Service Partners are currently investigating an issue that may prevent the Addition/Deletion of E911 profile entries. This DOES NOT impact routing of 911 traffic.
Report: "E911 Profile Provisioning"
Last updateThis incident has been resolved.
Emergency Service Partners are currently investigating an issue that may prevent the Addition/Deletion of E911 profile entries. This does not impact routing of 911 traffic.
Report: "Intermittent 400-level Errors for outbound SMS and MMS"
Last updateThis incident has been resolved.
A fix has been implemented and we are reviewing the results.
We are working with our SMS/MMS peer to investigate intermittent 400-level errors on outbound SMS and MMS traffic.
Report: "Unable to forward calls from Canadian numbers to Canadian numbers"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring results.
We have identified an issue where if a Canadian DID is forwarded to another Canadian DID through Manage both sides are getting dead air. This is an issue with the terminating carrier. To work around this please use SIP Registration, SIP URI, or Host-based routing for your inbound calls on Canadian DIDs.
Report: "E911 Profile Provisioning"
Last updateThis incident has been resolved.
Our Emergency Service Partners are continuing to investigate this outage. We will provide more information as we receive it.
Emergency Service Partners are currently investigating an issue that may prevent the Addition/Deletion of E911 profile entries. This does not impact routing of 911 traffic.
Report: "CDR Exports"
Last updateThis incident has been resolved.
A fix has been applied and CDR Exports appear to be operational. We'll continue monitoring this further.
A fix has been implemented and we are monitoring the results.
Our team is aware of an issue that is currently preventing CDR Exports from being generated. Resources are working to apply a resolution.
Report: "Ticketing system"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue with our ticketing system. This affects our ability to answer tickets through email and our ability to answer support calls.
Report: "www.flowroute.com went down"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring it.
Report: "Intermittent Inbound Failures from calls originating with AT&T"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have received several reports of calls that originate on the AT&T network are getting prompted with a "High Volume" message indicating that there may be limited connection of calls at this time. The messages from the AT&T side then go on to indicate a digit entry may be needed to connect calls in the event of an emergency. We've reached out to carriers in the call path for further investigation on this and will provide updates as those become available.
Report: "E911 Provisioning"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Emergency Service Partners are currently investigating a matter on their end that is impacting the creation and editing of E911 profiles through the Flowroute Portal page, as well as through the use of the API. We'll provide more updates as they become available.
Report: "Issues with T-Mobile reaching Flowroute numbers"
Last updateThis incident has been resolved. Thank you for your patience.
Our peer has informed us that T-Mobile has resolved this issue. We are having customers retest and confirm.
Our peer has notified us that issue has been identified and T-Mobile and their peer are working on resolving this. The current estimated time to repair is 3 - 4 hours. We will continue to post updates as we receive them. Thank you for your patience in this matter.
There is an issue with T-Mobile cell users reaching Flowroute numbers. We have engaged the carrier and they are speaking with T-Mobile. We will post updates as we receive them.
Report: "Issues with Receiving T-Mobile Calls"
Last updateOur peer has identified the issue and is saying the problem should be resolved. Thank you for your patience in this matter.
Our peer has informed us that T-Mobile is working with their underlying carrier to identify and resolve this.
Our peer continues to investigate the root cause of this issue. We will provide more updates as we receive them.
We are experiencing issues with receiving calls from T-Mobile cell users where calls take multiple tries going through. We have engaged our peer and they have confirmed that they are having issues with calls from T-Mobile. They are investigating this and have no current ETTR. We will provide updates as we receive them.
Report: "Inbound routing affecting 3CX users"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating inbound routing over registration that is affecting 3CX PBXs.
Report: "MMS Message Latency - Outbound"
Last updateThis incident has been resolved
Messaging peers are experiencing issues with outbound MMS Latency to T-Mobile and Verizon within the US, as well as MMS traffic to term side parties in the Canadian market.
Report: "AMEX payment issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently working with our payment processor to investigate an issue with American Express credit cards.
Report: "Inbound MMS Delivery Latency"
Last updateMessaging peers have been informed of, identified the cause, and implemented a fix to remediate issues experienced with inbound MMS latency between late July 13th and into July 14th. During this incident inbound MMS may have experienced lengthy delivery delays and/or failure to deliver messages during the event. We would like to thank everyone for their patience while our messaging peers investigated the event.
Report: "Messaging Latency to AT&T"
Last updateMessaging Peer has noted that this issue regarding periods of intermittent latency should now be resolved following their monitoring window.
Messaging Peer has noted that latency issues should now be remediated and messaging behavior should return to normal. They will continue monitoring for the time being and provide updates when deemed fully resolved.
Messaging peers noted that an event was experienced causing latency to AT&T destinations along with numbers on AT&T's extended carrier network. The issue has been identified and they are working with AT&T to implement a fix.
Report: "Message Deliverability/Latency - Canadian Peering Networks"
Last updateThis incident has been resolved.
Messaging peers have been informed of and identified an issue associated with messaging binds for traffic involving Canadian numbers. Inbound, outbound, and Delivery Reports may be impacted during this event.
Report: "Call Quality Issues"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Our Peer has identified the issue and has implemented a fix. We are asking customers to please retest.
The issue has been identified and a fix is being implemented.
This issue has been escalated with our peer.
We are continuing to investigate this issue.
We are getting reports of call quality issues with a downstream peering carrier. We are working with our peer to get this resolved asap.
Report: "Call Quality Issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently experiencing call quality issues on certain inbound calls. We have engaged our peer and escalated with our peer and currently waiting for an update. We will post updates as receive them. We apologize for this inconvenience and are working diligently to resolve this.
Report: "Messaging Latency - 05/16/2022"
Last updateOur team noticed minor latency with messages being delivered from ~5:15-5:45pm PST, May 16th, 2022. Queued messages should now be cleared and delivery has returned to normal.
Report: "Packet loss at our East NJ PoP"
Last update**Investigation results:** On April 14, 2022 we were provided with the official investigation results from a Senior Interop Engineer at 3CX: > After looking into this we found that the 147.75.65.0/24 range had been added in our global blacklist. We have now removed that value but it will take approximately 6 hours for all systems to receive the change. Normally by now all affected systems must have switched to one of the other IP addresses so it should be OK. If any more customers have issues please let us know. > ---Your 3CX Interop Team **Scope of impact:** This incident affected some 3CX PBX systems that were directly connected to the New Jersey Edge proxy \([us-east-nj.sip.flowroute.com](http://us-east-nj.sip.flowroute.com), 147.75.65.192/28\) by IP address or had cached DNS records. **Mitigation plan:** 1. Immediately escalate incidents to all involved parties \(3CX Support, Flowroute Support\) 2. Whitelist all IP ranges listed in [Firewall Requirements](https://manage.flowroute.com/accounts/interconnection/setup/) 3. Avoid using direct IP connections and/or cashed DNS records when it’s possible
Our engineering team has done extensive testing and has not been able to reproduce the reported packet loss at the East NJ PoP. The customers that had issues with this PoP had their issues resolved when switching to the East VA PoP.
We are continuing to investigate this issue.
We are currently investigating reports of outbound packet loss at our East NJ PoP.
Report: "Message Latency - Canadian Peers"
Last updateAfter a period of monitoring SMS traffic messaging networks have noted that delivery timeframes are back to normal and this incident is being marked as Resolved.
Messaging Peers are currently monitoring an implemented fix to address latency with SMS MO / MT messaging from/to Canadian Messaging Networks.
Report: "Messaging Latency with Outbound SMS/MMS to AT&T Customers"
Last updateIssues with latency for outbound SMS/MMS and DLRs between our Messaging Peer and AT&T have been resolved.
We are continuing to monitor for any further issues.
Carrier has applied a fix and the queued messaging is being processed. Monitoring will take place before this is announced as Resolved.
AT&T has identified the issue and is investigating further. Messaging peer is queuing messages until the carrier can apply a remedy. Outbound SMS/MMS/DLR associated with AT&T are the only items affected at this time.
We are currently observing delayed Outbound messaging to AT&T customers, as well as latency with DLRs. There are no issues with Inbound at this time.
Report: "T-Mobile Messaging Latency"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently observing delayed messaging with T-Mobile. Our Messaging Peer is investigating the delays with T-Mobile operations. Delivery Receipts have been disabled at this time until a complete resolution is in place.
Report: "Emergency Services Routing Maintenance Window (22:00-23:00 MST, 2022-03-01)"
Last updateApologies for this incident notification. This is scheduled maintenance for March 1st, 2022. You will receive appropriate notifications going forward.
Emergency Service Partners will be performing planned maintenance on routing systems. No outage is anticipated in relation to this planned activity.
Report: "E911 Address Provisioning Degraded - 20220203"
Last updateThis incident has been resolved.
Provisioning has been restored. Efforts to monitor and mitigate intermittent provisioning with our vendor will continue.
We are continuing to work on a fix for this issue.
Our vendor has the issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
There is an ongoing E911 address provisioning outage. This is linked to a maintenance window with a vendor. This issue is being investigated and updates will be provided as they become available.
Report: "Support Portal Issues"
Last updateSupport.Flowroute.com is operational
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results. We discovered there was an outage on LiveAgent's EU and US-based accounts.
We're seeing HTTP ERROR 500 when trying to access the Support Portal support.flowroute.com. We're currently investigating this issue.
Report: "Messaging Delayed"
Last updateThe incident has been resolved, all queued messages have drained and the issue is now resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are investigating and issue with Outbound Messages being delayed.
Report: "Customer Support ticketing system upgrade - delay in response time"
Last updateThis incident has been resolved.
The ticketing system for Customer Support is currently undergoing a system upgrade. There will be minor down time, during which phones will be out of service. Forms and email tickets are queuing and we will respond to them when the upgrade is complete.
Report: "Messaging failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue that is causing messaging failures. Updates will follow.