Historical record of incidents for QuadraNet
Report: "DTLA Maintenance"
Last updateThis incident has been resolved.
Dear Customers, Thank you for your continued patience as we work through our maintenance in our Los Angeles facilities. The issues have affected multiple services at both locations. We are working 24/7 to resolve this, but as expected this is not enough to resolve all. Once your services are restored for our dedicated server customers only, please submit for an SLA credit via the Neo portal. For further escalations, you can contact our CEO and Owner Jon Eaves, directly at jon.eaves@quadranet.com. We apologize for the extended duration and any downtime you have experienced.
The maintenance window has taken longer than expected, but we have a team on-site working diligently to get every customer back up and running. Please refrain from opening multiple tickets or calling into the data center. Our team is doing everything they can to assist each customer as quickly as possible.
Our team is making excellent progress in getting all bare metal customers back online. Some single colocation customers are still experiencing issues, and we have all hands on deck to restore service for everyone affected as quickly as possible.
The recovery operations still in progress, more detailed report shall be rendered by the end of the event. We appreciate your patience.
Our team has identified some connectivity issues relating to our maintenance being performed today. We are working to resolve this as quickly as possible.
We appreciate your continued patience while we work though this maintenance window. We hope to have everyone back online as quickly as possible.
Thank you for your continued patience while we work through this maintenance window. Our team has run into a few challenges that were not a planned part of the maintenance. We appreciate your business and understanding.
We are still undergoing our network and cabinet restructuring at our Los Angeles facility and will provide additional updates as they become available.
Please note, we are restricting all non critical access to the data center during this time.
We are currently undergoing our first round of maintenance at our DTLA facility. We are half way through this process and customers should start to regain access in waves shortly.
Report: "Chicago Network Outage"
Last updateThis incident has been resolved.
Our team has been able to restore service connectivity to 90% of effected customers. We hope to have this finalized by end of day.
Thank you for your continued patience. Our on-site team has replaced a core device. Our networking team will begin modifying the configuration and get service restored shortly.
We appreciate your patience during the downtime in Chicago. We are in constant communication with our on-site partners and are urgently working to restore service as quickly as possible.
We have identified an issue with our Core Node around 3PM PST today and are actively working with our team to get this back online. We apologize for the delay.
Report: "Urgent Update: Service Disruption in LAX Facilities Due to Cogent Network Issue"
Last updateCogent was having memory issues in their Core Routers at the Data Center and an upgrade was completed to resolve the issue. This issue has now been resolved.
This incident has been resolved.
There is currently an ongoing connectivity issues impacting our LAX facilities, which we’ve traced to a critical network disruption involving Cogent Communications. Please accept our sincere apologies for the inconvenience this has caused. Current Status: Our engineering team is working closely with Cogent Communication to resolve the disruption affecting both of our Los Angeles peering points. We’ve escalated the issue to Cogent’s senior technical team and are collaborating closely with their engineers to expedite resolution. At this time, Cogent has confirmed the root cause and is prioritizing repairs. Next Steps: We will provide a detailed update as soon as possible. We deeply regret any inconvenience this may have caused and appreciate your patience.
Report: "WHMCS access situation"
Last updateThis incident has been resolved.
Dear Customers, For those amongst our clients who are using the WHMCS part of our CRM. For the time being as a temporary solution, please use our legacy NEO portal for any helpdesk requirements and service communications via https://neo.quadranet.com, if you don't have the logins to NEO, you may simply email to support@quadranet.com. We will keep you up to date once WHMCS issue has been rectified. Apologies for the inconvenience caused, QuadraNet Team
Report: "Networking Related Issue"
Last updateThis incident has been resolved.
We are currently experiencing a networking issue. This is being worked on by our networking engineers, once resolved a RFO will be provided.
Report: "Dallas Outage"
Last updateThis incident has been resolved.
Our team has successfully replaced the Aggregation Switch, and services are gradually coming back online. If you are still experiencing any network-related issues, please open a ticket in the support portal.
We are currently addressing a power issue with our Aggregation Switch. To expedite the resolution, we have brought in additional network engineers to our data center. Our team is working diligently, and we are confident that service will be restored as quickly as possible.
We are currently experiencing a power issue with our Aggregation Switch. Our team is fully engaged, both locally and remotely, to resolve this issue as quickly as possible. We are confident that service will be restored within the next 12 hours.
We are experiencing a power issue with our Aggregation Switch. Our team is actively working on a solution to restore service as quickly as possible. At this time, we do not have an estimated time of resolution.
Report: "LAX100 DWDM Ring (Partial Outage)"
Last updateWe are in the process of transitioning the remaining individual channels for specific customer's to our 530 W. DWDM side of the ring at this time. Ongoing communication for those customer's will be conducted via QuadraNet's support portal as the broader issue from this incident has been repaired. https://neo.quadranet.com 1-888-5-QUADRA
Please find an update below: Our fiber engineering team has identified damage to the fiber and is dispatching a repair crew to repair the damage. We will provide updates as they become available. Regards, Crown Castle Fiber NOC
Connectivity has been restored by isolating the impacted DWDM path. We are still working to restore a small amount of customers that utilize direct DWDM channels on our 624 S Grand Multiplexer, however.
We are currently experiencing an outage on one of our DWDM legs that connect between 624 S Grand and 6171 W Century BLVD. Issue is being investigated now.
Report: "Miami Network Impact"
Last updateWe are pleased to inform you that connectivity has been fully restored. Our network team is in the process of compiling a detailed incident report, which will be available for customers upon request. If you require this report, please submit a formal request through the NEO portal. Thank you for your patience and understanding.
We experienced a brief recovery, but the network remains unstable. Our teams are fully engaged and working diligently to resolve the issue. We will continue to monitor the situation closely and provide updates as we make progress. Thank you for your patience, and we apologize for any inconvenience this may cause.
The network is back to normal, currently being monitored.
We are aware of ongoing network interruptions affecting our Miami datacenter. Our team is actively investigating the issue and working diligently to restore full service as soon as possible. We appreciate your patience and will provide further updates as progress is made. Thank you for your understanding.
Report: "Chicago Network Impact"
Last updateRoot Cause: NTT upstream was down causing a routing issue leading to a short outage. Status: Network is back to normal state. Next Step: Our team is monitoring the situation and inspecting the network to ensure this issue doesn't occur again.
We are aware of the ongoing network issue in our Chicago facility. Our senior network architect is actively working with the onsite team to isolate and resolve the problem. Further updates will be provided as progress is made. Thank you for your patience.
Report: "[DFW] Restoration Complete - Monitoring"
Last updateThis incident has been resolved.
We have made repairs to our critical network infrastructure in order to restore primary public network connectivity to downstream customers, we are still monitoring the situation at this time so this case will remain open until further notice. If you are still experiencing public traffic connectivity difficulties, please open a support ticket with QuadraNet at https://neo.quadranet.com
A fix has been implemented and we are monitoring the results.
QuadraNet's CORE router has failed to energize after the abrupt power failure in the Dallas Datacenter. We are currently working on a solution at this time.
Power has been restored in the Dallas datacenter and we have partially been restored. QuadraNet is continuing to work with onsite engineers to resolve any outstanding issues. Thank you.
Report: "[DFW] Utility Issues / Heat Issues"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
At approximately 11:00 AM CT, The DAL data center proactively transferred to generator power on a ATS3 due of excessive temperatures on one of the upstream utility transformers. The transfer was successful. After further investigation of the transformer, insulation damage showing exposed electrical conductors was discovered and Oncor Electric, our upstream utility provider disconnected the utility feed to enable further investigation of the transformer. At approximately 1:58 PM CT, the generator servicing ATS 3 had a fan belt failure. As a result, any non-redundant, single corded critical loads associated with UPS #10 and #12 ran out of battery reserve at approximately 2:24 PM CT. Generator and Electrical vendors are actively engaged. A spare generator fan belt is on site and the generator vendor is currently working to replace the failed component. MTTR on the belt replacement is approximately 3:30 PM CT. Due to the utility and subsequent generator failure, temperatures in the data center will increase. The local team is currently working to re-feed HVAC units, associated with ATS 3, from an alternate power source within the facility. The HVAC refeed process should be complete by 3:45 PM CT.
Report: "NEO Portal Availability Notice"
Last updateUpdate: NEO portal issue solved. Full functionality has been restored. Thank you for your patience!
The NEO portal is currently experiencing partial outages. While some features may be operational, others are not functioning properly. We are aware of this issue and are actively working to isolate and restore full functionality. If you encounter any difficulties performing urgent actions, such as a remote OS reinstallation, please contact us directly for assistance. Should you be unable to file a request via NEO, you may call our 24x7 NOC at 1-888-578-2372. We will continue to provide updates on this page as we make progress. Thank you for your patience and understanding.
Report: "Seattle Network Impact"
Last updateThis incident has been resolved.
Cogent network has been restored. Everything should be up now. Root cause: Cogent had a planned network maintenance which wasn't supposed to have any impact. Unfortunately - an unexpected impact has been occurred causing a total of two hours outage. We sincerely apologize for the inconvenience and thank you for your patience during this incident. Should you still experience any issues, please raise a ticket or call our support center. Thank you.
Our network team is working to activate a backup link while Cogent is offline. Update shall follow.
Currently, our network team has observed a network impact within our Seattle facility, could cause some parts of our Seattle network to be unreachable. This is being investigated and an update shall follow.
Report: "[EWR] - Network Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating a potential network issue that is impacting our New Jersey (EWR) datacenter and we are working towards a resolution.
Report: "Network Interruption in Miami"
Last updateIssue: We experienced connectivity issues due to a TeleFornia upstream issue. Resolution: The issue has been resolved. All traffic has been successfully rerouted. Outage Period: From 1:30 AM PST until 2:50 AM PST. Status: Resolved.
Update: The initial root cause appears to be stemming from the upstream provider. Our team is actively investigating the matter. We appreciate your patience and understanding.
Currently observed network interruption across several sections of Miami network, our engineers working on this and more details to follow. Thank you.
Report: "Emergency Maintenance to QuadraNet Portal Services"
Last updateCompleted.
Maintenance completed. Portals are back to normal. Thank you.
Customer fronting portals are currently offline for emergency maintenance. We apologies for any inconvenience caused. Thank you.
Report: "Seattle Partial Impact"
Last updateUpdate: Status: Network is now recovered to normal state. Outage Period: 2:15 AM - 2:45 AM PST / 30 minutes of partial network outage (on/off) Root Cause: Core router degraded performance (was very slowly responding to any input). Solution: A hard reboot of the router was needed. -------------------- Next Step: Network team will be monitoring the network performance to ensure stability and take any actions needed if any further anomalies to be detected. -------------------- We apologize for any inconvenience this outage may have caused. Thank you for your patience. Thank you.
Our engineers are currently investigating an issue that is causing partial network outages in our Seattle location. The root cause appears to be an anomaly at the core router level. We apologize for any inconvenience this may be causing. Our team is working diligently to identify the problem and implement a fix as quickly as possible. We will provide further updates here on the status page as we have more information to share. Thank you for your patience while we work to restore full network connectivity in Seattle.
Report: "Network Outage in Seattle"
Last updateThe total outage time was from 0h20 AM to 3h AM (PST). The network has been confirmed as stable for now. Our team is still investigating to get the complete details on the root cause. We will provide further updates once we have finalised the investigation. Thank you for your patience and understanding during this outage. Best regards, The QuadraNet Team
We are pleased to report that the network has returned to normal operation since the last 15 minutes. Our team is still investigating with Seattle to get the complete details on the root cause. Thank you for your patience and understanding during this outage. We will provide further updates as more information becomes available. Best regards, The QuadraNet Team
We are currently experiencing a network outage in our Seattle location. Our network team is actively investigating the issue and working to resolve it as quickly as possible. We apologize for any inconvenience this may cause and will provide updates as more information becomes available. Thank you for your patience and understanding.
Report: "Network Stability Impact in Seattle"
Last updateThe issue has been resolved and our network is now functioning normally. Our network team is currently working with Seattle/Cogent to determine the underlying cause of the unexpected outage. Duration of Total Outage: 1 hour (from 4:13 AM to 5:15 AM) We sincerely apologize for any inconvenience caused and appreciate your patience and understanding during this time. Thank you for your cooperation!
We are continuing to investigate this issue.
We are writing to inform you that our Seattle location has been experiencing network stability issues for the past 30 minutes. Initial observations indicate that the issue is related to a recent upgrade that was performed by one of our bandwidth providers, Cogent. Cogent was upgrading one of the route processors on their core and aggregation routers, as recommended by the manufacturer. This work was supposed to begin at 9:00 AM UTC on June 22, 2023, and was not expected to impact our customers. However, we are now experiencing some network instability as a result of this upgrade. Our network engineers are currently working with the Seattle and Cogent teams to isolate the issue and restore full service as soon as possible. We apologize for any inconvenience this may cause. We will provide updates as they become available. Thank you for your patience.
Report: "Chicago Minor Impact, Partial Packet Loss."
Last updateFaulty cable connected the switch caused packet loss. Cable replacement fixed the problem. Total partial packet loss time: 3 Hours Absolute outage time: 55 Minutes
Faulty cable replaced. No further partial packet loss can be observed. We will continue to monitor to ensure the link is healthy. Thank you for your patience during this incident. Incident status: Resolved.
Certain devices may experience a complete packet loss as our engineers replacing a faulty cable at this very moment. We will report back once replacement is accomplished.
Update: Our network engineers are still working on the incident. We would like to clarify that the network is not down, it is just a minor section of our Chicago network with a partial packet loss.
We have identified a small section of the Chicago network with unstable connectivity, which is causing partial packet loss. Network engineers are working to isolate the issue and restore full connectivity. Updates will follow.
Report: "Chicago Network Stability"
Last updateIssue: Partial packet loss in Chicago Date: 2023-05-18 Time: 10:00 PM - 3:15 AM PST Total Impact: 5h:15 minutes Root Cause: Distributed Denial of Service (DDoS) attack Course of Action: Mitigated the DDoS attack. Action Plan: An internal meeting will be held between network engineers and the facility CTO to discuss the impact of the DDoS attack and enhance the mitigation process to minimize future impacts. Thank you for your patience during this time. We appreciate your business. If you have any questions or concerns, please do not hesitate to contact us.
Our network team is still working on resolving the partial packet loss issue. More updates to follow. Thank you.
Our monitoring has detected network stability issues causing partial packet loss across certain sections of our Chicago network. Our network engineers already investigating the anomaly and further updates shall follow.
Report: "Miami: Partial packet loss on TELEFONICA upstream"
Last updateIncident: Partial packet loss on TELEFONICA upstream in Miami Date: 2023-04-28 Time: 0:55 AM PST Resolution: Resolved at: 1:25 AM PST Total Impact: Approx. 30 Minutes Impact: - Partial packet loss on outbound traffic for clients routed on TELEFONICA upstream - No impact on inbound traffic - No impact on clients routed on other upstreams Details: We experienced partial packet loss on our TELEFONICA upstream for about 30 minutes, from 0:55 AM PST to 1:25 AM PST. We quickly identified and resolved the issue, and all services are now restored. We apologize for any inconvenience this may have caused. As always, we are committed to providing our customers with a reliable and stable network. Next Steps: We will continue to monitor our network and will update this status page as needed.
Impact: The partial packet loss caused some disruption to our services. However, we were able to quickly resolve the issue and restore full service to our customers. Resolution: We deactivated the problematic upstream and are currently working with our upstream partner to resolve the issue. We will continue to monitor the stability of our network and will update this report as needed.
Currently investigating possible anomalies within some sections of our Miami network. Network stability issues being reported and within an on going investigation. Update shall follow.
Report: "AMS Network Outage"
Last updateThis incident has been resolved.
We are currently experiencing network connectivity disruptions in our Amsterdam datacenter and we are working to identify and resolve the problem.
Report: "LAX13 - UPS2 Power Incident"
Last updateThe issue has been identified as a faulty power supply unit located within LAX13-Z2.UPS-2 during the power transfer between sources. Currently the critical power vendor has replaced the faulty power supply unit and have normalized the unit back to UPS protected power at this time. Please open a ticket regarding this incident should you require additional information.
QuadraNet has identified a customer impacting power incident located in our LAX13 datacenter pod related to our UPS-2 equipment. Power has currently been normalized at this time and we are working with impacted customers via our ticketing system in order to resolve any issues that are currently outstanding for equipment. Please utilize the QuadraNet NEO control panel to open a ticket located at: https://neo.quadranet.com or email support@quadranet.com for additional assistance. We currently have a critical power vendor located onsite working to determine the root cause of this incident and additional information will be provided once it is available. Regards, QuadraNet Management
Report: "LAX9 - UPS2"
Last updateThe latest update was sent in error. The work for the UPS-2 repair has been completed in full and all customers have been returned to UPS protected power at this time. Thank you and please let us know if you need any further assistance.
The defective unit has been full removed from the site and we are still operating in maintenance bypass at this time. Estimated time to complete the work has been updated to: Saturday June 25th, 2022: 3:00PM PST. Please open a support ticket if you have any questions.
Critical Power Vendor has arrived at the datacenter to begin the removal of the defective UPS unit. No customer impact is expected as a result of this work at this time. We will provide updates as they become available.
Due to a defective Power Module in LAX9 UPS2, related to the critical power even that happened last night (6/22/2022). We have currently isolated this unit and removed it from our critical power topology. Any customer equipment connected (without redundancy) to LAX9-UPS2 is currently operating in an unprotected state. QuadraNet has secured a replacement unit expeditiously and we anticipate to have this incident resolve and the replacement unit installed as follows: Friday June 24th, 2022: Critical Power Vendor will arrive onsite to begin the process of removing the defective UPS unit. Friday June 24th, 2022: Critical Power Vendor will deliver the new unit onsite to QuadraNet's LAX9 datacenter zone and begin assembly. We anticipate all work to be completed by 11PM PST on 6/24/2022 - however, if this timeline changes we will provide updates. We have been assured by our vendor that all work will be completed by no later than 11PM PST on 6/25/2022. Thank you,
Report: "LAX100 Chiller Status"
Last updateThis incident has been resolved.
We are currently experiencing a fault with our chiller system and we have engaged our chiller vendor to perform an emergency dispatch to rectify the issue. We anticipate the vendors will arrive at approximately 6:30PM PST
Report: "Support Portal Maintenance"
Last updateThis incident has been resolved.
We are currently performing ongoing work on our support portal. You may notice times of inaccessibility during this work.
Report: "[ATL01] Power Outage"
Last updateAll repairs have been completed to our edge router and traffic has been gracefully shifted back to it without service interruption.
At this time we have restored network connectivity to our environment(s) and some customers have also been restored. At this time we will begin the recovery of customers that are not currently restored. If you have an open (pending) ticket regarding this incident, we will review your request and work to restore your service (if not already restored). The status of this situation is being changed at this time as QuadraNet has successfully deployed our disaster recovery playbook and we will continue efforts to restore customers who are still impacted by the original power failure.
QuadraNet has not been successful at this time in restoring our edge routing equipment from the power failure. At this time we will be exercising all available options in our disaster recovery plan in order to restore service to our customers. Updates will be provided once they are available.
At this time we have not been successful in restoring our routing equipment and we are still troubleshooting this with the datacenter engineers.
At this time QuadraNet network engineers are working with local engineers to restore service to our routing equipment. Due to the hard power disconnect, it has resulted in the failure of several components in our edge routing equipment in this datacenter. We will provide updates as they become available.
Engineers have been provided access to the building after being cleared by the Atlanta Fire Department. We are currently working directly with the engineers onsite to restore service.
The datacenter has informed us of the following: "A contractor working on the 2nd floor of the 55 Marietta building cut into a water main and flooded the 2nd and 1st level of the building. The flooding extended to the generator room which took down the datacenter generator and ATS (Automatic Transfer Switch). The Atlanta Fire Department is currently on the scene and is not allowing access to the building until it has been cleared for safety." QuadraNet is currently working diligently with our datacenter partner in the Atlanta location to restore service. We will provide additional updates as they become available. Thanks,
The issue has been identified and a fix is being implemented.
The datacenter at 55 Marietta has informed us of a power event that has impacted the entire datacenter floor. We are currently awaiting updates from the datacenter engineers.
Report: "BR2.LAX Emergency reboot"
Last updateissue is now resolved
Patching completed, traffic switched over to BR2 and we are monitoring.
All traffic moved to BR1 successfully with no service interruption, Patches are being applied on BR2
Our network team identified an issue with BR2.LAX100 ad require an emergency reboot to apply patches. network degradation is expected for under 5 minutes
Report: "LAX DC - 530 - Network Degradation"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Network Engineers have retracted announcements to our upstreams for the impacted prefixes at this time in order to restore network stability. We are currently monitoring the situation and will make any necessary adjustments as required.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
We are currently tracking a large network attack in our Los Angeles datacenter that is impacting network performance. Network Engineers are working diligently to restore service to impacted customers.
Report: "VEST DDoS Emergency Maintenance"
Last updateThis incident has been resolved.
Restoration time updated to: 7:30PM PST
QuadraNet has placed our VEST DDoS protection offering under emergency maintenance. During this maintenance -- all protection will be disabled and null routes will be applied to network attacks, rather then the VEST DDoS filtering service. We anticipate service restoration by: 6:30PM PST (Subject to change)
Report: "Fiber Cut"
Last updateZAYO has indicated that they will have their fiber cut repaired by potentially 2PM PST 3/13/2021 - QuadraNet at this time, has rerouted and migrated the impacted uplinks in order to restore service to impacted customers. Thanks,
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
Zayo dark fiber cut 20 miles away from facility, this issue effected only customer with Shared VLAN's across LAX100 <>DTLA facilities. We are working on a solution to remedy the situation via our redundant fibers.
Report: "LAX9 - Network Maintenance"
Last updateThis incident has been resolved.
We are performing an emergency maintenance in order to rollback a portion of the changes that were performed on Saturday (December 19th, 2020) You should expect partial degradation of service while traffic reshifts to the designated uplinks
Report: "LAX9 - Network Maintenance"
Last updateWe are currently investigating this issue.
Report: "LAX13 (DTLA - 530 W) Network Outage"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "[LAX - 530 W 6th ST] Onsite Visits Temporarily Restricted"
Last updateThis incident has been resolved.
Hello, In an effort to protect QuadraNet staff and our customer(s); At this time QuadraNet is restricting all onsite visits to our 530 W. 6th ST datacenter except for QuadraNet personnel due to on-going protests in the downtown area which may pose a risk. Any customer wishing to visit our datacenter should submit a request to support@quadranet.com instead. Thank you,
Report: "[*] - Network Performance"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
All system are still stable and running. if this continue we will close this incident ticket within the next 16 hours.
JTAC team discovered a new JunOS bug, patch applied VIA ISSU. Quadranet Engineers continue to monitor and work with the JTAC team. all systems are FULLY operational and we are keeping this incident report running for at least 24 hours to better updates our client.
Issue is Isolated to our Edge BR1 Router, we are working with the juniper JTAC team to isolate and solve the issue in hand. Current status is fully operational but we are monitoring the situation.
We are continuing to investigate this issue.
We have detected a network performance issue in our Downtown Los Angeles facility. Engineers have been notified and are currently working to identify the issue. At present, the issue appears to be isolated to Downtown Los Angeles. Customers with redundant uplinks may have not been impacted.
Report: "Amsterdam Degraded network"
Last updateNetworking performance has been restored. This was the result of a misconfiguration on our upstream's router that resulted in packets being dropped sparsely. If you are experiencing any further difficulties - please email support@quadranet.com
We are currently experiencing degraded network performance in our Amsterdam location. We are working to restore stable service at this time.
Report: "[Degraded Networking] New Jersey"
Last updateCogent has repaired the fiber cut and we have re-enabled our BGP sessions with AS174. Thank you for your patience.
Cogent Technicians are en-route to 110 B Meadowlands to investigate the outage. We are currently awaiting updates.
Cogent's status page has been updated to reflect that this outage is affecting 110 B Meadowlands Pkwy, Secaucus,NJ. There is still currently no ETR at this time for repairs.
This has been identified as a fiber cut on Cogent's side in NYC. They are currently working to repair the cut and have no ETR at this time. We are continuing to work with them to determine the ETR for service restoration. Updates will follow.
We are currently investigating degraded networking performance in New Jersey. Currently it appears that Cogent has had an outage upstream and we have retracted our routes with them until this issue has been resolved. We are continuing to work on this issue.
Report: "Miami - Power Disruption"
Last updateThis incident has been resolved.
Dear QuadraNet Customer(s), EMERGENCY UPS MAINTENANCE - SERVICE IMPACTING Location: 36 NE 2nd ST Miami, FL 33132 Date: Saturday, June 8th 2019 Start Time: 4:30PM PDT End Time: 5:30PM PDT Expected Outage: 10 minutes Risk: Critical load will be shut down and transferred to another UPS. Description: QuadraNet critical power vendors will be transferring the critical load out of wrap around bypass back to protected load on a secondary standby UPS unit. QuadraNet engineers are on standby to assist with bringing up equipment and assisting customers after the cut over. QuadraNet recommends that all customers power off their equipment during this maintenance window to avoid any data corruption. We will place a 20 minute notice on http://status.quadranet.com before the electricians transfer power back to protected load If you have any questions regarding the maintenance at any point, please contact our support department at support@quadranet.com or 1-888-5-QUADRA. We appreciate your understanding and patience during this maintenance and welcome all feedback. Thank You, Management Team QuadraNet Enterprises, LLC. www.QuadraNet.com 1-888-5-QUADRA support / sales @quadranet.com
We are continuing to work on a fix for this issue.
We have been able to temporarily restore service(s) until the electrical contractors arrive onsite. We are running in maintenance bypass at this time. Updates to follow.
A faulty input breaker has been detected for UPS-2. Electrical contractors are en route to the data center to transfer critical load to UPS-4 (Standby UPS). Current estimated time for electrical contractors to arrive onsite is 45 minutes.
We have detected a power disruption in our Miami facility. Engineers have been notified and are currently working to identify the issue. At present, power service to affected customers' equipment is offline, and the issue appears to be isolated to A powered cabinets. Redundant power service has not been impacted.
Report: "[New Jersey] Network Performance Degradation"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Network Engineers have traced the issue and have been able to repair it. A final reboot will commence now to fully restore the networking functionality in this location. CONNECTIVITY WILL BE INTERRUPTED DURING THIS EVENT.
Network Engineers have encountered an error with the firmware that was booted - a rollback has been commenced.
The EWR-CORE01 router is being upgraded now - internet connectivity will be disrupted during this event.
A firmware update is being pushed to the ERW-CORE01 to rectify the issue. More information to follow.
Network Engineers are currently investigating a network performance issue within our New Jersey facility that is interrupting network traffic for a handful of customers. More information to follow soon.
Report: "ORD-C1 to ORD-CH3 [Migration]"
Last updateAll devices are reporting online at this time and we are marking this incident as resolved. Please open up a ticket with us at support@quadranet.com.
All cabinets are energized and our team is working though a small handfull of devices that are still reporting down.
Hello, We are currently going to go over our maintenance window slightly - the final two cabinets are being cabled at this time and energized. We should have all equipment back online very soon.
Report: "Dallas Network Maintenance [Rev - 3]"
Last updateThis incident has been resolved.
All customers have been restored at this time.
CORE01 has been re cabled and sessions are beginning to restore. Customer access is being restored at this time.
Original CORE01 has been reracked into cabinet and blades are being inserted at this time. Further updates to follow.
The network topology is being reverted at this time due to extenuating circumstances with the limitations imposed on one of the new routing appliances. Further information will be provided once this has been completed.
An additional supplemental network appliance is being configuring now to offload routes from the impacted appliance. Network engineers are currently still working to remedy this issue.
The issue has been tracked back to a limitation on one of the routing appliances installed. The amount of IPv6 routes in the system have reached a hard limitation, which is resulting in some IPv4 routes being improperly managed by the appliance. Network engineers are currently reducing the amount of IPv6 routes in order to restore IPv4 functionality for all customers.
The issue has been identified and a fix is being implemented.
There are currently a small handful of devices still impacted due to a small announcement issue on one of our subnets. Network engineers are working to repair this - all other customers should have restored connectivity at this time, including Infracloud users.
Report: "DDoS Mitigation Outage"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Control Panel Upgrade"
Last updateThis issue is now resolved, the upgrade is complete. Apologies for the lack of notification, this will not occur again.
QuadraNet's control panel is experiencing a minor upgrade. Please reach out to 213-614-9371 x1 for support.
Report: "Century - Network Performance"
Last updateThis incident has been resolved.
At midnight site time we are going to make some modifications. We do not anticipate any issues.
A fix has been implemented and we are monitoring the results.
We have detected a network performance issue in our Century facility. Engineers have been notified and are currently working to identify the issue. At present, the issue appears to be isolated to a specific router. Customers with redundant uplinks have not been impacted.
Report: "[LAX100] Utility Power Outage"
Last updateUtility power was restored at 11:45AM local time and has been stable since that point. We are now marking this issue as resolved.
QuadraNet has been made aware of a utility power outage at our Century facility. At this time load has been transferred to our generators and no customer impact is expected. The ETA for power restoration is 1:30PM local time and we will provide an update if this changes.
Report: "LAX3-UPS1 - Power Outage"
Last updateThis incident has been resolved.
Utility power has been restored to all circuits fed from LAX3-C1-UPS1. Customer equipment is fully energized.
Work is done, being checked for faults and will energize in about 10 minutes.
We are waiting for a proper ETA from electricians. We apologize for the increasing estimates of completion.
Further delays in this repair, at least 30 minutes additional repair time.
The repair has been delayed by about 15 minutes. We still anticipate a 10:30AM restoration but the window has extended to 10:45AM.
Electricians expect power to be restored by 10:30AM. Updates to follow.
The problem has been back traced to a failed main breaker, electricians are currently replacing the breaker.
Technicians are currently working to restore power, bypassing the failed UPS unit.
A power outage has occurred on our LAX3-UPS1 system. We are currently working with our vendor to restore power to the unit.
Report: "ORD-C1 - Outage"
Last updateThis incident has been resolved.
The incident has been identified and resolved. Services are restoring now.
At this time we are investigating an outage in our Chicago facility. Further information will be provided as it is discovered.
Report: "Packet Loss in Miami"
Last updateThis issue is now resolved, the last of the providers has made adjustments to fully block attack traffic toward their infrastructure.
QuadraNet has discovered an issue affecting connectivity in the Miami area. We have reached out to our upstream providers and are working on a solution.