Is Bigleaf Networks Down Right Now? Check if there is a current outage ongoing.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

We are conducting routine maintenance on our Bigleaf Cloud Connect and Alerting system.Expected Impact: Network traffic should be unaffected during this time. Our Bigleaf Cloud Connect Dashboard may be inaccessible for up to 5 minutes. Network alerts may be delayed for up to 5 minutes.If you're curious about the types of maintenance that we perform, feel free to check out the Maintenance section of our SLA here: http://www.bigleaf.net/sla . If you have any questions or concerns about this maintenance, please let us know.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Bigleaf has identified an issue affecting some "Backup-Only" circuits where they cannot re-establish connectivity with the Bigleaf servers if the "Backup-Only" circuit goes offline. A fix to this issue is currently being pushed out to resolve the issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Bigleaf has identified an issue affecting some "Backup-Only" circuits where they cannot re-establish connectivity with the Bigleaf servers if the "Backup-Only" circuit goes offline.A fix to this issue is currently being pushed out to resolve the issue.

This incident has been resolved.

We are currently investigating network congestion issues that are impacting a subset of sites connected to our Frankfurt gateway cluster.

This incident has been resolved.

We are currently investigating network congestion issues that are impacting a subset of sites connected to our Frankfurt gateway cluster.

Customers connected to our NYC gateway clusters may have experienced up to two brief periods of packet loss between 12:12am PST and 12:22am PST.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating network congestion issues that are impacting a subset of sites connected to our NYC and CHI gateway clusters.

Our phone service provider has resolved the issue and inbound call service is functioning normally. We apologize for any inconvenience.

Our Bigleaf team is continuing to monitor an on-going issue with inbound call routing, but at this time we are only seeing some intermittent problems reaching Bigleaf via our (888) 244-3133 and (503) 985-8298 numbers.

We are experiencing some issues with our phone system provider that may cause difficulty in contacting Bigleaf Technical Support. Inbound calls to our main phone numbers at (888) 244-3133 and (503) 985-8298 may be affected. If you are experiencing this, please email our Support team at support@bigleaf.net. Our team is monitoring the emails, ticket requests, and voicemails. If you are experiencing an urgent issue please contact us immediately via email and we will contact you as soon as possible. Future updates will be posted here. Thanks for your patience!

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating network congestion issues that are impacting a subset of sites connected to our NYC and CHI gateway clusters.

We have identified the root cause and deployed a fix for the problem. All site and circuit alerts should now be sent out successfully.

We've found a problem with site and circuit alert emails from your Bigleaf service. These alerts have been delayed for the last 40 minutes. Production traffic is not impacted. We plan to provide an update on this incident within an hour.

This incident has been resolved.

A fix has been implemented and we are monitoring results

We are continuing to investigate this issue.

Bigleaf is currently investigating issues with Network Performance in our ATL and NYC regions. We are working to analyze and mitigate this problem. More updates to follow shortly.

This incident has been resolved.

Bigleaf is currently investigating issues with Network Performance in our Frankfurt, DE region. We are working to analyze and mitigate this problem as sites in this region are experiencing higher than normal latency.

This incident has been resolved.

A fix has been implemented and we are monitoring results

Bigleaf is currently investigating issues with Network Performance in our NYC region. We are working to analyze and mitigate this problem. More updates to follow shortly.

This incident has been resolved.

Bigleaf is currently monitoring network performance in the ATL region, but issues appear to be resolving.

Bigleaf is currently investigating issues with Network Performance in our ATL region. We are working to analyze and mitigate this problem. More updates to follow shortly.

There are some missing site metrics in the performance page of the web dashboard between 11:00am PDT and 2:00pm PDT. This issue has been resolved.

We've resolved all known reachability problems. If you are experiencing any access issues, or have questions, please don't hesitate to reach out to Bigleaf Support!

We’re currently investigating reachability problems to certain websites and services in our DEN PoP

We've resolved all known reachability problems. If you are experiencing any access issues, or have questions, please don't hesitate to reach out to Bigleaf Support!

ADP reachability is restored. We're continuing to investigate other potential reachability issues.

We’re currently investigating reachability problems to certain websites such as ADP.

We've resolved the reachability problems. From 9am to 12pm Central time, a few websites reachable through a direct peering route were being blackholed inadvertently. Avoiding that peering route resolved the issue. We'll follow up to ensure permanent resolution.

We've implemented a workaround and are evaluating the results of it.

We are currently investigating reachability problems to certain websites such as ADP and advisorcompass.

Our phone service provider has resolved the issue and inbound call service is functioning normally. We apologize for any inconvenience.

We are experiencing some issues with our phone system provider that may cause difficulty in contacting Bigleaf Technical Support. Inbound calls to our main phone numbers at (888) 244-3133 and (503) 985-8298 may be affected. If you are experiencing this, please email our Support team at support@bigleaf.net. Our team is monitoring the emails, ticket requests, and voicemails. If you are experiencing an urgent issue please contact us immediately via email and we will contact you as soon as possible. Future updates will be posted here. Thanks for your patience!

This issue has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue with our Web Dashboard live metrics and loading of performance metrics.

This incident has been resolved

We are currently investigating and working to resolve an issue with reaching our Support team via our (888) 244-3133 phone line as well as impacting emails to support@bigleaf.net

We are currently investigating an issue with reaching our Support team via our (888) 244-3133 phone line

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are now seeing the packet loss issues resolving in the ATL region. Our team is continuing to monitor.

We are currently investigating on-going packet loss on some circuits connected to our services in the ATL region.

This issue has been resolved.

We are currently working with our web team to resolve issues reaching our Bigleaf.net homepage. No impact to the Bigleaf Web Dashboard (https://app.bigleaf.net) or other service impacts for customers.

Our Provider's emergency maintenance window is completed.

One of our EU providers is performing ‘Emergency Hardware maintenance’ from 00:00 - 02:00 UTC in LON that has intermittently affected traffic to/from the LON POP. Our LON customers may experience some latency and possibly 2 incidents of service degradation that may each last 5-10 minutes. We are monitoring this situation closely.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate the issue. We have been able to implement a fix for impacted sites as they are identified. Please contact Bigleaf Support if you are encountering any connectivity issues to specific applications.

We are currently investigating some routing and reachability issues for certain applications reported by a subset of customers in relation to our San Jose, Chicago and New York POPs. We're currently investigating a root cause, and will provide an update as soon as possible.

This incident has been resolved.

Earlier today, from approximately 11:37 AM - 11:46 AM PT (18:37 - 18:46 UTC), our New York POP experienced a brief network connectivity disruption. During this event, a subset of customers may have experienced a brief increase in latency, jitter, packet loss while their service automatically rerouted traffic to their Backup POP. Affected customers automatically transferred their traffic back to NYC shortly after. We believe the issue has been resolved, however our Engineering team will continue to monitor for any additional anomalies.

This incident has been resolved.

Our notification system is back online, and the current Amazon Web Services incident appears to be resolved. Our engineering team has verified that all pending alerts and email notifications have been sent. We will continue monitoring the issue.

We are currently experiencing issues with our notification system for circuit and site events. It appears that this is due to an incident within Amazon Web Services, where this system is hosted. Notifications for issues may be delayed or not received. Customer network traffic should not be affected, as all customer network traffic runs through our owned-and-operated Cloud Access Network. We will continue to investigate and provide an update this incident as it unfolds.

This incident has been resolved. We have performed a software update to address the issue on the affected Tunnel Endpoint servers. We will be performing an urgent maintenance window later this evening to deploy the software update to the remaining subset of Tunnel Endpoint servers as a proactive measure.

Today our redundant network experienced multiple events on Tunnel Endpoint Servers within our Dallas and Seattle Gateway Clusters. Customers who were connected through the affected servers automatically failed over to their secondary POP. Our engineering team was automatically alerted and immediately began working to determine the cause of these events and restore the affected servers. When these servers were brought back online, affected customer sites automatically reconnected back to their primary servers. While connected to their secondary POP, affected customers may have experienced an increase in latency and/or packet loss. Our Engineers are actively investigating the cause of these events and we will provide updates as soon as we have additional information.

No further issues have been detected. This incident has been resolved.

We believe we have identified the issue and have taken corrective action to isolate the problem. Service has been restored to normal, and we do not believe there will be any further impact. Our Engineers will continue to monitoring the affected systems for anomalies this evening.

Today our redundant network experienced multiple events on two Tunnel Endpoint Servers within our Los Angeles Gateway Cluster. Customers who were connected through the affected servers automatically failed over to their secondary POP. Our engineering team was automatically alerted and immediately began working to determine the cause of the events and restore the affected servers. When these servers are brought back online, affected customer sites will automatically reconnect back to their primary servers. For the duration of this incident, affected customers may experience an increase in latency. Our Engineers are still actively investigating the cause of these events and we will provide updates as soon as we have additional information.

Earlier today, we experienced two brief network events affecting our ATL and DAL gateway clusters. Event Window 1: 11:19 to 11:22 PDT [18:19 to 18:22 UTC] Event Window 2: 11:38 to 11:42 PDT [18:38 to 18:42 UTC] During these events, customers connected to the affected POPs may have experienced increased latency, packet loss, or other connectivity issues. We believe the issue has been resolved, however our Engineering team will continue to monitor for any additional anomalies.

Today, our redundant network experienced an isolated server failure within our Chicago POP, from ~11:30 - 11:54 GMT. Customers connected through the affected server were automatically transitioned to their secondary POP, and reconnected back to their primary server once it was restored. Our engineering team was automatically alerted and began investigating the issue. During this event, customers with primary tunnels on the affected server may have experienced a slight increase in latency and/or packet loss during the failure. Our engineering team is continuing to investigate root cause.

This incident has been resolved.

Today, our redundant network experienced an isolated server failure within our San Jose (SJC) POP, from ~06:21 - 06:29 GMT. Customers connected through the affected server were automatically transitioned to their secondary POP, and reconnected back to their primary server once it was restored. Our engineering team was automatically alerted and began investigating the issue. As a precautionary measure, we are redistributing customers with primary tunnels on the affected server to other tunnel servers within the POP. During this event, customers with primary tunnels on the affected server may have experienced a slight increase in latency and/or packet loss during the failure, and may experience a similar impact again when their tunnel is migrated to a new primary server. Our engineering team is continuing to investigate root cause.

Today our redundant network experienced two isolated server hardware failures within our NYC POP, from 11:20-12:40 GMT, and from 17:22-18:06 GMT. Customers who were connected through the affected servers automatically failed over to their secondary POP servers. Our engineering team was automatically alerted, brought the servers back online, and customers automatically reconnected back to their primary servers. For the duration of each of these incidents, affected customers may have experienced a small increase in latency and some missing circuit health metrics in our web dashboard. Our engineers are working to investigate the root cause, but no further incidents are expected.

Our phone service provider has resolved the issue and inbound call service is functioning normally. We apologize for any inconvenience.

We are continuing to work on a fix for this issue.

Our phone service provider is currently experiencing issues with inbound calls -- During this time you may be unable to reach Bigleaf support by phone. We are currently awaiting an ETA on repair. If you require immediate assistance, please contact our Support team via email at: support@bigleaf.net and we will do our best to respond to your request as quickly as possible.

Our team has resolved the root cause of this incident and no further impact is expected. If you are still experiencing any unexpected circuit or site connectivity issues, please contact our Support team at 888-244-3133, Opt. 1 or support@bigleaf.net

We’re currently tracking a small percentage of customer sites experiencing a loss of connectivity with 1 or more circuits that are configured as backup-only. We've identified the root cause and are working on an automated resolution. Our support team can also typically apply a manual resolution, so if you’re currently experiencing any circuit downtime, or have any questions about any connectivity issues at your location, please contact Bigleaf Support at 888-244-3133, Opt. 1 or support@bigleaf.net

We continue to see full stability of the affected systems (web dashboard, email alerts, phone system) and believe the incident is fully resolved. If you have any questions or concerns please contact our support team at 888-244-3133 or support@bigleaf.net.

According to our monitoring and other data, we believe that the issue has been resolved. We are continuing to monitor intensively and will provide any updates as appropriate.

It appears that as of the last 20-30 minutes, our web dashboard and phone system have mostly recovered (though some people may still be experiencing intermittent issues connecting to the dashboard). We continue to investigate the issue and will provide updates as appropriate.

Some of our technical support agents are having issues connecting to the phone system we use. If you're attempting to reach our support team and are having trouble getting through, please email the team at support@bigleaf.net.

We are currently investigating an issue with the reachability of our web dashboard (app.bigleaf.net). It appears this is due to an incident in Amazon Web Services where that is hosted. This will also impact email alerts that come from the dashboard. We don't believe customer network traffic is affected, as all customer network traffic runs through our owned-and-operated Cloud Access Network and we don't see any issues with that platform currently. We will continue to investigate and provide an update within the next hour.

As you may be aware due to widespread news coverage, there is a software vulnerability recently disclosed within the Java log4j library that is impacting many software solutions world-wide. On Friday we performed an in-depth review of all of the systems and software tools that we control. Since our use of Java is very limited, we had fairly low exposure. We identified a few internal tools for development and logging that were at risk, and have mitigated those risks. We now believe all systems and tools that we control are safe from exploitation of this vulnerability. We also use common third-party SaaS tools like Office365, so we are monitoring the situation with our third-party vendors and will respond if we become aware of any risks that need to be addressed. At this time we do not believe that any customer data or systems were exposed or were at risk from this log4j vulnerability. Please let us know if you have any questions - You can always reach our 24x7 support team at 888-244-3133. Thank you, Joel Mulkey | Founder and CEO Bigleaf Networks, Inc. www.bigleaf.net

We have implemented a permanent resolution and affected customers should not experience any further impact. We apologize for any inconvenience that this incident caused and are taking steps to ensure that similar issues won't happen in the future.

We have identified the cause of the issue and have implemented a workaround. The issue should now be resolved for those impacted customer sites. We are working on the permanent resolution and will post an update soon.

We are investigating an issue that is impacting a subset of customers who connect through our Denver, CO POP. The issue is impacting graph data in our web dashboard, and may be hindering some traffic load-balancing optimizations. We are working to identify the root cause and will post an update once we have more information.

This incident has been resolved.

The issue has been mitigated and our engineering team is monitoring the situation.

We are observing network performance issues in New York. We are currently investigating.

The incident occurring with our phone system provider has been resolved. Calls to our main phone numbers at (888) 244-3133 and (503) 985-8298 are operating as expected. We will be monitoring in order to ensure stability. The Bigleaf Support line is available for you at this time. Thank you again.

We are experiencing some issues with our phone system provider that may cause difficulty in contacting Bigleaf Technical Support. Inbound calls to our main phone numbers at (888) 244-3133 and (503) 985-8298 may be affected. If you are experiencing this, please email our Support team at support@bigleaf.net. Our team is monitoring the emails, ticket requests, and voicemails. If you are experiencing an urgent issue please contact us immediately via email and we will contact you as soon as possible. Future updates will be posted here. Thanks for your patience!

A packet loss issue with one of our Network Carriers impacted network traffic at our Atlanta and NYC Gateway Clusters. This issue has been resolved.

We have resolved the issue where sites are missing metrics on the Web Dasboard.

The fix has been deployed and we are monitoring.

We have identified the issue where sites are missing metrics on the Web Dasboard and working on a fix to resolve.

We are currently investigating an issue where sites are missing metrics on the Web Dasboard.

During the morning of March 25th 2021, some of our customers connected to the New York and Chicago gateway clusters may have experienced Bigleaf service degradation. We believe this is related to an isolated incident that was associated to one of our customers in the region. We sincerely apologize for any impact and inconvenience this may have caused. We are thoroughly reviewing the incident and exploring ways to minimize any future risks. While this is a rare event and it's unlikely that you will experience any further issues or outages, we know how critical the Internet is and will continue to monitor this closely. We are committed to providing the best possible experience for you and your customers and will do everything we can to work toward permanent resolutions.

From 6:45 PM to 7:36 PM PST, there was an issue with Web Dashboard. Customers would have received a "Service Unavailable" page when accessing. The root cause was a disk mount issue. We sincerely apologize for the impact and inconvenience this incident caused. We know how critical the Web Dashboard is and will continue to monitor this closely. We are committed to providing the best possible experience for you and your customers and will do everything we can to learn from this event and use it to improve the Bigleaf service.

We are investigating an issue with our Web Dashboard. It is currently Unavailable

The web dashboard has been stable, with several system changes, optimizations, and improvements in place to ensure stability and optimal performance. The Bigleaf web dashboard was experiencing latency and errors due to system resource contention. The slow performance issue did not affect customer network traffic, it was solely an issue with the web-based management system. We have verified consistently good performance, and we expect no further interruptions in service. We are sorry for any inconvenience this caused for you. We understand that network visibility and alerting are important for your business. We’ve isolated the root cause and have taken steps to prevent this problem from happening in the future.

The issue has been identified and a fix is being implemented. We plan to provide another update within the next hour.

After making system changes, the web dashboard performance has improved but pages may be slow to render intermittently. The slow performance issue does not affect customer network traffic, it it solely an issue with the web-based management system. We’re monitoring the situation for now, and we're continuing to take steps to address the root cause to prevent this type of performance problem from happening in the future. We plan to provide another update within the next hour.

We are continuing to investigate this issue. We have deployed system changes to help us further debug and isolate the issue. We plan to provide an update within the next hour.

We are continuing to investigate this issue. We apologize for any inconvenience this is causing for you, and our teams are working hard to resolve the issue as quickly as possible. We plan to provide an update within the next hour.

We are continuing to investigate this issue. This has been escalated to our senior engineers and leadership team. We plan to provide an update within the next hour.

We are continuing to investigate this issue.

We are investigating an issue with our Web Dashboard that is causing pages to load slowly.

A temporary upstream carrier issue in the New York City gateway cluster caused packet loss for a subset of customers there. We mitigated the effects of the problem, the network has been stable for the past hour, and no further impact is expected.

Packet loss on an upstream carrier circuit in our New York City Gateway Cluster began around 7AM Pacific time. A subset of customer circuits in the area were affected by this issue. We have mitigated the effects of the problem, and traffic should return to normal over the next few minutes. We're continuing to monitor the situation to ensure resolution.

The network is stable and no further impact is expected.

At about 8am Pacific, CenturyLink/Level3 reported that they had resolved a BGP routing problem in their network that had prevented some router sessions from establishing correctly since approximately 4am. Around that time, we had noticed a restoration of normal traffic levels, and network problems appear to have been resolved. Site-to-cloud traffic and some CenturyLink/Level3 and other carrier circuits were affected while the problem was occurring. We worked to route around their network, which helped restore service for some customers. However, even though we withdrew our route announcements to them, CenturyLink/Level3 were inadvertently continuing to announce withdrawn routes and were black-holing some customers' traffic. The problem wasn't fully resolved until the carrier fixed their BGP peering issues around 8am Pacific time. CenturyLink/Level3 is a major part of the global internet, so any issue with their network can cause an impact on all services for many carrier networks. Due to the widespread nature of this problem, multiple major internet carriers have temporarily de-peered from CenturyLink/Level3, causing some services hosted in that network to be unavailable. Please let us know if you are experiencing any such problems through your Bigleaf service, and we'll be happy to help troubleshoot it. We are working with CenturyLink/Level3 to determine the root cause of the problem on their network and to ensure the problem does not happen again. We will leave this incident open for the next 6 hours while we monitor the situation.

About 15 minutes ago, CenturyLink/Level3 has reported that they have resolved a BGP routing problem in their network that was preventing router sessions from establishing correctly. We've noticed a restoration of traffic levels, and most problems appear to have been resolved. We're monitoring the situation now.

We have pushed out a change to disable our CenturyLink/Level3 carrier nationwide due continuing effects of the packet loss in multiple Gateway Clusters. We're reviewing data and impact reports to ensure this is resolved.

Our automated monitoring systems have detected packet loss on an upstream carrier circuit in our Atlanta Gateway Cluster over the past two hours. The problem may be affecting transit traffic for other POPs as well, since it is a top-tier internet carrier. Unfortunately, the carrier is continuing to announce routes and black-hole providers' traffic even when their BGP sessions are disabled. Some customers are experiencing packet loss due to this issue, and we are currently in the process of mitigating the effects. We will provide an update once the issue is fully mitigated.

The web dashboard has been stable since yesterday morning, with several system changes, optimizations, and improvements in place to ensure stability and optimal performance. The Bigleaf web dashboard was experiencing latency and errors from 9 to 10am Pacific time yesterday when a newly deployed service temporarily caused a problem with resource contention. The slow performance issue did not affect customer network traffic, it was solely an issue with the web-based management system. We have verified consistently good performance, and we expect no further interruptions in service. We are sorry for any inconvenience this caused for you. We understand that network visibility and alerting are important for your business. We’ve isolated the root cause and have taken steps to prevent this problem from happening in the future.

The web dashboard has been stable since 10am, with several system changes in place to ensure optimal performance. We're continuing to monitor this incident for the next 24 hours to ensure resolution.

After making system changes, the web dashboard is now performing as expected. The Bigleaf web dashboard was experiencing latency and errors from 9 to 10am Pacific time. The slow performance issue did not affect customer network traffic, it was solely an issue with the web-based management system. We’re monitoring the situation for now, and we're continuing to take steps to address the root cause to prevent this type of performance problem from happening in the future. We plan to provide another update within the next 6 hours.

The Bigleaf web dashboard at app.bigleaf.net is currently experiencing latency and errors and is intermittently unavailable. This issue is not affecting customer network traffic, it's solely an issue with the web-based management system. Until the issue is resolved, no configuration changes are possible. We apologize for any inconvenience this is causing for you, and our teams are working hard to resolve the issue as quickly as possible. We plan to provide an update within the next hour.

We are investigating an issue with our Web Dashboard that is causing pages to load slowly.

Beginning at approximately 10:25am until 12:25pm Pacific time, an infrastructure change caused site and circuit alert emails from your Bigleaf service to Office365 email addresses to be held by our mail server and not to be sent out. The issue was resolved, all delayed emails have been sent and no further impact is expected at this time. We sincerely apologize for the impact and inconvenience this incident caused. We know how critical it is to receive alerts in a timely manner, and we are working diligently to prevent this type of problem from happening in the future.

Some site and circuit alert emails from your Bigleaf service to Office365 email addresses may have been delayed by up to an hour. We believe we've resolved the issue, and we're currently monitoring the situation. We plan to provide an update on this incident within an hour.

The problem has been resolved. The Bigleaf web dashboard was experiencing latency and errors from 11:33 to 11:53am Pacific time, when certain graphs failed to load. This issue did not affect customer network traffic, it was solely an issue with the management and alerting system. During that time, no configuration changes were possible, and no email alerts were received for circuit issues. We are sorry for any inconvenience this caused for you. We understand the network visibility and alerting are important for your business. We’ve isolated the root cause and have taken steps to prevent this problem from happening in the future.

The Bigleaf web dashboard at app.bigleaf.net is currently experiencing latency and errors. This issue is not affecting customer network traffic, it's solely an issue with the management and alerting system. Until the issue is resolved, no configuration changes are possible, and no email alerts will be received for circuit issues. We apologize for any inconvenience this is causing for you, and our teams are working hard to resolve the issue as quickly as possible. We will provide an update within the next hour.

Earlier today, our Seattle gateway cluster experienced a UPS power problem affecting the 25th floor of the building. The administrators of the datacenter are continuing their work on the power issues at the datacenter. Our Seattle gateway cluster has been online and stable since power was restored. We're closing the statuspage incident, as we expect no further interruptions in service. We may send a post-mortem update on this incident at a later time. Please be assured that we take these events seriously and are working diligently to ensure stability and uptime for the future.

Earlier today, our Seattle datacenter experienced a UPS power problem affecting the 25th floor of the building. The administrators of the datacenter said they are continuing their work to fully resolve the power issue. While our Seattle datacenter is online and stable at this time, we plan to monitor the situation and provide another update within the next 6 hours.

Our Seattle datacenter experienced a UPS power problem affecting the 25th floor of the building. The organization managing the datacenter does not yet have final confirmation that their work on resolving the power issue has been completed. While our Seattle datacenter is online and stable at this time, we plan to monitor the situation and provide another update within the next 3 hours.

Our Seattle datacenter experienced a UPS power problem affecting the 25th floor of the building. The issue at the datacenter appears to be resolved and we will continue to monitor for stability. As the Seattle datacenter came back online, two servers experienced performance issues for a short duration affecting a subset of customers from 11:29 to 11:44am, but those issues have been resolved, and we're working ensure this does not recur in the future. One more server is not online, so a small subset of subscribers will remain connected to a redundant datacenter for a few more minutes until that issue is resolved. Then they will experience 10-second outages while reconnecting to our Seattle datacenter.

Our datacenter co-location provider in Seattle has verified there is a UPS power issue affecting the 25th floor of their datacenter. This means the outage is not being caused by Bigleaf's equipment or core routers. Users are connected to redundant datacenters as expected, and all customer traffic is flowing successfully with temporarily higher than normal latency. Once the Seattle datacenter is back online, there will be a brief 10-second outage while connections are moved back to Seattle. We plan to provide another update within the next hour.

We're currently investigating an issue in our Seattle datacenter in which the core routers went offline. Customers' connections should reconnect to redundant datacenters within 10 seconds. We plan to provide an update within the next hour.

The issue of false latency alarms, which was caused by a time synchronization issue, has been identified and is gradually resolving itself. Some sites may continue to see false level 1 or 2 upload or download latency alarms in the next few hours as time synchronization completes. No impact to production traffic is expected. We're sorry for the false alarms. We understand that accurate circuit alarms are very important to your business, and we're diligently working to improve our time synchronization infrastructure to ensure that such false alarms do not occur in the future.

We have detected upload-direction latency alarms for a subset of sites in multiple gateway clusters since 11:26pm Pacific time, but it appears that there is no actual additional latency on these circuits. Production traffic does not appear to be negatively impacted at this time. We're investigating the issue and plan to provide more information within an hour.

A backbone carrier issue in the Chicago gateway cluster triggered a loss of circuit performance metrics for 10 minutes twice tonight, from 22:47 to 22:57 Pacific and from 23:33 to 23:42 Pacific time. This issue may have also affected site-to-site and inter-POP peering traffic in Chicago during these times. No further impact is expected because we've steered traffic to our redundant backbone carrier in Chicago. We're sorry for the impact to your network. We will follow up to determine the root cause of the issue and to ensure stability.