ngrok

Is ngrok Down Right Now? Check if there is a current outage ongoing.

ngrok is currently Operational

Last checked from ngrok's official status page

Historical record of incidents for ngrok

Report: "Traffic Inspector events in EU region not populating for some customers"

Last update
resolved

This issue has been resolved

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Traffic Inspector events in EU region not populating for some customers"

Last update
Investigating

We are currently investigating this issue.

Report: "Rate limit action causing connection issues"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

Endpoints using the rate limit action are having connection issues due to a recent release. We are working on getting a fix out.

Report: "Rate limit action causing connection issues"

Last update
Identified

Endpoints using the rate limit action are having connection issues due to a recent release. We are working on getting a fix out.

Report: "Traffic Inspector and Events are delayed"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We have identified the issues and are working on a fix

Report: "Traffic Inspector and Events are delayed"

Last update
Resolved

This incident has been resolved.

Monitoring

A fix has been implemented and we are monitoring the results.

Identified

We have identified the issues and are working on a fix

Report: "Sites unavailable"

Last update
resolved

This incident has been resolved.

monitoring

We have stabilized and are monitoring status

identified

The issue has been identified and will be working to restore service

monitoring

We have mostly recovered and are monitoring

identified

The issue has been identified and will be working to restore service.

monitoring

We are recovering and monitoring performance

investigating

We are currently investigating this issue.

Report: "Customers receiving 3200 errors and inaccessible endpoints"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Intermittent connectivity issues"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are continuing to work on a fix for this issue.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and a fix is being implemented.

Report: "Dashboard SSO logins are failing"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Unable to upgrade to pro or enterprise plans"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Getting a high rate of ERR_NGROK_503 in the EU region - No available reserved addresses"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

We have deployed the fix and are monitoring

identified

We have identified the issue and are deploying a fix.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Dashboard failures: Billing Page and Account Creation down"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been deployed which resolves the issue and restores all dashboard functionality. We are monitoring to ensure that the fix is correct.

identified

We are tracking an issue that prevents new accounts from being created, existing accounts from being deleted and also results in failures to access the billing subscription page to view or manage an account's subscription. We are preparing a fix for deployment. No other services are known to be affected at this time.

Report: "Intermittent service outages in the past 12 hours"

Last update
resolved

The fix has been confirmed and the issue is now resolved.

monitoring

A fix has been implemented and we are monitoring the results.

Report: "New endpoints failing to start in all regions"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Partial outage in EU for endpoint connections"

Last update
resolved

This incident has been resolved

monitoring

We have deployed a fix and are monitoring our systems

identified

We believe we have identified the underlying issue and are working on a fix

investigating

We have a failure in our EU region that will impact a limited number of connections to that region.

Report: "Delayed certificate provisioning for wildcard domains created in the last two days"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "IPv6 Network Connection Issues"

Last update
resolved

The incident has been resolved and connections are working as expected.

monitoring

We have pushed out a fix and are monitoring

identified

The underlying issue has been identified, and we are working on fix.

investigating

We are currently investigating intermittent issues with agents and endpoints connecting to some of our IPv6 addresses.

Report: "Agents experiencing tunnel connection errors"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Agents are experiencing tunnel connection errors (3200) for a subset of users. We are investigating this issue.

Report: "Some accounts getting suspended without warning"

Last update
resolved

This incident has been resolved.

monitoring

We have identified and remediated the core issue impacting our billing system. All accounts impacted have the correct subscription plan enabled on their account, and will receive an email today. Thank you for your patience.

identified

We have identified impacted accounts that were improperly suspended by our billing system not recognizing invoices as paid. We have worked to ensure that accounts are no longer suspended, and are working to reinstate their paid subscriptions, where applicable. If your paid subscription is missing, we will work to add it again. In the meantime, you can resubscribe with the following steps: 1. Log in to your ngrok account 2. Click on “Billing”on the bottom left 3. Resubscribe to your original plan Thank you for your patience.

investigating

We are currently investigating this issue.

Report: "Degraded service in the EU region"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Investigating networking disruption to our documentation and front page"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Intermittent request failures in eu and us regions"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

Report: "JP region degraded state"

Last update
resolved

This incident has been resolved.

monitoring

We have fixed the issues with JP region and are monitoring.

identified

We have a identified an issue with the jp region and are working to resolve the issue.

Report: "`jp` region is down for new tunnel starts and endpoint configuration changes"

Last update
resolved

We are fully recovered in `jp`.

monitoring

We are seeing `jp` recover and are continuing to monitor.

identified

We are continuing to work on a fix for this issue.

identified

The `jp` region is unable to update state, as such new tunnel starts will fail and changes to configuration for endpoints and Edges will fail. Existing tunnels are still working. The team has identified the issue and is working on mitigation.

Report: "`jp` region experiencing degraded tunnel start errors"

Last update
resolved

An issue with pod resources and deploys caused intermittent failures for some new tunnels. This issue is now resolved.

investigating

We are seeing tunnel start issues in `jp` and investigating.

Report: "`ap` region degraded"

Last update
resolved

After monitoring, we believe this issue to be resolved. Please reach out to our support team if you continue to see issues.

monitoring

We are seeing a partial recovery and continuing to monitor.

identified

We have identified an issue with a service rollout in the `ap` region and are resolving this issue. Customers may see a percentage of requests fail at ngrok's edge. We are resolving the issue and monitoring for recovery.

Report: "OAuth, OIDC, and SAML modules for Edges degraded"

Last update
resolved

The system should be recovered for all users.

monitoring

Customers using SAML modules may in rare cases need to clear cookies if you are seeing `BAD REQUEST` errors on login.

monitoring

Fixes are rolled out. The team will continue monitoring to confirm a full recovery.

identified

We have identified an issue with the following Edge modules that may be impacting customers' ability to access content served by the Edge: OAuth, OIDC, and SAML. We are rolling out a fix and monitoring. This issue started approximately ten minutes ago.

Report: "Intermittent errors affecting ngrok services in US"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Errors in US region"

Last update
resolved

The `us` region is recovered and dashboard is back to working. If you continue to see issues please contact support.

monitoring

We've rolled out updates to resolve the identified issue and are monitoring for recovery. We're expecting a full recovery over the next few minutes.

investigating

We are currently investigating this issue.

Report: "Unable to start new tunnels - all regions affected"

Last update
resolved

Service is restored in all regions and the incident is resolved.

identified

Service is restored in AP, AU, and EU regions.

identified

Service is restored in SA region.

identified

Service is restored in IN regions.

identified

Service is restored in US regions.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and we are addressing the issue.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Agent connection failures in US"

Last update
resolved

We are no longer seeing customer impact for this incident.

monitoring

We believe the issue has been identified and fixed. Customers should see a recovery in the dashboard and our US regions. We're continuing to monitor the situation.

investigating

We are rolling out mitigations for the issue and seeing some recovery. Customers may also see errors on dashboard.ngrok.com when attempting to stop or view agent sessions.

investigating

We are continuing to investigate this issue.

investigating

We're investigating an increased failure rate for agents starting tunnels in the US.

Report: "Intermittent Errors in AP, AU, IN, JP, and SA regions"

Last update
resolved

This incident has been resolved.

monitoring

We have deployed a fix and are monitoring all regions.

identified

We have identified the underlying issue and are working to get a fix out.

Report: "ngrok.com and api down due to DDOS"

Last update
resolved

The ngrok.com website and our `api` were impacted by a DDOS attack from approximately 06:50 AM until 09:45 AM Pacific. We do not yet know the source, but we have implemented new mitigations for future attacks.

Report: "dashboard.ngrok.com is degraded with some state not displaying"

Last update
resolved

The ngrok dashboard is operational again. Customers should no longer see issues.

monitoring

The team has rolled out a mitigation for the dashboard. We are continuing to monitor, but customers should see a recovery of the dashboard now.

identified

The ngrok `api`, Agent, tunnels, and Edge are all working as expected. We continue to monitor the dashboard and will provide an ETA to recovery as soon as we have one.

identified

State on the ngrok dashboard is delayed or not displaying in some cases. The team is working to mitigate the issue.

Report: "ngrok.dev resolving to incorrect name servers"

Last update
resolved

We have resolved the issue with our hosting providers and see ngrok.dev is fully recovered.

monitoring

We are seeing a recovery of ngrok.dev domains. This incident will remain open until we are confident all issues are worked out with our hosting providers. Please reach out to support@ngrok.com if you are seeing issues.

identified

We're seeing some recovery with our hosting providers.

investigating

We are continuing to investigate the issue and work with our hosting providers. We do not yet have a timeline to recovery for ngrok.dev domains.

investigating

We're seeing ngrok.dev resolve to the wrong nameserver through all DNS providers. The team is investigating and working with hosting providers.

Report: "Tunnel disconnects and start failures in au and eu"

Last update
resolved

This incident has been resolved.

identified

We have identified and resolved the issue with a misconfiguration of CAA records.

identified

We now understand this incident to impact all regions except `us`. We have also determined a potential cause and are attempting a mitigation now.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to identify the issue and believe the issue only impacts custom domains.

investigating

We are seeing widespread tunnel disconnect and start failures in the `au` and `eu` regions. Our team is investigating.

Report: "Partial ap outage"

Last update
resolved

Networking disruptions caused some services to fall behind on data replication. Existing endpoints and agent connections should have continued working, but updates to configuration or new connections would have intermittently failed from 11:00 to 12:00 Pacific.

investigating

Starting at approximately 11:00 Pacific, the `ap` region began experiencing issues with new tunnel starts and some existing tunnels receiving traffic. Some but not all traffic is impacted. The team is investigating as monitoring suggests this is improving.

Report: "False positive on McAfee for windows binary"

Last update
resolved

We have worked with McAfee to remove the block on ngrok executables. Customers should no longer see an issue with McAfee blocking ngrok.

investigating

Noting the impacted component. We have reached out to McAfee and are waiting for a response.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue. Please add ngrok.exe to your allowlist manually while we work with McAfee for a resolution.

Report: "Events in us-cal-1 are delayed or failing to process"

Last update
resolved

The fix has was rolled out at 3 pm Eastern, and events are now being processed correctly for us-cal-1.

identified

Traffic event sources for the us-cal-1 region are failing to process and be forwarded to event destinations. The team has identified the issue and is working on a fix.

Report: "General service disruption affecting SA"

Last update
resolved

This incident has been resolved

monitoring

A fix is in place and we are monitoring the results

identified

The issue has been identified and a fix is in process.

investigating

We are currently investigating this issue.

Report: "ngrok.com returning permanent redirect"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being made.

investigating

We are currently investigating this issue.

Report: "Network connectivity issues affect high traffic customers"

Last update
resolved

This incident has been resolved.

monitoring

The implemented fix is resolving the issue.

identified

The issue was identified and a fix has been deployed.

investigating

We are continuing to investigate this issue.

investigating

There is a working theory to resolve the issue

investigating

We are currently investigating this issue.

Report: "Some Tunnel Restarts Break Endpoints"

Last update
resolved

The fix is now rolled out to all regions, and our team has confirmed we are seeing no more instances of the issue.

monitoring

After further monitoring, we have a set of fixes we believe to be comprehensive. We are rolling these out to all regions now.

monitoring

We are still seeing intermittent failures around rare edge cases. The team is continuing to work on a fix.

monitoring

A fix has been rolled out, and we are monitoring for additional impact or recurrence.

identified

The issue has been identified, and a fix is being implemented.

investigating

In some cases, when a user restarts a tunnel the new endpoint does not function.

Report: "New tunnel starts in `jp` failing intermittently"

Last update
resolved

We see a full recovery of `jp` and believe this is resolved. Please reach out to our support team if you still see issues.

monitoring

We believe we have found the root cause and are rolling out a fix. No ETA on full recovery.

identified

After an upgrade to some infrastructure, we're seeing intermittent failures to tunnel starts in our `jp` region. Retries should succeed in most cases.

Report: "Some endpoints with certificates seeing 3200 errors"

Last update
postmortem

At approximately 2200 Pacific, ngrok engineers progressively rolled out a change to improve endpoint update speed and resiliency. Testing and monitoring across regions did not see issues then; however, customer reports of 3200 errors started coming in several hours later. After working with Customer Support, engineers identified a problem with the change that caused some endpoints with associated certificates to not be exposed at our edge. This caused the 3200s for impacted endpoints. Only endpoints created by a new ngrok Edge or an agent connecting/reconnecting during the incident window were impacted, and only then if they had an associated certificate through "automatic TLS certificates" or manually uploaded TLS certificates. At 0625 Pacific, the scope of the issue became apparent, and teams began remediation. By 0830 an appropriate fix was identified, and the team started rolling this out. The fix was released to regions in serial to assess any impact. At 0930, the highest traffic regions were complete, with other regions following up to 15 minutes later. The team identified several tests that can catch this in the future. We are also working on monitoring that catches edge case errors, as seen in this incident, that our usual monitors don't detect due to the sheer volume of traffic handled by ngrok.

resolved

This incident should be resolved for all customers in all regions. Please let our support team know if you see any further issues.

identified

We are continuing to work on a fix for this issue.

identified

We are continuing to work on a fix for this issue.

identified

We are continuing to work on a fix for this issue.

identified

We are continuing to work on a fix for this issue.

identified

We are rolling back changes identified as causing this issue. Customers will see recovery in regions over the next 30 minutes. We are prioritizing the highest traffic regions first.

identified

We are investigating an incident in which customers with certificates attached to an endpoint, primarily custom domains, are seeing 3200 errors. The issue has been identified, and we are working on remediation.

Report: "Investigating an issue with global routing to the sa region"

Last update
resolved

The sa region is back to healthy.

investigating

We have restarted an impacted node in the region and believe we are recovering.

investigating

We are continuing to investigate this issue.

investigating

We are investigating issues with global routing in our sa region. This should only be an issue for customers utilizing Edges or other global features in that region. Other ngrok regions are working as expected.

Report: "ngrok fallback certs being served for regional domains outside the region"

Last update
resolved

We believe this issue to be resolved for all customers.

monitoring

Remediation is deployed in all regions. We believe the issue is resolved for all customers and are monitoring for recurrences.

identified

We are working on a remediation and will update within the next hour.

identified

We have discovered and reproduced an error in which customers with regional, reserved, wildcard domains will see an ngrok owned certificate when accessing their endpoint from a region their tunnel was not started in. This is not the default path or possible via the dashboard.

Report: "Failed DNS change causing global outage"

Last update
resolved

Rollback complete and customers should no longer see impact.

monitoring

Rollback in progress and regions are starting to recover.

identified

We are continuing to work on a fix for this issue.

identified

A failed DNS change is causing a global outage. We are actively rolling the change back.

Report: "Tunnel Sessions cannot be stopped via API in JP"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

In JP only, we are currently investigating an issue that prevents stopping of tunnels via the API

Report: "New tunnels cannot be started in the India (in) region"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating. Existing tunnels are still online not affected.

Report: "General Service Disruption"

Last update
resolved

This incident has been resolved.

monitoring

Continuing to monitor and address degraded performance in JP, but all other regions are fully operational.

monitoring

Continuing to monitor and address degraded performance in JP, but all other regions are fully operational.

monitoring

A general outage is impacting the platform. Recovery is in-progress.

Report: "Globally degraded availability"

Last update
resolved

Global recovery confirmed.

investigating

We are continuing to investigate this issue.

investigating

We're recovered globally and will continue to monitor.

investigating

We're seeing issues with connections terminating successfully in all regions. Retries are likely succeeding, but error rates are elevated. We're investigating and will update shortly. This also impacts our dashboard.