Historical record of incidents for ngrok
Report: "Traffic Inspector events in EU region not populating for some customers"
Last updateThis issue has been resolved
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Traffic Inspector events in EU region not populating for some customers"
Last updateWe are currently investigating this issue.
Report: "Rate limit action causing connection issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Endpoints using the rate limit action are having connection issues due to a recent release. We are working on getting a fix out.
Report: "Rate limit action causing connection issues"
Last updateEndpoints using the rate limit action are having connection issues due to a recent release. We are working on getting a fix out.
Report: "Traffic Inspector and Events are delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified the issues and are working on a fix
Report: "Traffic Inspector and Events are delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified the issues and are working on a fix
Report: "Sites unavailable"
Last updateThis incident has been resolved.
We have stabilized and are monitoring status
The issue has been identified and will be working to restore service
We have mostly recovered and are monitoring
The issue has been identified and will be working to restore service.
We are recovering and monitoring performance
We are currently investigating this issue.
Report: "Customers receiving 3200 errors and inaccessible endpoints"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Intermittent connectivity issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Report: "Dashboard SSO logins are failing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Unable to upgrade to pro or enterprise plans"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Getting a high rate of ERR_NGROK_503 in the EU region - No available reserved addresses"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We have deployed the fix and are monitoring
We have identified the issue and are deploying a fix.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Dashboard failures: Billing Page and Account Creation down"
Last updateThis incident has been resolved.
A fix has been deployed which resolves the issue and restores all dashboard functionality. We are monitoring to ensure that the fix is correct.
We are tracking an issue that prevents new accounts from being created, existing accounts from being deleted and also results in failures to access the billing subscription page to view or manage an account's subscription. We are preparing a fix for deployment. No other services are known to be affected at this time.
Report: "Intermittent service outages in the past 12 hours"
Last updateThe fix has been confirmed and the issue is now resolved.
A fix has been implemented and we are monitoring the results.
Report: "New endpoints failing to start in all regions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Partial outage in EU for endpoint connections"
Last updateThis incident has been resolved
We have deployed a fix and are monitoring our systems
We believe we have identified the underlying issue and are working on a fix
We have a failure in our EU region that will impact a limited number of connections to that region.
Report: "Delayed certificate provisioning for wildcard domains created in the last two days"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "IPv6 Network Connection Issues"
Last updateThe incident has been resolved and connections are working as expected.
We have pushed out a fix and are monitoring
The underlying issue has been identified, and we are working on fix.
We are currently investigating intermittent issues with agents and endpoints connecting to some of our IPv6 addresses.
Report: "Agents experiencing tunnel connection errors"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Agents are experiencing tunnel connection errors (3200) for a subset of users. We are investigating this issue.
Report: "Some accounts getting suspended without warning"
Last updateThis incident has been resolved.
We have identified and remediated the core issue impacting our billing system. All accounts impacted have the correct subscription plan enabled on their account, and will receive an email today. Thank you for your patience.
We have identified impacted accounts that were improperly suspended by our billing system not recognizing invoices as paid. We have worked to ensure that accounts are no longer suspended, and are working to reinstate their paid subscriptions, where applicable. If your paid subscription is missing, we will work to add it again. In the meantime, you can resubscribe with the following steps: 1. Log in to your ngrok account 2. Click on “Billing”on the bottom left 3. Resubscribe to your original plan Thank you for your patience.
We are currently investigating this issue.
Report: "Degraded service in the EU region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Investigating networking disruption to our documentation and front page"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Intermittent request failures in eu and us regions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "JP region degraded state"
Last updateThis incident has been resolved.
We have fixed the issues with JP region and are monitoring.
We have a identified an issue with the jp region and are working to resolve the issue.
Report: "`jp` region is down for new tunnel starts and endpoint configuration changes"
Last updateWe are fully recovered in `jp`.
We are seeing `jp` recover and are continuing to monitor.
We are continuing to work on a fix for this issue.
The `jp` region is unable to update state, as such new tunnel starts will fail and changes to configuration for endpoints and Edges will fail. Existing tunnels are still working. The team has identified the issue and is working on mitigation.
Report: "`jp` region experiencing degraded tunnel start errors"
Last updateAn issue with pod resources and deploys caused intermittent failures for some new tunnels. This issue is now resolved.
We are seeing tunnel start issues in `jp` and investigating.
Report: "`ap` region degraded"
Last updateAfter monitoring, we believe this issue to be resolved. Please reach out to our support team if you continue to see issues.
We are seeing a partial recovery and continuing to monitor.
We have identified an issue with a service rollout in the `ap` region and are resolving this issue. Customers may see a percentage of requests fail at ngrok's edge. We are resolving the issue and monitoring for recovery.
Report: "OAuth, OIDC, and SAML modules for Edges degraded"
Last updateThe system should be recovered for all users.
Customers using SAML modules may in rare cases need to clear cookies if you are seeing `BAD REQUEST` errors on login.
Fixes are rolled out. The team will continue monitoring to confirm a full recovery.
We have identified an issue with the following Edge modules that may be impacting customers' ability to access content served by the Edge: OAuth, OIDC, and SAML. We are rolling out a fix and monitoring. This issue started approximately ten minutes ago.
Report: "Intermittent errors affecting ngrok services in US"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Errors in US region"
Last updateThe `us` region is recovered and dashboard is back to working. If you continue to see issues please contact support.
We've rolled out updates to resolve the identified issue and are monitoring for recovery. We're expecting a full recovery over the next few minutes.
We are currently investigating this issue.
Report: "Unable to start new tunnels - all regions affected"
Last updateService is restored in all regions and the incident is resolved.
Service is restored in AP, AU, and EU regions.
Service is restored in SA region.
Service is restored in IN regions.
Service is restored in US regions.
We are continuing to work on a fix for this issue.
The issue has been identified and we are addressing the issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Agent connection failures in US"
Last updateWe are no longer seeing customer impact for this incident.
We believe the issue has been identified and fixed. Customers should see a recovery in the dashboard and our US regions. We're continuing to monitor the situation.
We are rolling out mitigations for the issue and seeing some recovery. Customers may also see errors on dashboard.ngrok.com when attempting to stop or view agent sessions.
We are continuing to investigate this issue.
We're investigating an increased failure rate for agents starting tunnels in the US.
Report: "Intermittent Errors in AP, AU, IN, JP, and SA regions"
Last updateThis incident has been resolved.
We have deployed a fix and are monitoring all regions.
We have identified the underlying issue and are working to get a fix out.
Report: "ngrok.com and api down due to DDOS"
Last updateThe ngrok.com website and our `api` were impacted by a DDOS attack from approximately 06:50 AM until 09:45 AM Pacific. We do not yet know the source, but we have implemented new mitigations for future attacks.
Report: "dashboard.ngrok.com is degraded with some state not displaying"
Last updateThe ngrok dashboard is operational again. Customers should no longer see issues.
The team has rolled out a mitigation for the dashboard. We are continuing to monitor, but customers should see a recovery of the dashboard now.
The ngrok `api`, Agent, tunnels, and Edge are all working as expected. We continue to monitor the dashboard and will provide an ETA to recovery as soon as we have one.
State on the ngrok dashboard is delayed or not displaying in some cases. The team is working to mitigate the issue.
Report: "ngrok.dev resolving to incorrect name servers"
Last updateWe have resolved the issue with our hosting providers and see ngrok.dev is fully recovered.
We are seeing a recovery of ngrok.dev domains. This incident will remain open until we are confident all issues are worked out with our hosting providers. Please reach out to support@ngrok.com if you are seeing issues.
We're seeing some recovery with our hosting providers.
We are continuing to investigate the issue and work with our hosting providers. We do not yet have a timeline to recovery for ngrok.dev domains.
We're seeing ngrok.dev resolve to the wrong nameserver through all DNS providers. The team is investigating and working with hosting providers.
Report: "Tunnel disconnects and start failures in au and eu"
Last updateThis incident has been resolved.
We have identified and resolved the issue with a misconfiguration of CAA records.
We now understand this incident to impact all regions except `us`. We have also determined a potential cause and are attempting a mitigation now.
We are continuing to investigate this issue.
We are continuing to identify the issue and believe the issue only impacts custom domains.
We are seeing widespread tunnel disconnect and start failures in the `au` and `eu` regions. Our team is investigating.
Report: "Partial ap outage"
Last updateNetworking disruptions caused some services to fall behind on data replication. Existing endpoints and agent connections should have continued working, but updates to configuration or new connections would have intermittently failed from 11:00 to 12:00 Pacific.
Starting at approximately 11:00 Pacific, the `ap` region began experiencing issues with new tunnel starts and some existing tunnels receiving traffic. Some but not all traffic is impacted. The team is investigating as monitoring suggests this is improving.
Report: "False positive on McAfee for windows binary"
Last updateWe have worked with McAfee to remove the block on ngrok executables. Customers should no longer see an issue with McAfee blocking ngrok.
Noting the impacted component. We have reached out to McAfee and are waiting for a response.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue. Please add ngrok.exe to your allowlist manually while we work with McAfee for a resolution.
Report: "Events in us-cal-1 are delayed or failing to process"
Last updateThe fix has was rolled out at 3 pm Eastern, and events are now being processed correctly for us-cal-1.
Traffic event sources for the us-cal-1 region are failing to process and be forwarded to event destinations. The team has identified the issue and is working on a fix.
Report: "General service disruption affecting SA"
Last updateThis incident has been resolved
A fix is in place and we are monitoring the results
The issue has been identified and a fix is in process.
We are currently investigating this issue.
Report: "ngrok.com returning permanent redirect"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being made.
We are currently investigating this issue.
Report: "Network connectivity issues affect high traffic customers"
Last updateThis incident has been resolved.
The implemented fix is resolving the issue.
The issue was identified and a fix has been deployed.
We are continuing to investigate this issue.
There is a working theory to resolve the issue
We are currently investigating this issue.
Report: "Some Tunnel Restarts Break Endpoints"
Last updateThe fix is now rolled out to all regions, and our team has confirmed we are seeing no more instances of the issue.
After further monitoring, we have a set of fixes we believe to be comprehensive. We are rolling these out to all regions now.
We are still seeing intermittent failures around rare edge cases. The team is continuing to work on a fix.
A fix has been rolled out, and we are monitoring for additional impact or recurrence.
The issue has been identified, and a fix is being implemented.
In some cases, when a user restarts a tunnel the new endpoint does not function.
Report: "New tunnel starts in `jp` failing intermittently"
Last updateWe see a full recovery of `jp` and believe this is resolved. Please reach out to our support team if you still see issues.
We believe we have found the root cause and are rolling out a fix. No ETA on full recovery.
After an upgrade to some infrastructure, we're seeing intermittent failures to tunnel starts in our `jp` region. Retries should succeed in most cases.
Report: "Some endpoints with certificates seeing 3200 errors"
Last updateAt approximately 2200 Pacific, ngrok engineers progressively rolled out a change to improve endpoint update speed and resiliency. Testing and monitoring across regions did not see issues then; however, customer reports of 3200 errors started coming in several hours later. After working with Customer Support, engineers identified a problem with the change that caused some endpoints with associated certificates to not be exposed at our edge. This caused the 3200s for impacted endpoints. Only endpoints created by a new ngrok Edge or an agent connecting/reconnecting during the incident window were impacted, and only then if they had an associated certificate through "automatic TLS certificates" or manually uploaded TLS certificates. At 0625 Pacific, the scope of the issue became apparent, and teams began remediation. By 0830 an appropriate fix was identified, and the team started rolling this out. The fix was released to regions in serial to assess any impact. At 0930, the highest traffic regions were complete, with other regions following up to 15 minutes later. The team identified several tests that can catch this in the future. We are also working on monitoring that catches edge case errors, as seen in this incident, that our usual monitors don't detect due to the sheer volume of traffic handled by ngrok.
This incident should be resolved for all customers in all regions. Please let our support team know if you see any further issues.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are rolling back changes identified as causing this issue. Customers will see recovery in regions over the next 30 minutes. We are prioritizing the highest traffic regions first.
We are investigating an incident in which customers with certificates attached to an endpoint, primarily custom domains, are seeing 3200 errors. The issue has been identified, and we are working on remediation.
Report: "Investigating an issue with global routing to the sa region"
Last updateThe sa region is back to healthy.
We have restarted an impacted node in the region and believe we are recovering.
We are continuing to investigate this issue.
We are investigating issues with global routing in our sa region. This should only be an issue for customers utilizing Edges or other global features in that region. Other ngrok regions are working as expected.
Report: "ngrok fallback certs being served for regional domains outside the region"
Last updateWe believe this issue to be resolved for all customers.
Remediation is deployed in all regions. We believe the issue is resolved for all customers and are monitoring for recurrences.
We are working on a remediation and will update within the next hour.
We have discovered and reproduced an error in which customers with regional, reserved, wildcard domains will see an ngrok owned certificate when accessing their endpoint from a region their tunnel was not started in. This is not the default path or possible via the dashboard.
Report: "Failed DNS change causing global outage"
Last updateRollback complete and customers should no longer see impact.
Rollback in progress and regions are starting to recover.
We are continuing to work on a fix for this issue.
A failed DNS change is causing a global outage. We are actively rolling the change back.
Report: "Tunnel Sessions cannot be stopped via API in JP"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
In JP only, we are currently investigating an issue that prevents stopping of tunnels via the API
Report: "New tunnels cannot be started in the India (in) region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating. Existing tunnels are still online not affected.
Report: "General Service Disruption"
Last updateThis incident has been resolved.
Continuing to monitor and address degraded performance in JP, but all other regions are fully operational.
Continuing to monitor and address degraded performance in JP, but all other regions are fully operational.
A general outage is impacting the platform. Recovery is in-progress.
Report: "Globally degraded availability"
Last updateGlobal recovery confirmed.
We are continuing to investigate this issue.
We're recovered globally and will continue to monitor.
We're seeing issues with connections terminating successfully in all regions. Retries are likely succeeding, but error rates are elevated. We're investigating and will update shortly. This also impacts our dashboard.