Historical record of incidents for Foxpass
Report: "Syncronization issue with identity providers"
Last updateThis incident has been resolved. Any customer see users missing from groups sync'd from Okta or Google, please contact support.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented
There is an issue with users being deactivated after a sync with external identity providers. We are currently investigating this issue.
Report: "RADIUS/TTLS can't authenticate"
Last updateThis issue has been fixed.
This is affecting TTLS and PEAP connections to our non EAP-TLS-enabled RADIUS cluster.
We are currently investigating this issue.
Report: "EAP-TLS failing for some customer sites"
Last updateA small subset of customer sites cannot authenticate with EAP-TLS
Report: "Long-running LDAP connections return an error code for 'search' commands."
Last updateThis incident has been resolved.
This has been resolved.
We are actively working on this incident. In the meantime, forcing a re-connect (restarting sssd or nslcd) should create new connections which will not be affected.
Report: "LDAP binds are sluggish for some requests"
Last updateThis incident has been resolved.
A small % of LDAP binds are slow to respond. We are working to improve the performance.
Report: "Google Sync Errors"
Last updateGoogle has rolled back the code that caused the error. We are not seeing any more issues.
We are continuing to work on a fix for this issue.
We are getting errors from Google when trying to access their APIs currently. We have escalated to their support.
Report: "Service disruption"
Last updateThis incident has been resolved. Root cause was a failure of our primary database. Automated failover worked as designed and the system recovered about 2 minutes later.
We are currently investigating this issue.
Report: "Console outage"
Last updateThis incident has been resolved.
This is the result of an AWS Route53 DNS issue. More details here: https://status.aws.amazon.com/
We are currently investigating this issue
Report: "G Suite Outage"
Last updateG Suite is reporting that the issue has been fixed and we're seeing recovery as well.
We're seeing error rates improve and continue to monitor the situation.
Google G Suite is having issues signing users in and checking their passwords for delegated authentication. We are monitoring the situation. https://status.cloud.google.com/incident/developers-console/19008
Report: "Increased Error Rate"
Last updateA misconfigured container went into production over the weekend and caused an increase in LDAP error rates. The number of errors was a small enough percentage across the total traffic that it did not raise any internal alerts. Once the configuration error was found, the container was removed from production and service returned to normal levels. We apologize for those impacted by the service interruption.
A misconfigured container caused in increase in LDAP error rates
Report: "Delegated authentication failures"
Last updateThis incident has been resolved.
We are seeing slow or failing authentication to third-party sources. We are investigating.
Report: "Service Outage"
Last updateThe service has recovered and is operating normally.
The primary database failed and a secondary automatically took its place within a few minutes. Service is recovering and we are monitoring performance.
We are currently investigating this issue.
Report: "Some EAP-TTLS RADIUS connections not succeeding"
Last updateThis incident has been resolved but requires action if you connect to our cloud-RADIUS servers for 802.1x. If you did not receive instructions via email, please contact us.
This appears to be an incompatibility with an update to Mac OS's root certificate store and our certificate provider. It is affecting a small number of connections currently but we believe the number will increase daily. We have set up a work-around, you will receive details via our Intercom system.
We are currently investigating this issue. Connection failure are intermittent.
Report: "Issues With Delegated Authentication"
Last updateCustomers that wish to read a detailed postmortem may email us at help@foxpass.com.
A fix has been implemented and the problem has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating issues with our delegated authentication mechanism
Report: "LDAP Outage"
Last updateSee here: http://blog.foxpass.com/august-2016-post-incident-review/
This incident has been resolved.
We are currently investigating this issue.
Report: "Radius Interruption"
Last updateA new set of outbound IP addresses have been set up, and almost all customers have switched over. We are working to get more information from Okta about the block.
Okta is accepting traffic from the new IPs, we are continuing to monitor the situation.
Our traffic to Okta's API is now coming from 52.5.197.216 and 35.153.22.124. You may need to update your Okta on-net whitelist. (please keep the existing ones active).
We are unable to connect to Okta's API.
We are continuing to investigate this issue.
We are currently investigating an interruption in our RADIUS service.
Report: "LDAP scheduled maintenance beginning"
Last updateMaintenance has been completed.
We are performing maintenance on the LDAP server.
Report: "Okta authentication failing for some customers"
Last updateOkta's incident has been resolved. See https://status.okta.com/.
Please follow Okta updates at https://status.okta.com/
Report: "Okta Incident"
Last updateOkta has resolved the issue.
Okta is currently undergoing a service disruption. More information at https://status.okta.com/
Report: "Degraded RADIUS Performance"
Last updateThe issue has been fixed.
We are currently investigating this issue.
Report: "Scheduled Maintenance"
Last updateMaintenance completed
Maintenance beginning
Report: "Empty groups from sync'd sources"
Last updateGroups sync'd from external sources (Google, Azure, Okta, etc) may have been temporarily empty starting at approx. 5:20UTC. Most groups recovered within 30 minutes. All group memberships were fully restored by 7:00UTC.
Report: "Intermittent Google OAuth failures"
Last updateThis incident has been resolved.
Google's OAuth API is returning failures for some requests.
Report: "Increased LDAP bind times"
Last updateWe moved off of the affected Amazon services and have recovered fully.
Amazon's issues are affecting our LDAP bind times. We are monitoring the situation.
Report: "Increased Error Count"
Last update5 minute period of increased errors for API/RADIUS during database maintenance.
We are currently investigating this issue.
Report: "Increased LDAP Bind Times"
Last updateWe experienced an LDAP slowdown for 20 minutes. The system has fully recovered at 3:52 Pacific for the majority of our customers.
We are currently monitoring system recovery.
We are currently investigating this issue.
Report: "Google API Incorrectly Marking Users As Disabled"
Last updateGoogle's Auth API has stopped throwing errors and we have re-enabled our internal user sync.
Google's API is returning an invalid state for some users. We've temporarily disabled our sync; please contact us over the Intercom chat system on our website or by email to get these accounts back into a valid state.
Report: "Authentication errors with Bitium delegated auth"
Last updateThis incident has been resolved.
Customers who delegate authentication to Bitium are encountering errors when trying to authenticate. We are working with Bitium to identify and remedy this issue.
Report: "Okta Incident"
Last updateOkta's issue has been resolved and our error rates have returned to normal.
Okta is currently undergoing an event. As a result, Okta delegated authentication and sync may not be fully functional. Follow here for further updates: https://status.okta.com/
Report: "RADIUS Performance Degraded"
Last updateA RADIUS host failed. Full service has since been restored.
We are currently investigating this issue.
Report: "Intermittent failures returning SSH keys"
Last updateOne process was "stuck" from 04:30-07:30 UTC resulting in intermittent failures returning SSH keys.
Report: "DNS resolution issue"
Last updateThis incident has been resolved.
The DNS issues affecting the eastern region have been fixed, but Okta delegated auth and sync is still having intermittent issues.
There is an issue with DNS resolution. This is affecting many services, but especially delegated authentication.
Report: "Trouble connecting to Google's OAuth servers for some customers"
Last updateWe believe this incident self-resolved quickly. No further errors seen.
We are currently investigating this issue.
Report: "Increased error rates and timeouts for customers using Okta authentication delegation"
Last updateAll calls are returning successfully as of 17:03 Pacific.
We are seeing an increased number of errors when communicating with Okta's API. Around 10% of connections are impacted. We are investigating.
Report: "Intermittent failures to bind when using Okta backend"
Last updateWe identified and fixed an issue causing intermittent auth failures to Okta's backend when using our LDAP interface for a portion of this afternoon.
Report: "Degraded RADIUS Performance"
Last updateGoogle’s endpoint is returning normally now. All systems fully operational.
Google’s auth endpoint was intermittently timing out, degrading our RADIUS servers’ performance. We are monitoring the situation.
We are currently investigating this issue.
Report: "Degraded RADIUS performance"
Last updateAround 23:25 UTC, RADIUS performance was degraded. Some queries experienced slow response times. This was resolved around 23:45 UTC.