LaunchDarkly Incident History

Between 11:32 and 11:48 PT, customers experienced increased latency and elevated error rates on API endpoints. Latency and error rates have returned to normal, and the incident has been resolved.

All error rates have normalized.

We have rolled out a fix and error rates are returning to normal

We are continuing to investigate this issue.

At 13:21 Pacific Time, we experienced a high volume of error alerts affecting users' ability to interact with the LaunchDarkly application including logging in and UI / API responsiveness. We quickly identified the root cause relating to flag delivery. We are expecting errors decreasing now.

All LaunchDarkly services are operational, though some customers need to take action to restore their connections to our Flag Delivery Network. Customers continue to successfully reestablish connections by completing the action guidance below and we continue to support customers working through the residual impact. SDKs and Relay Proxy instances which attempted to establish a connection to LaunchDarkly on July 10 between 09:11 and 11:42 AM PT were intermittently affected. This includes situations where the host application was restarted or redeployed during this time. SDKs and Relay Proxy instances which did not attempt to establish a connection during this time were unaffected. We recommend reviewing your application logs for the incident window. Affected SDKs emit messages—varying by language—that contain any of the following: "giving up permanently," "Invalid SDK key," "unauthorized," "not authorized," or "401." Any service that logged these messages may be unable to receive flag updates or send event data until it is restarted. Here are the recommended actions for each affected application: - If you use the Relay Proxy, please restart the Relay Proxy to re-establish connections. It should not be necessary to restart the applications behind it, but we recommend reviewing your application logs to confirm if restarting your applications is necessary. - If you use server-side SDKs without the Relay Proxy, please restart the application to re-establish its connection to LaunchDarkly. - Client-side and Mobile SDKs will reconnect automatically as their applications attempt to reconnect to LaunchDarkly under normal circumstances, for example when the application loads or the device regains network connectivity. - If this does not resolve the issue for an affected application (e.g. you still see the log messages from above), please log a support ticket and we will follow up with you directly to assist with remediation.

We are continuing to monitor for any further issues, see our previous update for recovery steps.

Customers continue to successfully reestablish connections to our streaming services. We recommend reviewing your application logs for the incident window. Affected SDKs emit messages—varying by language—that contain any of the following: "giving up permanently," "Invalid SDK key," "unauthorized," "not authorized," or "401." Any service that logged these messages may be unable to receive flag updates or send event data until it is restarted. Here are the recommended actions for each affected application: - If you use server-side SDKs, please restart the application to re-establish its connection to LaunchDarkly. - If re-establishing connection to the server-side SDK does not work, please create and use a second SDK key. - If you use the Relay Proxy, please restart the Relay Proxy to re-establish connections. It should not be necessary to restart the applications behind it. - Client-side SDKs will reconnect automatically as they restart. - If this does not resolve the issue for an affected application (e.g. you still see the log messages from above), please log a support ticket and we will follow up with you directly to assist with remediation.

Our Flag Delivery services have recovered and we're observing an increase in customer traffic successfully reestablish connections to our streaming services. Customers who continue to encounter 401 connection errors in their SDKs or Relay Proxy instances are advised to restart their applications and/or Relay Proxy instances.

We are investigating continued impact to customers' mobile and server SDK caused by the previous incident (https://status.launchdarkly.com/incidents/chywwz01ptb0). If customers are receiving SDK or Relay Proxy 401s please re-establish connections.