DigitalOcean Incident History

Our Engineering team has identified the cause of the issue causing delays in Droplet event processing in our FRA1 region, and has implemented a fix. While investigating this issue, our Engineers also observed some events appearing to be stuck or delayed in the AMS2, BLR1, SGP1, NYC1, and SYD1 regions. As of now, the event processing rates have returned to normal in all these regions.We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue causing delays in Droplet event processing in our FRA1 region. This includes operations such as creating, resizing, powering on/off, or deleting Droplets. These actions may take longer than usual to complete or may appear to be temporarily stalled.Our engineering team is actively working to identify the root cause and restore normal processing speeds as quickly as possible. We will continue to provide updates here as we make progress.

Our Engineering team is investigating an issue causing delays in Droplet event processing in our FRA1 region. This includes operations such as creating, resizing, powering on/off, or deleting Droplets. These actions may take longer than usual to complete or may appear to be temporarily stalled.Our engineering team is actively working to identify the root cause and restore normal processing speeds as quickly as possible. We will continue to provide updates here as we make progress.

Our Engineering team is aware of an issue with an upstream provider that is impacting the delivery of emails from DigitalOcean to DigitalOcean customers. Beginning 20:55 UTC, users may experience delays in receiving emails from DigitalOcean, including sign-in verification codes, sign-up verifications, maintenance notifications, and other transactional emails. Our team is closely monitoring the situation and has begun to see signs of recovery, with both delayed and recent emails starting to be delivered. We will provide further updates as they are available.

Our Engineering team is currently investigating an issue with creating Spaces through the Cloud Control Panel. During this time, users may experience difficulties when selecting the Data Center option, as it is not listing correctly. We apologize for the inconvenience and will provide an update as soon as we have more information.

Our Engineering team is investigating an issue with Mongo 8 database creation in the AMS3 region. At this time, users may experience errors while attempting to create a new Mongo 8 database instance in the AMS3 region.We apologize for the inconvenience and will provide another update as soon as possible.

Our Engineering team is currently investigating an issue impacting networking in the LON1 region. Users may experience network connectivity loss to the resources residing in this region.We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team is investigating an issue affecting GenAI Platform. The GenAI platform is seeing errors creating agents with certain LLMs, specifically OpenAI and Llama 3.1 8B. The user is shown an agreement checkbox even though they have already accepted the agreement. If they try to accept the agreement, they will be thrown an error and prevented from creating the agent.We apologize for the inconvenience, we will share an update once we have more information.

Our Engineering Team is currently investigating an issue where some customers are unable to view their GenAI platform agent in the console. As a result, they are also unable to update their agent via the console. We are working to determine which types of agents are affected, including whether newly created agents are impacted.We apologize for the inconvenience and will provide another update as soon as possible.

As of 00:13 UTC, our Engineering team is investigating an issue with creating Managed Database clusters.During this time, users may face issues creating Redis Caching clusters.We apologize for the inconvenience and will share an update once we have more information.

From 23:31 to 23:46 UTC, users may have experienced errors when trying to use the API or while accessing our Cloud Control Panel at https://cloud.digitalocean.com. CRUD operations for all products would have been unavailable, but deployed services continue running without issue.Our Engineering team has implemented a fix and confirmed full resolution of the issue. Users should be able to access the API and Cloud Control Panel without issue now.If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has confirmed the full resolution of the issue affecting database engine metrics in the Cloud Control Panel for Managed Databases.Users with PostgreSQL, MySQL, Kafka, OpenSearch, or Caching database clusters should now be able to see the metrics.Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team continues to monitor the issue affecting Database Engine metrics in the Cloud Control Panel for Managed Databases. The impact is mitigated and we are rolling out the update gradually during scheduled maintenance windows to avoid disruptions. We will post another update once we confirm the full resolution. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has identified and is currently repairing an issue affecting database engine metrics in the Cloud Control Panel for Managed Databases.Users with PostgreSQL, MySQL, Kafka, OpenSearch, or Caching database clusters may be unable to see metrics. Mongo database clusters are not impacted. Additionally, storage metrics and alerts are not impacted and continue to function normally.While we are working to correct this issue as quickly as possible, we anticipate that a resolution will take multiple days. We will post an update once we have a more definite ETA or additional information becomes available.We apologize for the inconvenience. If you have any questions, please feel free to reach out to our Support team from within your account.

Our Engineering team has confirmed the full resolution of the issue with the Billing page for Teams on the Cloud Control Panel. Services should now be operating normally. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team has now mitigated the issue that affected the Billing page for Teams on the Cloud Control Panel and is currently been monitored. Users should no longer see errors when attempting to access or perform actions within the Cloud Control Panel.We will continue to monitor this at our end and will share an update once the problem is resolved completely.

Our Engineering team has Identified the cause of the issue with Billing page on the Control Panel and is actively working on a fix. We will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue with the Billing page for Teams on the Cloud Control Panel. During this time, users may receive error messages when attempting to access or perform actions within the Cloud Control Panel. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has confirmed the full resolution of the issue with Cloud Control Panel. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team is currently monitoring the issue which affected Cloud Control Panel logins and registrations for a brief window of time between 20:55 and 22:15 UTC.Users should no longer see errors while trying to login Cloud Control Panel. New users should now be able to register in the Cloud Control Panel as normal.We will continue to monitor this at our end and will share an update once the problem is resolved completely.

Our Engineering team continues to investigate the root cause of the issue with our Cloud Control Panel, but has confirmed error rates are dropping. Most users should now be seeing recovery and be able to login and navigate the site. New users should be able to register.If you continue to see errors, please reach out to us via https://www.digitalocean.com/company/contact/support

Our Engineering team is continuing to investigate the issue with our Cloud Control Panel. At this time, users may experience errors while attempting to navigate any pages under https://cloud.digitalocean.com, which includes both login and registration attempts. If you need to contact our support team and are unable to do so from within your account, you can still reach us via https://www.digitalocean.com/company/contact/support

Our Engineering team is investigating an issue with our Cloud Control Panel. During this time, users may experience errors reaching https://cloud.digitalocean.com/login, issues while signing in there, as well as issues navigating the Cloud Control Panel.We apologize for the inconvenience and will share an update once we have more information.

The scheduled maintenance for our core networking infrastructure in the LON1 region, originally planned for March 18, 2025, from 20:00 UTC to 23:00 UTC, has been cancelled. No further changes will be made at this time, and all services are currently operating as expected.During the scheduled window, we encountered an issue that caused a brief impact on services. As a result, we decided to roll back the maintenance to prevent further disruption. We appreciate your patience and understanding during this time.We will provide an update once a new date for the maintenance has been confirmed. If you have any questions or experience any ongoing issues, please reach out to our support team via Cloud Support.Thank you for your understanding.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Start: 2025-03-18 20:00 UTCEnd: 2025-03-18 23:00 UTCDuring the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the LON1 region.Expected impact:During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

From 14:23 UTC to 16:16 UTC, users were unable to complete the signups for new accounts. Our Engineering team has resolved the issue affecting new account sign-ups. Users should no longer experience errors and are now able to complete the sign-up process successfully.If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix to resolve the issue impacting new account signups and monitoring the situation. At this time, new signups should be functioning as expected. We will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue with new customer signups. During this time, some new customers are unable to finish the signup process.We apologize for the inconvenience and will provide an update as soon as possible.