Is DigitalOcean Down Right Now? Check if there is a current outage ongoing.

Our Engineering team has successfully resolved the issue that was impacting the DBaaS cluster logs. API responses should be successful without any error messages while retrieving logs, and the "Recent Logs" section is also working as expected. We have continued to monitor the system and have observed stable performance. The issue is now fully resolved. Thank you for your patience while we worked to address the issue. If you are still facing issues, please open a ticket from within your Cloud Control Panel for further review.

Our Engineering team has now mitigated the issue that affected the DBaaS cluster logs and is currently being monitored. Users should no longer see 500 errors when attempting to retrieve the logs via the API. The "Recent Logs" section in the Logs & Queries tab of the UI should now display logs as expected. We are continuing to monitor the situation closely to ensure stability. Thank you for your patience during the process and we will provide a final update once we confirm the issue is fully resolved.

Our Engineering team is currently investigating an issue with fetching DBaaS cluster logs, affecting all non-MongoDB clusters. Affected users may encounter 500 errors when attempting to retrieve logs via the API, and no logs will appear in the Recent Logs tab. We apologize for the inconvenience and will provide updates as soon as we have more information. Thank you for your patience.

Our Engineering team is currently investigating an issue with fetching DBaaS cluster logs, affecting all non-MongoDB clusters.Affected users may encounter 500 errors when attempting to retrieve logs via the API, and no logs will appear in the Recent Logs tab.We apologize for the inconvenience and will provide updates as soon as we have more information. Thank you for your patience.

Our Engineering team identified an issue impacting GenAI agents via Cloud between 19:55 UTC and 20:36 UTC During this time, users were unable to view or create GenAI agents in the Cloud Control Panel. A fix was deployed at 20:36 UTC by our Engineering team and the issue stands resolved now. We apologize for the inconvenience. If you continue to experience any issues, please open a support ticket from within your account.

Our Engineering team identified an issue impacting GenAI agents via Cloud between 19:55 UTC and 20:36 UTCDuring this time, users were unable to view or create GenAI agents in the Cloud Control Panel.A fix was deployed at 20:36 UTC by our Engineering team and the issue stands resolved now.We apologize for the inconvenience. If you continue to experience any issues, please open a support ticket from within your account.

From May 28 2025 20:13 UTC to 20:24 UTC, users may have experienced issues accessing the Cloud Control Panel. Specifically, users attempting to access the Projects, Load Balancers, Networking, and Droplet pages may have seen no content or a blank page. This incident did not impact the DigitalOcean API or the functionality of any existing resources. Our Engineering team was able to fully resolve the issue and as of 20:24 UTC, the Cloud Control Panel is functioning normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

From May 28 2025 20:13 UTC to 20:24 UTC, users may have experienced issues accessing the Cloud Control Panel. Specifically, users attempting to access the Projects, Load Balancers, Networking, and Droplet pages may have seen no content or a blank page. This incident did not impact the DigitalOcean API or the functionality of any existing resources. Our Engineering team was able to fully resolve the issue and as of 20:24 UTC, the Cloud Control Panel is functioning normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Start: 2025-05-27 22:00 UTCEnd: 2025-05-28 00:00 UTCDuring the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the NYC3 region.Expected impact:During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. Additionally, regular requests via the Public API, as well as viewing and submitting requests through the Cloud Control Panel, may time out or fail for up to 5 minutes. We will endeavor to keep this to a minimum for the duration of the change.If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Our Engineering team has confirmed full resolution of the issue on the Managed Databases control plane. Users can continue to create and deploy applications without any issue. We appreciate your patience. If you continue to experience any problems, please open a ticket from your account for further investigation.

Our Engineering team has taken steps to mitigate the impact on the Managed Databases control plane that was causing delays in API calls, resulting in create and deploy errors for users. At this time, services remain stable and users can create or deploy applications normally. We will monitor the environment for an extended period to ensure full function continues. We will post another update by 14:00 UTC today, May 23 or as soon as we have any further information. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience

Our Engineering team continues to investigate an issue with the Managed Databases control plane that is causing delays in the API calls. At this time, we are seeing performance improvement; therefore, Database and App Platform users can create or deploy applications normally. We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience. We will provide further updates as soon as they are available.

Our Engineering team is continuing to investigate the root cause of the issue affecting Managed Database and App Platform control plane operations. We apologize for any inconvenience caused and appreciate your patience. We will post further updates as soon we have more information.

Our Engineering team is currently investigating an issue impacting App Platform and Managed Databases across all regions. We've determined this to be related to another recent incident, which was posted on our Status page at the following address: https://status.digitalocean.com/incidents/2ch98sc024vv During this time, some users may experience issues with control plane operations (such as listing resources, creating and scaling resources, etc) for Managed Databases and App Platform deployments which have a database. We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience. We will provide further updates as soon as more information is available

Our Engineering has confirmed full resolution of the issue impacting Droplet creation in all regions. Users can now create Droplets and Droplet-based services, including Managed Kubernetes, Managed Databases, and Load Balancers without any issues. We appreciate your patience while we worked to fix this issue. If you continue to experience any problems, please open a ticket from your account for further investigation.

Our Engineering team has identified the reason behind the failed Droplet creates and implemented a fix to resolve the issue. Users should now be able to create Droplets and Droplet-based services, including Managed Kubernetes, Managed Databases, and Load Balancers without any issues. Users should no longer see any errors when resizing their existing Droplets from the API as well as the Cloud Panel. We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering team is continuing to investigate the issue with Droplet creation in all regions. During this time, users won't be able to see the Droplet sizes when creating regular/basic Droplets from the Cloud Panel, blocking creates except for the default size Droplet in each region. Resizes for existing basic Droplets are unavailable. Users are also unable to create Load Balancers, Managed Databases, and Kubernetes Clusters via the Cloud Control Panel and will see errors on those pages. API requests for creating and resizing Droplet-based resources are also impacted. At this time, our team is working as quickly as possible to mitigate the impact from this incident and is investigating potential workarounds. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team is investigating an issue with Droplet creation in multiple regions. During this time, users may see an elevated failure rate for the creation of Droplets and other Droplet-based products, such as Load Balancers, Managed Databases, or Kubernetes in multiple regions. Already running Droplets should not be affected, however, users might not be able to resize their Droplets during this time. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team is currently investigating an issue impacting App Platform and Managed Databases across all regions. We've determined this to be related to another recent incident, which was posted on our Status page at the following address: https://status.digitalocean.com/incidents/2ch98sc024vv During this time, some users may experience issues with control plane operations (such as listing resources, creating and scaling resources, etc) for Managed Databases and App Platform deployments which have a database. We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience. We will provide further updates as soon as more information is available

Our Engineering team is investigating an issue with Droplet creation in multiple regions. During this time, users may see an elevated failure rate for the creation of Droplets and other Droplet-based products, such as Load Balancers or Kubernetes in multiple regions.We apologize for the inconvenience and will share an update once we have more information.

From 07:19 UTC to 08:15 UTC, users may have experienced errors with app deployment via App Platform. Our Engineering team has now confirmed the resolution of the issue and app deployment should be working as expected. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

As of 07:19 UTC, our Engineering team is currently investigating an issue causing delays in the creation of apps deployed via App Platform. During this time, some users may see app deployment failing to complete. We are actively working on identifying the root cause of issues and will post further updates as more information is available. We apologize for the inconvenience.

Our Engineering team has confirmed the complete resolution of the issue that was impacting Spaces, App Platform and Container Registry in our BLR1 region. Users should now see improved performance, and error rates have decreased. User should no longer experience errors while deploying their application in App Platform and uploading to Container Registry in the same region. We appreciate your patience and understanding while we worked to restore normal service. If you continue to experience any issues, please open a support ticket from within the Cloud Control Panel for further review.

Our Engineering team has implemented a fix with the issue affecting Spaces, App Platform and Container Registry in our BLR1 region. Users should now see improved performance, and error rates have decreased. User should no longer experience errors while deploying their application in App Platform and uploading to Container Registry in the same region. We are continuing to monitor the situation closely to ensure stability. Thank you for your patience during the process and we will provide a final update once we confirm the issue is fully resolved.

As of 06:56 UTC, our Engineering team is currently investigating an issue affecting Spaces, App Platform and Container Registry in our BLR1 region. During this time, users may experience a high error rate when attempting to access objects in spaces in this region. Users may also experience errors while deploying their application in App Platform and uploading to Container Registry in the same region. We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience. We will provide further updates in due course.

Our Engineering team is currently investigating an issue affecting the availability of Spaces in our BLR1 region. During this time, users may experience a high error rate when attempting to access objects in Spaces in this region.We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience. We will provide further updates in due course.

Our Engineering team is currently investigating an issue affecting the availability of Spaces in our BLR1 region. During this time, users may experience a high error rate when attempting to access objects in Spaces in this region.We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience. We will provide further updates in due course.

From 12:00 UTC to 20:05 UTC, users may have experienced an issue creating new apps and adding domains to the apps deployed via App Platform. Our engineering team has now confirmed the full resolution of the issue and all new apps are working as expected. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix to the issue that caused delays in the creation of new apps and adding domains to the apps deployed via App Platform. The team is monitoring the situation, and we will share an update once this is fully resolved.

Our Engineering team is currently investigating an issue causing delays in the creation of new apps deployed via App Platform. During this time some users might be experiencing delays while attempting to link the domains during app deployment. This issue does not affect existing apps or domains that are already correctly configured. We are actively working on identifying the root cause of issues and will post further updates as more information is available. We apologize for inconvenience.

Our Engineering team is currently investigating an issue causing delays in the creation of new apps deployed via App Platform. During this time some users might be experiencing delays while attempting to link the domains during app deployment. This issue does not affect existing apps or domains that are already correctly configured. We are actively working on identifying the root cause of issues and will post further updates as more information is available. We apologize for inconvenience.

From 13:45 UTC to 15:50 UTC, users may have experienced an issue affecting the GenAI Platform agents using Llama 3.3 70B. Our Engineering team has confirmed full resolution of the issue. If you continue to experience any issues, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Our Engineering team has implemented a fix to address the issue impacting the GenAI Platform agents using Llama 3.3 70B. As of now, GenAI Platform agents using Llama 3.3 70B should respond without any issues. We are actively monitoring the situation and will post an update as soon as the issue is fully resolved. We apologize for the inconvenience and appreciate your patience.

Our Engineering team has identified the cause of the issue affecting the GenAI Platform. During this time, GenAI Platform agents using Llama 3.3 70B will fail to respond. A fix is in progress, and we will provide an update as soon as we have more information.

Our Engineering Team is currently investigating an issue with the GenAI Platform where the GenAI Platform agents using Llama 3.3 70B are failing to respond. We are actively working to identify the root cause. We apologize for the inconvenience and will share an update once we have more information.

From 03:20 UTC to 06:01 UTC, users may have experienced an issue affecting the GenAI Platform using Llama 3.370B where chatbots are failing to respond Our Engineering team has confirmed full resolution of the issue. Users should now be able to run chatbots normally. If you continue to experience any issues, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Our Engineering team has implemented a fix to address the issue impacting with the GenAI Platform agents using Llama 3.370B where chatbots are failing to respond. We are currently monitoring the issue. We will provide an update as soon as more information becomes available. We apologize for the inconvenience and appreciate your patience.

We are investigating an issue where GenAI Platform agents are failing to respond. We apologize for the inconvenience this may cause.

We are investigating an issue where GenAI Platform agents are failing to respond. We apologize for the inconvenience this may cause.

Our Engineering team has completely resolved the issue that was affecting the Droplet search functionality within the DigitalOcean Cloud Control Panel. We have observed normal functionality following the fix and continued monitoring has shown stable behavior. Users should be able to successfully search for and locate their Droplets. The issue is now considered resolved. We appreciate your patience and apologize for any inconvenience this may have caused. However, if you continue to face issues then please open a ticket with our Support team for further review.

Our Engineering team has implemented a fix for the issue affecting the droplet search functionality within the DigitalOcean Cloud Control Panel. Users should now be able to search for the droplets using the search box on the droplets listing page. We are continuing to monitor the situation closely to ensure stability. Thank you for your patience during the process and we will provide a final update once we confirm the issue is fully resolved.

Our Engineering team is currently investigating an issue affecting the Droplet search functionality within the DigitalOcean Cloud Panel. Some customers may be experiencing difficulties when attempting to search for and locate their Droplets. We are actively working to identify the root cause and will provide updates as we learn more. We understand how critical this functionality is for managing your infrastructure, and we appreciate your patience during the process. We will share an update as more information becomes available.

Our Engineering team is currently investigating an issue affecting the Droplet search functionality within the DigitalOcean Cloud Panel. Some customers may be experiencing difficulties when attempting to search for and locate their Droplets. We are actively working to identify the root cause and will provide updates as we learn more. We understand how critical this functionality is for managing your infrastructure, and we appreciate your patience during the process. We will share an update as more information becomes available.

Our Engineering team has confirmed that the issue affecting the GenAI Platform DeepSeek model has been fully resolved. Users should no longer encounter the message "It looks like the agent ran out of tokens while reasoning. Please try again or increase the max tokens." All functionality remains unaffected, and users should no longer experience any interruptions. If you continue to experience any issues, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Our Engineering team has implemented a fix to address the issue impacting the GenAI Platform's DeepSeek model, where some users were encountering the message "It looks like the agent ran out of tokens while reasoning. Please try again or increase the max tokens". At this time, users should be seeing agents run normally. We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering Team is currently investigating an issue with the GenAI Platform DeepSeek model where some users may encounter the message "It looks like the agent ran out of tokens while reasoning. Please try again or increase the max tokens." We will provide an update as soon as more information becomes available. We apologize for the inconvenience and appreciate your patience.

Our Engineering team has confirmed the resolution of the issue affecting certain Droplet metrics in the BLR1 region. Users should no longer see intermittent gaps in monitoring graphs for Droplets in the BLR1 region. We appreciate your patience and apologize for any inconvenience this may have caused.

Our Engineering team has implemented a fix to address the intermittent unavailability of Droplet metrics and delayed Resource Alerts in the BLR1 region. Our team is now and monitoring the results and will continue to do so for an extended period to ensure this sporadic issue is addressed by the implemented fix. At this time, we expect users not to experience further metrics gaps or delays in alerts. We will post an update as soon as we confirm this incident is fully resolved or once we have further information. Thank you for your patience.

Our Engineering team is continuing to investigate the root cause of the issue with intermittent unavailability of Droplet metrics in the BLR1 region. Multiple reproduction efforts and testing against potential root causes are underway. Due to the sporadic nature of the issue, we anticipate incident updates for this issue to be less frequent, but we will share new information as soon as it is available. At this time, users impacted by this incident will experience intermittent gaps in monitoring graphs for Droplets in BLR1, as well as delayed notifications for Resource Alerts (for any resources in BLR1). We apologize for the inconvenience, and we'll share an update once we have more information.

Our Engineering team continues to investigate the intermittent unavailability of certain Droplet metrics in the BLR1 region. Due to the sporadic nature of this issue, our analysis requires additional time to correlate patterns and isolate the root cause. We apologize for the inconvenience, and we'll share an update once we have more information.

Our Engineering team is currently investigating issues with certain Droplet metrics that are missing for the BLR1 region. Users may experience issues when accessing certain metrics on Droplets. We apologize for the inconvenience, and we'll share an update once we have more information.

Start: 2025-05-07 20:00 UTCEnd: 2025-05-07 23:00 UTCDuring the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the LON1 region.Expected impact:During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Our Engineering team is currently investigating issues with certain Droplet metrics that are missing for the BLR1 region. Users may experience issues when accessing certain metrics on Droplets. We apologize for the inconvenience, and we'll share an update once we have more information.

Our Engineering team has confirmed the complete resolution of the issue that was affecting Droplet creation and console access in the NYC1 region. Users should now be able to deploy new droplets and access them normally in the NYC1 region. All systems are now operating normally. We appreciate your patience and apologize for any inconvenience this may have caused.

Our Engineering team has implemented a fix for the issue affecting Droplet creation and console access in the NYC1 region. Impacted users may have seen 504 Gateway Timeout errors during Droplet creation or when accessing the Droplet console. Our team is currently monitoring the situation to ensure stability. We will provide a final update once we confirm the issue is fully resolved.

Our Engineering team is currently investigating an issue with creating Droplets in NYC1 region. During this time, users may see 504 Gateway Timeout errors when creating droplets in the NYC1 region. We apologize for the inconvenience and will provide an update as soon as we have more information.

Our Engineering team is currently investigating an issue with creating Droplets in NYC1 region. During this time, users may see 504 Gateway Timeout errors when creating droplets in the NYC1 region. We apologize for the inconvenience and will provide an update as soon as we have more information.

Our Engineering team has confirmed the complete resolution of the issue that was impacting the availability of Spaces in our FRA1 region. Users should no longer experience elevated error rates when accessing Spaces in the region. We appreciate your patience and understanding while we worked to restore normal service. If you continue to experience any issues, please open a support ticket from within the Cloud Control Panel for further review.

Our Engineering team has implemented a fix for the issue affecting the availability of Spaces in our FRA1 region. Users should now see improved performance, and error rates have decreased. We are continuing to monitor the situation closely to ensure stability. Thank you for your patience during the process and we will provide a final update once we confirm the issue is fully resolved.

Our Engineering team is currently investigating an issue affecting the availability of Spaces in our FRA1 region. During this time, users may experience high error rate when attempting to access Spaces in this region. We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience. We will provide further updates in the due course.

Our Engineering team is currently investigating an issue affecting the availability of Spaces in our FRA1 region. During this time, users may experience high error rate when attempting to access Spaces in this region.We are actively working on identifying the root cause and resolving the issue as quickly as possible. We understand the inconvenience this may cause and appreciate your patience.We will provide further updates in the due course.

Start: 2025-04-30 13:00 UTCEnd: 2025-04-30 21:00 UTCHello,During the above window, our Engineering team will be performing maintenance on core control plane infrastructure. Please note that the existing infrastructure will continue running without issue. This maintenance may impact create, read, update, and delete (CRUD) operations in all regions.Expected Impact:During the maintenance window, users may experience increased latency during two 5 minute windows with the following platform operations:Cloud Control Panel and API operationsEvent processingDroplet creates, resizes, rebuilds, and power eventsManaged Kubernetes reconciliation and scalingLoad Balancer operationsContainer Registry operationsApp Platform operationsManaged Database creation and scalingWe do not expect any impact to customer traffic due to this maintenance. If an unexpected issue for the control plane arises, we will endeavor to keep any impact to a minimum and may revert if required.If you have any questions related to this maintenance please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticketThank you,Team DigitalOcean

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Our Engineering team has confirmed full resolution of the network issue with new DOKS worker nodes in our SFO2 region. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team identified the cause of the issue and has rolled out a fix to resolve the network issue with new DOKS worker nodes. At this time, all services should be operating normally. We are actively monitoring the fix and will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue causing new DOKS worker nodes to have network issues in our SFO2 region. Operations such as creating new DOKS nodes, manually or autoscaled, will have network connectivity issues within the clusters they are added to during this time. Our Engineering team is actively working to identify the root cause and restore service as quickly as possible. We will continue to provide updates here as we make progress.

Our Engineering team is investigating an issue causing new DOKS worker nodes to have network issues in our SFO2 region. Operations such as creating new DOKS nodes, manually or autoscaled, will have network connectivity issues within the clusters they are added to during this time.Our Engineering team is actively working to identify the root cause and restore service as quickly as possible. We will continue to provide updates here as we make progress.

This incident has been resolved.

Between 09:58 UTC and 10:11 UTC, our Engineering team observed issues impacting DNS resolution in our LON1 region. During this time, some users may have experienced errors while trying to resolve domain names from within Digitalocean resources. Our team was able to take quick action to mitigate the impact and resolve the issue, and all services in the LON1 region are now functioning normally. Thank you for your patience, and we apologize for any inconvenience. If you are still experiencing issues or have additional questions, please open a Support ticket right away.

Between 09:58 UTC and 10:11 UTC, our Engineering team observed issues impacting DNS resolution in our LON1 region. During this time, some users may have experienced errors while trying to resolve domain names from within Digitalocean resources. Our team was able to take quick action to mitigate the impact and resolve the issue, and all services in the LON1 region are now functioning normally. Thank you for your patience, and we apologize for any inconvenience. If you are still experiencing issues or have additional questions, please open a Support ticket right away.

Our Engineering team has confirmed full resolution of the issue impacting event processing in the FRA1, AMS2, BLR1, SGP1, NYC1, and SYD1 regions. If you continue to experience any problems, please open a ticket with our Support team. Thank you for your patience, and we apologize for any inconvenience.

Our Engineering team has identified the cause of the issue causing delays in Droplet event processing in our FRA1 region, and has implemented a fix. While investigating this issue, our Engineers also observed some events appearing to be stuck or delayed in the AMS2, BLR1, SGP1, NYC1, and SYD1 regions. As of now, the event processing rates have returned to normal in all these regions. We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue causing delays in Droplet event processing in our FRA1 region. This includes operations such as creating, resizing, powering on/off, or deleting Droplets. These actions may take longer than usual to complete or may appear to be temporarily stalled. Our engineering team is actively working to identify the root cause and restore normal processing speeds as quickly as possible. We will continue to provide updates here as we make progress.

Our Engineering team has identified the cause of the issue causing delays in Droplet event processing in our FRA1 region, and has implemented a fix. While investigating this issue, our Engineers also observed some events appearing to be stuck or delayed in the AMS2, BLR1, SGP1, NYC1, and SYD1 regions. As of now, the event processing rates have returned to normal in all these regions.We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue causing delays in Droplet event processing in our FRA1 region. This includes operations such as creating, resizing, powering on/off, or deleting Droplets. These actions may take longer than usual to complete or may appear to be temporarily stalled.Our engineering team is actively working to identify the root cause and restore normal processing speeds as quickly as possible. We will continue to provide updates here as we make progress.

Our Engineering team is investigating an issue causing delays in Droplet event processing in our FRA1 region. This includes operations such as creating, resizing, powering on/off, or deleting Droplets. These actions may take longer than usual to complete or may appear to be temporarily stalled.Our engineering team is actively working to identify the root cause and restore normal processing speeds as quickly as possible. We will continue to provide updates here as we make progress.

Our Engineering team has resolved the issue with the email delivery delays. Services should now be operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has confirmed the email delivery delays caused by an issue with an upstream provider has been mitigated. We are now seeing successful delivery of previously delayed emails, as well as normal flow of new messages. We are currently monitoring the situation closely and will share an update as soon as the issue is fully resolved.

Our Engineering team is aware of an issue with an upstream provider that is impacting the delivery of emails from DigitalOcean to DigitalOcean customers. Beginning 20:55 UTC, users may experience delays in receiving emails from DigitalOcean, including sign-in verification codes, sign-up verifications, maintenance notifications, and other transactional emails. Our team is closely monitoring the situation and has begun to see signs of recovery, with both delayed and recent emails starting to be delivered. We will provide further updates as they are available.

Our Engineering team is aware of an issue with an upstream provider that is impacting the delivery of emails from DigitalOcean to DigitalOcean customers. Beginning 20:55 UTC, users may experience delays in receiving emails from DigitalOcean, including sign-in verification codes, sign-up verifications, maintenance notifications, and other transactional emails. Our team is closely monitoring the situation and has begun to see signs of recovery, with both delayed and recent emails starting to be delivered. We will provide further updates as they are available.

From 16:54 to 18:17 UTC, users may have experienced issues creating Spaces from control panel due to the datacenter region options failing to list. Our Engineering team has implemented a fix and confirmed full resolution of the issue. Users should be able to create Spaces from Cloud Control Panel without issues now. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix to resolve the issue impacting the creation of Spaces. Users should now be able to create Spaces without any issues. We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering team has identified an issue with creating Spaces through the Cloud Control Panel. During this time, users are unable to create Spaces buckets due to the datacenter region options failing to list. Creating Spaces buckets through the API remains functional. All other Spaces functionality is unaffected. A fix is in progress and we will provide an update as soon as we have more information.

Our Engineering team is currently investigating an issue with creating Spaces through the Cloud Control Panel. During this time, users may experience difficulties when selecting the Data Center option, as it is not listing correctly. We apologize for the inconvenience and will provide an update as soon as we have more information.

Our Engineering team is currently investigating an issue with creating Spaces through the Cloud Control Panel. During this time, users may experience difficulties when selecting the Data Center option, as it is not listing correctly. We apologize for the inconvenience and will provide an update as soon as we have more information.

Our Engineering team has resolved the issue affecting Mongo 8 Database operations. Users should be able to create new Mongo 8 Databases, and other operations such as upgrading, scaling, and forking should complete successfully now. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team is actively investigating the issue affecting Mongo 8 Database operations. We apologize for the continued inconvenience. We will provide additional updates as soon as more information becomes available.

Our Engineering team is continuing to investigate the cause of the issue with Mongo 8 Database creation in our AMS3 region and is taking action to resolve this issue. In addition to the database creation, users may see failures when scaling the instance or performing any control plane operations. We will continue to share updates as soon as we have new information.

Our Engineering team is investigating an issue with Mongo 8 database creation in the AMS3 region. At this time, users may experience errors while attempting to create a new Mongo 8 database instance in the AMS3 region. We apologize for the inconvenience and will provide another update as soon as possible.

Our Engineering team is investigating an issue with Mongo 8 database creation in the AMS3 region. At this time, users may experience errors while attempting to create a new Mongo 8 database instance in the AMS3 region.We apologize for the inconvenience and will provide another update as soon as possible.

Our Engineering team has confirmed the full resolution of the networking issue affecting the LON1 region. Users should be able to access all resources without any issues. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team made routing changes to remediate the networking issue due to an impact by an upstream provider and confirmed that it has successfully mitigated the connectivity issues in the LON1 region. Users should not be seeing any issues with networking in the LON1 region. We will monitor this incident for a short period to confirm full resolution.

Our Engineering team has identified the cause and applied a fix to mitigate the connectivity issue impacting the entire LON1 region. Users should be seeing improvements in reaching their resources in the LON1 region. We'll continue to monitor the situation to confirm this incident is fully resolved and will post an update soon.

Our Engineering team is currently investigating an issue impacting networking in the LON1 region. Users may experience network connectivity loss to the resources residing in this region. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team is currently investigating an issue impacting networking in the LON1 region. Users may experience network connectivity loss to the resources residing in this region.We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has implemented a fix to resolve the issue with creating certain GenAI Platform agents. Users should now be able to create OpenAI and Llama 3.1 8B agents without issue. If you continue to experience any problems, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Our Engineering team is investigating an issue affecting GenAI Platform. The GenAI platform is seeing errors creating agents with certain LLMs, specifically OpenAI and Llama 3.1 8B. The user is shown an agreement checkbox even though they have already accepted the agreement. If they try to accept the agreement, they will be thrown an error and prevented from creating the agent. We apologize for the inconvenience, we will share an update once we have more information.

Our Engineering team is investigating an issue affecting GenAI Platform. The GenAI platform is seeing errors creating agents with certain LLMs, specifically OpenAI and Llama 3.1 8B. The user is shown an agreement checkbox even though they have already accepted the agreement. If they try to accept the agreement, they will be thrown an error and prevented from creating the agent.We apologize for the inconvenience, we will share an update once we have more information.

Our Engineering Team has implemented a fix to resolve the issues impacting GenAI platform agents. They have also confirmed that the issue has been fully resolved. Now, customers should be able to view and update their GenAI platform agents in the console. If you continue to experience any problems, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Our Engineering Team is currently investigating an issue where some customers are unable to view their GenAI platform agent in the console. As a result, they are also unable to update their agent via the console. We are working to determine which types of agents are affected, including whether newly created agents are impacted. We apologize for the inconvenience and will provide another update as soon as possible.

Our Engineering Team is currently investigating an issue where some customers are unable to view their GenAI platform agent in the console. As a result, they are also unable to update their agent via the console. We are working to determine which types of agents are affected, including whether newly created agents are impacted.We apologize for the inconvenience and will provide another update as soon as possible.

From 00:13 UTC to 9:09 UTC, users may have experienced errors when attempting to create Managed Database Caching Clusters. Our Engineering team has confirmed the full resolution of the issue. Users should now be able to create Caching clusters normally. If you continue to experience issues, please open a support ticket from your Cloud Control Panel.

Our Engineering team has implemented a fix to resolve the issues impacting the creation of Managed Database Clusters. Users should no longer see any errors while creating a Caching Database Cluster. We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering team has identified the root cause of the issue affecting the creation of Managed Database Clusters and is actively working on a fix. Users who already have a Caching Database Cluster in their account should be able to create a new Cluster. During this time, users without an existing caching cluster in their account may still see an error while creating a new cluster. We apologize for the inconvenience and will provide another update as soon as possible.

As of 00:13 UTC, our Engineering team is investigating an issue with creating Managed Database clusters. During this time, users may face issues creating caching clusters. We apologize for the inconvenience and will share an update once we have more information.

As of 00:13 UTC, our Engineering team is investigating an issue with creating Managed Database clusters.During this time, users may face issues creating Redis Caching clusters.We apologize for the inconvenience and will share an update once we have more information.

From 23:31 to 23:46 UTC, users may have experienced errors when trying to use the API or while accessing our Cloud Control Panel at https://cloud.digitalocean.com. CRUD operations for all products would have been unavailable, but deployed services continue running without issue. Our Engineering team has implemented a fix and confirmed full resolution of the issue. Users should be able to access the API and Cloud Control Panel without issue now. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

From 23:31 to 23:46 UTC, users may have experienced errors when trying to use the API or while accessing our Cloud Control Panel at https://cloud.digitalocean.com. CRUD operations for all products would have been unavailable, but deployed services continue running without issue.Our Engineering team has implemented a fix and confirmed full resolution of the issue. Users should be able to access the API and Cloud Control Panel without issue now.If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Beginning at 17:56 UTC, users experienced issues with the using the Cloud Control Panel and the API. By 18:10 UTC, we started seeing signs of recovery and users would've been able to access the Cloud Control and API normally. We noticed a brief recurrence of the issue at 18:58 UTC in which users would've experienced latency, with recovery starting at around 19:02 UTC. At 19:02 UTC, our Engineering team has confirmed full resolution of the issue, the Cloud Control Panel and the API should be functioning normally now. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix regarding the issue impacting our Cloud Control Panel and API. We are monitoring the situation to ensure there is no recurrence. We will post another update once we confirm the issue is fully resolved.

Our Engineering team observed a recurrence of this issue from 18:58 to 19:02 UTC. During that time, API requests and navigating the Cloud Control Panel would have failed/experienced latency. As of 19:02 UTC, users should be able to access services normally, but until our team has confirmed a full resolution, there may be periods of instability. Our team is looking into this issue further and we will share an update as soon as we have more information.

As of 18:10 UTC, our Engineering team has identified the cause of the issue impacting our Cloud Control Panel and API to be a backend deployment and has rolled back the breaking change. We are already seeing signs of recovery, and users should now have access to the Cloud Control Panel and API. We will post an update as soon as additional information is available.

As of 17:56 UTC, our Engineering team is investigating an issue impacting our Cloud Control Panel and API. During this time, users will experience errors when trying to use the public API at api.digitalocean.com or while accessing our Cloud Control Panel at https://cloud.digitalocean.com. We are now confirming the full impact. We regret any inconvenience and will let you know as soon as more information is available.

Our Engineering team has confirmed the full resolution of the issue affecting database engine metrics in the Cloud Control Panel for Managed Databases. Users with PostgreSQL, MySQL, Kafka, OpenSearch, or Caching database clusters should now be able to see the metrics. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team continues to monitor the issue affecting Database Engine metrics in the Cloud Control Panel for Managed Databases. The impact is mitigated and we are rolling out the update gradually during scheduled maintenance windows to avoid disruptions. We will post another update once we confirm the full resolution. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has identified and is currently repairing an issue affecting database engine metrics in the Cloud Control Panel for Managed Databases. Users with PostgreSQL, MySQL, Kafka, OpenSearch, or Caching database clusters may be unable to see metrics. Mongo database clusters are not impacted. Additionally, storage metrics and alerts are not impacted and continue to function normally. While we are working to correct this issue as quickly as possible, we anticipate that a resolution will take multiple days. We will post an update once we have a more definite ETA or additional information becomes available. We apologize for the inconvenience. If you have any questions, please feel free to reach out to our Support team from within your account.

Our Engineering team has confirmed the full resolution of the issue affecting database engine metrics in the Cloud Control Panel for Managed Databases.Users with PostgreSQL, MySQL, Kafka, OpenSearch, or Caching database clusters should now be able to see the metrics.Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team continues to monitor the issue affecting Database Engine metrics in the Cloud Control Panel for Managed Databases. The impact is mitigated and we are rolling out the update gradually during scheduled maintenance windows to avoid disruptions. We will post another update once we confirm the full resolution. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has identified and is currently repairing an issue affecting database engine metrics in the Cloud Control Panel for Managed Databases.Users with PostgreSQL, MySQL, Kafka, OpenSearch, or Caching database clusters may be unable to see metrics. Mongo database clusters are not impacted. Additionally, storage metrics and alerts are not impacted and continue to function normally.While we are working to correct this issue as quickly as possible, we anticipate that a resolution will take multiple days. We will post an update once we have a more definite ETA or additional information becomes available.We apologize for the inconvenience. If you have any questions, please feel free to reach out to our Support team from within your account.

Our Engineering team has confirmed the full resolution of the issue with the Billing page for Teams on the Cloud Control Panel. Services should now be operating normally. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team has now mitigated the issue that affected the Billing page for Teams on the Cloud Control Panel and is currently been monitored. Users should no longer see errors when attempting to access or perform actions within the Cloud Control Panel. We will continue to monitor this at our end and will share an update once the problem is resolved completely.

Our Engineering team has Identified the cause of the issue with Billing page on the Control Panel and is actively working on a fix. We will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue with the Billing page for Teams on the Cloud Control Panel. During this time, users may receive error messages when attempting to access or perform actions within the Cloud Control Panel. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has confirmed the full resolution of the issue with the Billing page for Teams on the Cloud Control Panel. Services should now be operating normally. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team has now mitigated the issue that affected the Billing page for Teams on the Cloud Control Panel and is currently been monitored. Users should no longer see errors when attempting to access or perform actions within the Cloud Control Panel.We will continue to monitor this at our end and will share an update once the problem is resolved completely.

Our Engineering team has Identified the cause of the issue with Billing page on the Control Panel and is actively working on a fix. We will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue with the Billing page for Teams on the Cloud Control Panel. During this time, users may receive error messages when attempting to access or perform actions within the Cloud Control Panel. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has confirmed the full resolution of the issue with Cloud Control Panel. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team is currently monitoring the issue which affected Cloud Control Panel logins and registrations for a brief window of time between 20:55 and 22:15 UTC. Users should no longer see errors while trying to login Cloud Control Panel. New users should now be able to register in the Cloud Control Panel as normal. We will continue to monitor this at our end and will share an update once the problem is resolved completely.

Our Engineering team continues to investigate the root cause of the issue with our Cloud Control Panel, but has confirmed error rates are dropping. Most users should now be seeing recovery and be able to login and navigate the site. New users should be able to register. If you continue to see errors, please reach out to us via https://www.digitalocean.com/company/contact/support

Our Engineering team is continuing to investigate the issue with our Cloud Control Panel. At this time, users may experience errors while attempting to navigate any pages under https://cloud.digitalocean.com, which includes both login and registration attempts. If you need to contact our support team and are unable to do so from within your account, you can still reach us via https://www.digitalocean.com/company/contact/support

Our Engineering team is investigating an issue with our Cloud Control Panel. During this time, users may experience errors reaching https://cloud.digitalocean.com/login, issues while signing in there, as well as issues navigating the Cloud Control Panel. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has confirmed the full resolution of the issue with Cloud Control Panel. Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket right away.

Our Engineering team is currently monitoring the issue which affected Cloud Control Panel logins and registrations for a brief window of time between 20:55 and 22:15 UTC.Users should no longer see errors while trying to login Cloud Control Panel. New users should now be able to register in the Cloud Control Panel as normal.We will continue to monitor this at our end and will share an update once the problem is resolved completely.

Our Engineering team continues to investigate the root cause of the issue with our Cloud Control Panel, but has confirmed error rates are dropping. Most users should now be seeing recovery and be able to login and navigate the site. New users should be able to register.If you continue to see errors, please reach out to us via https://www.digitalocean.com/company/contact/support

Our Engineering team is continuing to investigate the issue with our Cloud Control Panel. At this time, users may experience errors while attempting to navigate any pages under https://cloud.digitalocean.com, which includes both login and registration attempts. If you need to contact our support team and are unable to do so from within your account, you can still reach us via https://www.digitalocean.com/company/contact/support

Our Engineering team is investigating an issue with our Cloud Control Panel. During this time, users may experience errors reaching https://cloud.digitalocean.com/login, issues while signing in there, as well as issues navigating the Cloud Control Panel.We apologize for the inconvenience and will share an update once we have more information.

The scheduled maintenance for our core networking infrastructure in the LON1 region, originally planned for March 18, 2025, from 20:00 UTC to 23:00 UTC, has been cancelled. No further changes will be made at this time, and all services are currently operating as expected.During the scheduled window, we encountered an issue that caused a brief impact on services. As a result, we decided to roll back the maintenance to prevent further disruption. We appreciate your patience and understanding during this time.We will provide an update once a new date for the maintenance has been confirmed. If you have any questions or experience any ongoing issues, please reach out to our support team via Cloud Support.Thank you for your understanding.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Start: 2025-03-18 20:00 UTCEnd: 2025-03-18 23:00 UTCDuring the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the LON1 region.Expected impact:During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

From 14:23 UTC to 16:16 UTC, users were unable to complete the signups for new accounts. Our Engineering team has resolved the issue affecting new account sign-ups. Users should no longer experience errors and are now able to complete the sign-up process successfully. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix to resolve the issue impacting new account signups and monitoring the situation. At this time, new signups should be functioning as expected. We will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue with new customer signups. During this time, some new customers are unable to finish the signup process. We apologize for the inconvenience and will provide an update as soon as possible.

From 14:23 UTC to 16:16 UTC, users were unable to complete the signups for new accounts. Our Engineering team has resolved the issue affecting new account sign-ups. Users should no longer experience errors and are now able to complete the sign-up process successfully.If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix to resolve the issue impacting new account signups and monitoring the situation. At this time, new signups should be functioning as expected. We will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue with new customer signups. During this time, some new customers are unable to finish the signup process.We apologize for the inconvenience and will provide an update as soon as possible.

From March 1st 12:00 UTC to March 10th 18:43 UTC, a subset of users experienced issues connecting Managed Kubernetes nodes to Droplets as a source to allow traffic, due to an issue with tags being applied internally. Some users may also have experienced issues adding Managed Kubernetes Clusters as a source in Cloud firewalls with related Droplets. Our Engineering team has confirmed full resolution of the issue. Users should now be able to connect Managed Kubernetes nodes to Droplets and should also be able to add Managed Kubernetes Cluster as a source in the Cloud Firewall with related Droplets. If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel.

Our Engineering team has identified the cause of the issue affecting recently created Managed Kubernetes nodes. These nodes are currently unable to connect to Droplets that use the Managed Kubernetes Cluster as a source to allow traffic. Additionally, users who add a Managed Kubernetes Cluster as a source in the Cloud firewall may experience issues with related Droplets not being added to the firewall as expected. As a temporary solution, users can continue to add the tag k8s:CLUSTER_UUID, where CLUSTER_UUID is the UUID of the user's Managed Kubernetes Cluster. The UUID can be found in the Cloud Panel URL of the Managed Kubernetes Cluster or by running the following command: doctl kubernetes cluster list We are working on a solution and will provide further updates as we progress. We appreciate your patience and understanding.

Our Engineering team is currently investigating an issue impacting recently created Managed Kubernetes nodes not being able to connect to Droplets that utilize the Managed Kubernetes Cluster source to allow traffic. Users adding a Managed Kubernetes Cluster as a source in the Cloud firewall may see related Droplets not added to the Cloud firewall. As a workaround users can add the tag "k8s:CLUSTER_UUID" where "CLUSTER_UUID" is the user's Clusters UUID. Users can find the UUID details in the Cloud Panel URL of Managed Kubernetes Cluster or by using the command "doctl kubernetes cluster list" We apologize for the inconvenience and will share more information as soon as it's available.

From 20:08 UTC to 21:49 UTC, users may have experienced issues creating new certificates, provisioning or scaling MongoDB clusters, and issuing certificates for use with LBAAS and CDN services. Additionally, a subset of new custom domains on the App Platform may have taken longer than usual to configure. Our Certificate Authority has fully resolved the issue with our managed certificates service, and all services have returned to normal operation. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Certificate Authority has implemented a fix for the issue with our managed certificates service. Services are beginning to recover, and we are closely monitoring the situation to ensure full stability. Users should now be able to create new certificates, provision and scale MongoDB clusters, and issue certificates for use with LBAAS and CDN services. Additionally, new custom domains on the App Platform should start configuring normally. We apologize for the inconvenience and will share another update once the matter is fully resolved.

Our engineering team is investigating an issue with our managed certificates service due to an unexpected outage from our Certificate Authority. During this time, users may experience issues creating new certificates, provisioning or scaling MongoDB clusters, and issuing certificates for use with LBAAS and CDN services. Additionally, this may cause a subset of new custom domains on App Platform to take longer than usual to become active. We apologize for the inconvenience and will provide updates as soon as we receive more information.

From 6:00 UTC to 17:45 UTC, users have experienced issues with Droplet creation and Droplet-based services in the NYC3 region. Our Engineering team has fully resolved the issue impacting Droplet creation in the NYC3 region. Users can now create Droplets and Droplet-based services, including Managed Kubernetes, Managed Databases, Load Balancers, and snapshots, without any issues. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix to resolve the issue impacting the creation of Droplets in the NYC3 region. Users should now be able to create Droplets and Droplet-based services including Managed Kubernetes, Managed Databases, Load Balancers, and snapshots without any issues. We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering team has identified the reason behind the failed and delayed Droplet creates in the NYC3 region and implemented a partial mitigation to address this issue. As of now, customers may be able to create Droplets and Droplet-based services including Managed Kubernetes, Managed Databases, Load Balancers, and snapshots, however, there might be still some delay in the creates. We apologize for the inconvenience and appreciate your patience as we work to resolve this issue. Further updates will be shared as they become available.

Our Engineering team is investigating an issue with Droplet creation in the NYC3 region. During this time, users may see an elevated failure rate for the creation of Droplets and other Droplet-based products, such as Load Balancers, Managed Databases, or Kubernetes in the NYC3 region. We apologize for the inconvenience and will share an update once we have more information.

From 17:14 UTC to 17:34 UTC, users experienced issues accessing the Cloud Control Panel and API. Our Engineering team has resolved the issue, and the Cloud Control Panel, API, and events should now be functioning normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix to resolve the issue impacting the Cloud Control Panel and API. Users should no longer experience issues with accessing the Cloud Control Panel and API. We will continue to monitor the situation and post an update once this issue is fully resolved. Thank you for your patience.

Our Engineering team has identified the cause of the issue impacting our Cloud Control Panel and API and is actively working on implementing a fix. We are already seeing signs of recovery, and users should now have access to the Cloud Control Panel and API. We will post an update as soon as additional information is available.

We are continuing to investigate this issue.

Our Engineering team is investigating an issue impacting our Cloud Control Panel and API. During this time, users will experience errors when trying to use the public API at api.digitalocean.com or while accessing our Cloud Control Panel at https://cloud.digitalocean.com. Users may also see issues with processing Droplet and Kubernetes cluster creations along with Droplet-based events. We are now confirming the full impact. We regret any inconvenience and will let you know as soon as more information is available.

From February 6 21:28:35 UTC to February 19 23:14:00 UTC, users may have experienced issues receiving monitoring alerts for tagged resources. Our Engineering team has resolved the issue, alerts for tagged resources are now working properly and users should now start receiving new alerts going forward. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our engineering team has implemented a fix for the issues that affect receiving monitoring alerts for tagged resources, and are currently monitoring the situation. Any alerts that should've been received beginning February 6th to today will not be received, but new alerts for tagged resources should work going forward. We appreciate your patience, we will post an update once we've confirmed the issue is fully resolved.

Our Engineering team is investigating reports that alerts are not received when alerts are configured with tags. Users may not receive email notification of alerts when using tags, although resources specified by name are still monitored and receiving alerts properly. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has confirmed the full resolution of the issue affecting the Spaces Buckets creation and App Platform deployment in BLR1 region . All our services in the region should now be functioning normally. Users should no longer experience any issues with the creation of Spaces Buckets and App Platform application creations and deployments. If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.

Our engineering team has identified the cause and deployed a fix for the issue with Spaces Buckets creation and App Platform deployment in BLR1 region. Spaces Buckets and App Platform application creations and deployments should now function as expected. We are actively monitoring the situation and will provide an update once we confirm the incident is completely resolved.

Our Engineering team has identified the cause of the issue affecting the creation of Spaces Buckets and App Platform application creations and deployments in BLR1. The impact is mitigated by implementing a temporary fix and is actively working on a permanent fix. Users should now be able to create Spaces Bucket and deploy App Platform application in the BLR1 region. We apologize for the inconvenience and will provide further updates as more information becomes available.

As of 08:32 UTC, our Engineering team is investigating an issue affecting the creation of Spaces Buckets in BLR1. During this period, users may also have experienced errors in DigitalOcean Container Registry creations and App Platform application creations and deployments. We apologize for the inconvenience and will share an update once we have more information.

From 00:04 UTC to 00:41 UTC, users experienced issues accessing the Cloud Control Panel and API as well as experienced event failures. Our Engineering team has resolved the issue, and Cloud Control Panel, API and events should now be functioning normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering team has implemented a fix regarding the issue impacting our Cloud Control Panel and API. We are monitoring the situation to ensure there is no recurrence. We will post another update once we confirm the issue is fully resolved.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

As of 00:16 UTC, our Engineering team is investigating an issue impacting our Cloud Control Panel and API. During this time, users will experience errors when trying to use the public API at api.digitalocean.com or while accessing our Cloud Control Panel at https://cloud.digitalocean.com. Users may also see issues with processing Droplet and Kubernetes cluster creations along with Droplet based events. We are now confirming the full impact. We regret any inconvenience and will let you know as soon as more information is available.

Our Engineering team identified an issue affecting Reserved IP assignments across all regions. From 00:26 to 03:26 UTC, users may have experienced issues with assigning Reserved IPs to Droplets. A fix has been implemented and as of 03:26 UTC, all Reserved IP assignments have been successful. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Our Engineering Team has confirmed that the issue affecting GenAI agents in the Cloud has been fully resolved. Users should no longer see the "Failed to Update" banner and all functionality remains unaffected. If you continue to experience any problems, please reach out to our support team by opening a ticket from within your Cloud Control Panel.

Our Engineering team has implemented a fix to resolve the issues impacting GenAI agents in the Cloud. The banner should no longer appear, and functionality remains unaffected. We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering Team has identified an issue with GenAI agents where some customers may see a "Failed to Update" banner in the Cloud. Despite this message, customers can still update their images and use them as normal. We apologize for the inconvenience and will provide another update as soon as possible.

Our Engineering team has confirmed the full resolution of the issue affecting Droplet, Managed Kubernetes, and Managed Database creation in the BLR region. All our services in the region should now be functioning normally. Users should no longer experience any issues with Droplet creation, event processing, and the creation and provisioning of Managed Kubernetes and Managed Database clusters. If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.

Our Engineering team has deployed a fix for the issue affecting Droplet, Managed Kubernetes, and Managed Database creation in the BLR region. As of 6:00 AM UTC, the issue has been fully mitigated, and users should no longer experience any disruptions. Droplet creation, event processing, and the creation and provisioning of Managed Kubernetes and Managed Database clusters should now function as expected. We are actively monitoring the situation and will provide an update once we confirm the incident is completely resolved.

As of 5:40 AM UTC, our Engineering team has identified the cause of the issue affecting Droplet, Managed Kubernetes, and Managed Database creation in the BLR region and is actively implementing a fix. Users should begin to see improvements in Droplet creation and event processing. Additionally, the successful creation and provisioning of Managed Kubernetes and Managed Database clusters should also resume shortly. We apologize for the inconvenience and will provide further updates as more information becomes available.

As of 3:16 AM UTC, our Engineering team is investigating an issue affecting Droplet, Managed Kubernetes, and Managed Database creation in the BLR region. Users may also experience delays or issues with event processing related to Droplets in the BLR region. Additionally, users may see errors with Managed Kubernetes and Managed Databases cluster creation and provisioning. We apologize for the inconvenience and will provide an update as soon as more information becomes available.

Our Engineering team has confirmed the full resolution of the issue impacting the GenAI Platform. Users should now be able to fetch resource data, usage, and other details from within the Cloud Panel without any issues. If you continue to experience problems, please open a ticket with our support team from within your Cloud Control Panel.

Our Engineering team has implemented a fix to resolve the issues impacting the GenAI Platform in the Cloud Panel. Users should now be able to interact with the GenAI Platform or fetch the resource data, usage, etc without any issues. We are actively monitoring the situation and will post an update as soon as the issue is fully resolved.

Our Engineering team is investigating an issue with the GenAI Platform. At this time, users may experience errors in the Cloud Panel when performing tasks such as requesting resource data, usage, etc. We apologize for the inconvenience and will provide another update as soon as possible.

The issue affecting Snapshots and Backups access from the DigitalOcean Cloud Panel has been fully resolved. Users should now be able to access these services without any errors. We appreciate your patience while we worked to fix this issue. If you continue to experience any problems, please open a ticket from your account for further investigation.

Our Engineering team have implemented a fix for the issue affecting Snapshots and Backups access from the DigitalOcean Cloud Panel. Services are now recovering, and we are closely monitoring the situation to ensure stability. If you continue to experience any issues, please open a ticket from your account for further review.

Our Engineering team is investigating an issue with Backups and Snapshots on Cloud Control Panel. During this time, users may experience error while accessing Backups and Snapshot webpage. We apologize for the inconvenience and will share an update once we have more information.

Our Engineering team has successfully resolved the issue affecting both the creation and regeneration of Spaces Access Keys. Between 13:05 UTC and 13:50 UTC, users may have faced intermittent errors when creating or regenerating Access Keys. All services are now fully restored and operating normally. If you're still encountering any issues, please open a ticket with our support team. We apologize for any inconvenience caused.

From 19:44 UTC to 23:19 UTC, users may have experienced issues with accessing the Droplet Recovery Console. We have confirmed that the issue is fully resolved, and users should be able to access the Recovery Console from the Cloud Control Panel as normal. If you continue to experience problems, please open a ticket with our support team. Thank you for your patience, and we apologize for any inconvenience.

Our Engineering team has implemented a fix to resolve the issue with the Droplet Recovery Console and are currently monitoring the situation. Users should no longer experience issues with accessing the Recovery Console in the Cloud Control Panel. We will continue to monitor the situation and post an update once this issue is fully resolved. Thank you for your patience.

Our Engineering team has identified the cause of the issue with accessing the Droplet Recovery Console in the Cloud Control Panel and is actively working on a fix. We will post an update as soon as additional information is available.

Our Engineering team is investigating an issue affecting the ability to access the Droplet Recovery Console in the Cloud Control Panel. At this time, users may experience issues with accessing the recovery console for their Droplets in all regions. The normal Droplet web console seems to be working fine. We apologize for the inconvenience and will share an update once we have more information.

Prior to Tuesday, January 21, 2025 at 19:20 UTC, customers that used an AWS CLI or AWS SDK version released on or after January 15, 2025 may have experienced issues with uploading files to a Spaces bucket. This impacted PutObject and UploadPart requests initiated by recent versions of the AWS CLI and AWS SDKs. Our Engineering team has confirmed resolution of this issue for almost all Spaces buckets (including all new Spaces buckets), which are now compatible with the latest versions of the AWS CLI and AWS SDKs (and tools and applications that depend on these versions). Note that Spaces does not currently verify data integrity checksums sent by the AWS CLI and AWS SDKs as part of upload requests (see Data Integrity Protections for Amazon S3 - https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html). Customers with a bucket that is still incompatible will receive an email from DigitalOcean with follow up steps. If you continue to experience problems, please open a ticket with our Support team. Thank you for your patience through this issue.

DigitalOcean Spaces Customers using the latest versions (released on Jan 15, 2025) of the AWS CLI or any AWS SDK, or relying on a tool or application that depends on any of these versions of the AWS CLI or an AWS SDK, are experiencing failed uploads to Spaces buckets in all regions. This is caused by a change made by AWS on January 15, 2024 to the AWS CLI and almost all AWS SDKs (including boto3) to require use of “Data Integrity Protections for Amazon S3” (https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html). The currently recommended solution is to downgrade (roll back) to a version of the AWS CLI or AWS SDK released prior to January 15, 2025, or to an earlier version of tools or applications that do not depend on one of these versions of the AWS CLI or AWS SDK. The list of known impacted AWS client tools are: AWS CLI v2.23.0 (workaround - move to last known good version: v2.22.35) AWS SDK for Python (boto3) v1.36.0 (workaround - move to last known good version: 1.35.99) AWS SDK for Go 2025-01-15 release (workaround - move to last known good release: 2025-01-14)

# Postmortem We post updates on incidents regularly to cover the details of our findings, learnings and corresponding actions to prevent recurrence. For this update, we will cover a recent incident related to network maintenance. Maintenance is an essential part of ensuring service stability at DigitalOcean, through scheduled upgrades, patches, and more. We work hard to plan for no interruptions from maintenance activities and we recognize the impact that maintenance can have on our customers if there is downtime. We apologize for the disruption that occurred and have identified action items to ensure maintenance activities are planned with minimal downtime and that we have plans to minimize impact, even when unforeseen errors occur. We go into detail about this below. ## **Incident Summary** On Jan. 7, 2025 12:10 UTC, DigitalOcean experienced a loss of network connectivity for Regional External Load Balancers, DOKS, and AppPlatform products in the SFO3 region. This impact was the result of an unexpected effect of scheduled maintenance work to upgrade our infrastructure and enhance network performance. The maintenance work to complete the network change was designed to be seamless, with the worst expected case being dropped packets. The resolution rollout began Jan. 7th at 13:15 UTC and was fully completed with all services reported operational Jan. 7th at 14:30 UTC. ## **Incident Details** The DigitalOcean Networking team started performing scheduled maintenance to enhance network performance at 10:00 UTC. As part of this maintenance, a routing change was rolled out at 12:10 UTC that would redirect the traffic over to a new path in the datacenter. There was an old routing configuration present on the Core switches that did not get updated. This resulted in traffic being dropped for products relying on Regional External Load Balancers \(e.g. Droplets, DOKS, AppPlatform\). As soon as we detected the drop in traffic, we reverted the changes to mitigate the impact.  ## **Timeline of Events** 12:12 UTC - Flip to the new workflow was started and the impact began  12:40 UTC - Alert fired for Load Balancers 13:00 UTC - First report from customers was received 13:13 UTC - Revert of the new code was started 13:21 UTC - Incident was spun up 14:19 UTC - Routing configuration was updated on Core Switches  14:30 UTC - Code revert was completed and service functionality was restored ## **Remediation Actions** Following a full internal postmortem, DigitalOcean engineers identified several areas of learning to prevent similar incidents. These include updating maintenance processes, increasing monitoring and alerting, and improving observability of network services.  Key Learnings: 1. Learning: Gaps in our automated validation of Network configuration setup for the rollout of this enhancement were identified. While our new, upgraded network results in a simpler state, we have inherited interim complexity as we transition from old to new. * Actions: The gaps in validation automation that resulted in this incident have been addressed. In addition, automated audit processes to further harden the validation process across all network configurations will be applied. ‌ 1. Learning: Gaps in our incremental rollout process for network datapath changes for both pre- and post-deployment were identified. The process for these changes allowed for quicker rollouts for small changes that passed validation. We recognize that "small changes" can have a high impact. * Actions: All network changes will follow an incremental rollout plan, regardless of whether or not validation processes determine that it is a minor change. Runbooks have been updated with reachability tests to encompass Load Balancers and Kubernetes clusters for this type of maintenance. Additional testing steps in both pre-maintenance and post-maintenance procedures to confirm the success of the maintenance have also been implemented.

From 13:09 - 14:48 UTC, we experienced an issue with IPv6 network connectivity in our SFO3 region. Our Engineering team was able to identify the root cause, revert changes, and resolve the issue. During that time, connectivity via IPv6 to services in SFO3 was impacted. This included, in particular, connectivity to and provisioning of Load Balancers, control plane connectivity to Kubernetes clusters, and connectivity to App Platform Apps. If you continue to experience problems, please open a ticket with our support team. Thank you for your patience and we apologize for any inconvenience.

Our engineering team has implemented a fix to resolve the networking issue in our SFO3 region. Users should no longer encounter connectivity issues, provisioning Load Balancers, latency, and timeout errors while interacting with resources in the SFO3 region. We will post an update as soon as the issue is fully resolved.

Our Engineering team is continuing to investigate the cause of the issue with networking in our SFO3 region and is taking action to mitigate this incident by reverting recent changes. In addition to connectivity issues, users may see failures with provisioning Load Balancers in SFO3. We are seeing some services recover, but expect to see a few more short periods of network connectivity issues as the reverts progress. We will continue to share updates as soon as we have new information.

Our engineering team is investigating an issue related to the SFO3 network maintenance here: https://status.digitalocean.com/incidents/lf0njg8ysj0w During this time users may experience connectivity issues, latency, and timeout errors while interacting with resources in the SFO3 data center. We apologize for the inconvenience and will share an update once we have more information.

From 11:29 to 12:31 UTC, our Engineering team observed a Networking issue in our BLR and SFO regions. During this time, users may have experienced Droplet connectivity issue, Users should no longer be experiencing these issues. We apologize for the inconvenience. If you have any questions or continue to experience issues, please reach out via a Support ticket on your account.

As of 18:55 UTC, our Engineering team has confirmed the full resolution of the issue that impacted Managed Kubernetes Clusters in all of our regions. All of the services should now be working normally. If you continue to experience problems, please open a ticket with our Support team from within your Cloud Control Panel. Thank you for your patience and we apologise for any inconvenience.

Our Engineering team is currently investigating an issue with Managed Kubernetes clusters. During this time, some users may experience delays when provisioning Kubernetes clusters, specifically version 1.31.1-5. We apologize for the inconvenience and will provide an update as soon as we have more information.

From 15:28 to 15:32 UTC, an internal service disruption may have resulted in users experiencing errors while using the Cloud Panel or API to manage Spaces Buckets, Apps, Managed Database Clusters, or Load Balancers as well as other actions due to impacted downstream services. If you continue to experience problems, please open a ticket with our support team. Thank you and we apologize for any inconvenience.