Is Pix4D Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're experiencing an elevated level of API errors and are currently investigating the issue.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We're experiencing an elevated level of API errors and are currently investigating the issue.

This incident has been resolved.

We are currently investigating an issue of elevated error rates in PIX4Dcloud

This incident has been resolved.

The service is back to normal after performing an operation to restore the service in a healthy state. We are monitoring the situation.

We're experiencing an elevated level of API errors on the license check endpoint and are currently investigating the issue.

The incident is resolved.

The error rate has decreased back to zero and we are monitoring the situation.

The issue has been identified and we are seeing the error rate decreasing.

We're experiencing an elevated level of API errors and are currently investigating the issue.

This incident has been resolved.

We have identified the cause and restored the DNS configuration. We are closely monitoring the resolution. We will restart projects that failed processing due to the incident.

We are currently investigating DNS resolution failures for some customers connecting to pix4d.com

This incident has been resolved.

The changes have been reverted. We are continuing to monitor the situation to ensure DNS resolution across different world servers is working correctly.

A DNS change affected the DNS resolution of all our domains. We reverted the change and are monitoring DNS propagation. The changes were applied this morning at 9:31 UTC which resulted in an invalid DNS path for naming resolution of our domains.

This incident has been resolved.

The remediation is active and the service is responding again as normal. We are monitoring the traffic / error rate.

The issue has been identified and a fix is being implemented.

We're experiencing an elevated level of API errors and are currently investigating the issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We're experiencing an elevated level of API errors and are currently investigating the issue.

The incident is solved. The errors were temporary failure events during a deployment.

The issue causing the login failure has been identified, we are working to correct it.

We are observing some errors for some user profiles on the log in action.

The incident is resolved. All point cloud conversions have be re-triggered with the fix in place and are either in progress or already done.

A fix has been implemented and we will reprocess all the point clouds for the affected projects.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We observe an issue with the processing of point cloud for online visualization, and we are investigating.

This incident has been resolved. The Cloud UI and API are currently stable, we're working on hardening the system against similar issues.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

2023-06-15 13:40 CEST - Investigating We are experiencing significant delays with queued tasks. The following tasks are affected: - Project processing - Project tasks - Invoice generation - Mail delivery 2023-06-15 14:45 CEST - Monitoring The queues are processing again as expected. The delays concern the tasks queued today between 8:15 and 14:40 CEST 2023-06-21 - 16:00 CEST - Resolved All the delayed tasks from June 15th have been recovered and re-triggered. No data was lost, but some emails will contain old or inconsistent dates because of the delay to reschedule them. This incident is now resolved.

This incident has been resolved. Affected projects have been successfully requeued. We are closely monitoring the service, but are confident any effects have been resolved.

The configuration has been reverted and the processing is working normally again. We are re-activating the different regions. New projects submitted for processing from 12:37 UTC will be be processed correctly. We keep monitoring the situation and will re-launch all the failed projects from today.

The issue has been identified and we are working to resolve the invalid configuration.

We're experiencing an elevated level of processing errors and are currently investigating the issue. The incident started around 10am CEST / 8am UTC.

This incident has been resolved.

Application servers are recovered, and cloud.pix4d.com is accessible now. We are monitoring processing of projects and requests.

Our services in cloud.pix4d.com are not accessible.

This incident has been resolved.

The services are all running fine since more than 1 hour. We will keep monitoring them closely during the day.

The API service is restored and we are monitoring the services.

The issue has been fixed and we restarting the services.

We are currently investigating an issue on our API servers which are not responding.

The incident is resolved, web UI platforms are fully functional again.

Our engineers have pushed a fix for the issue which has been deployed. Accessing dataset and processing new ones from the UIs is functional. There may be some delays in processing for a short while as we process our backlog. We are carefully monitoring the situation.

Currently accessing projects and uploading new ones through the web UI is not working.

We identified the impacting services and are working to mitigate the incident.

We are investigating the outage on the services blocking projects from being loaded on the web UI.

We are currently experiencing an outage in Cloud and Inspect. Our engineers are investigating the issue.

The incident is closed, all projects are processing again in all regions successfully.

The situation looks back to normal from our monitoring, we have started re-processing the failed projects by batches. We keep monitoring the situation.

We are monitoring the situation. The patch has been rolled out at 09:15 UTC.

The issue has been identified and a corresponding patch has been deployed.

We will reschedule failed projects due to the incident for all customers, no need to reschedule them on your side. We understand the failure but we are still investigating to understand the root cause.

Still investigating the issue, we will post any update later.

We're experiencing an elevated level of processing errors and are currently investigating the issue.

This incident has been resolved.

It has been observed the timeouts in the APIs have ceased, we are still monitoring.

We observed a spike in some queries that caused an overload in the CPU utilization of the database instances.

We are experiencing degradation on database performance. It is impacting the APIs response, and processing of projects. We are currently investigating the issue.

After monitoring for sometime, it is observed that network timeouts have stopped, and all the Cloud API instances are up and working as normal.

The rate of the network timeouts is decreasing, now all the Cloud API instances are working as normal.

We are experiencing interruptions in the APIs due to network timeouts in our Cloud Provider infrastructure.

All the pending projects have started processing. The situation is back to normal.

We identified that the provisioning of processing nodes is slow due to network issues on our cloud provider.

We are currently investigating this issue.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're experiencing an elevated level of API errors and are currently investigating the issue.

The incident is resolved.

We are monitoring the situation.

We experienced an unusual load on the database. We mitigate the issue and are working on restoring the failed processes.

We're experiencing an elevated level of API errors and are currently investigating the issue.

The system is back to normal.

We are continuing to investigate this issue.

We're experiencing an elevated level of API errors and are currently investigating the issue.

This incident has been resolved.

We have identified an issue with our database, and we're monitoring the incident.

Our data processing infrastructure is running behind which is causing delays in the project outputs. No data has been lost and we are investigating the issue.

The post-processing tasks work normally in all the processing clusters.

We are continuing to monitor for any further issues.

We have deployed a configuration change that mitigates the problem. Post-processing tasks should work again in EU, KR, and JP.

We have identified the problem and are working on mitigation.

We are currently investigating an issue related to failing post-processing tasks in Europe, Japan, and Korea.

We are not observing any issues anymore in our services, processing is back to normal. All failed projects have been sent for reprocessing.

All processing clusters are resuming normal operations. Our cloud provider has not yet closed the incident. We keep monitoring the situation while reprocessing all failed projects.

The issues impacting our cloud provider seems mostly resolved and we are not observing any error anymore. We are putting back in the queue the failed processing jobs.

The issue has been identified and we are monitoring our cloud provider services.

We're experiencing an elevated level of errors on our backend due to issues from our cloud provider. Processing might be delayed or failing due to this underlying issue.

This incident has been resolved.

A patch has been deployed in production to address the issue and we are monitoring the service.

We have identified the cause of the failures and are working on restoring the regular service.

We observed increase of failing requests in tasks processing subsystem, we are currently investigating the issue.

This incident has been resolved.

Our services are back to normal and the processing is running normally. We are monitoring the situation in order to act quickly in case the connectivity issues are not entirely solved on our cloud provider platform.

Our cloud provider is aware of connectivity issues to instances and are investigating.

We identified the faulty services and are working to replace them.

Our data processing infrastructure is running behind which is causing delays in the project outputs. No data has been lost and we are investigating the issue.

All pending tasks have been processed, and queues processing is back to normal processing rate.

The new processing instance is provisioned, and pending tasks are being processed.

Processing capacity degraded due to failure of one of the instances. We are currently replacing the failed instance with a healthy one.

Observed increase in the post processing tasks queue in Japan cluster, we are currently investigating the issue.

There is no more delay in the processing queue, all projects are in a processing state.

The queue in the US cluster is catching up. We added more capacity in order to catch up on the processing. We are monitoring the situation until the processing queue is back to normal.

We identified the issue and are working on restoring the processing of the cluster.

Our data processing infrastructure in the US is not processing since Friday June 4th which is causing delays in the project outputs. No data has been lost and we are investigating the issue.

The new deployment has correctly addressed the issue, and the incident is resolved now.

A patch has been deployed in production and we are monitoring the service.

We are continuing to work on a patch for this issue.

We are continuing to work on a patch for this issue.

We observed some degradation in the internal tasks processing system. The issue has been identified, and we are working on preparing the patch.

To improve our observability of production we activated some tracing in our backend services which causes our API to fail for a small percentage of the calls. The component throwing those errors were not compatible with the tracing mechanism we activated and we failed to see those problems during the test phase. We rolled back our change and will fix this incompatibility issue before activating the performance tracing again.

We've experienced an elevated level of API errors and have found the issue. A fix will be shortly deployed.

The US cluster is back to nominal operations. New incoming projects are processed with regular processing capacity.

We're increasing the total processing capacity in the US cluster to speed up processing of the pending projects.

The US cluster started catching up on processing the enqueued projects.

We have identified the source of the degraded US cluster processing, and we're working on restoring the full operation.

We are currently investigating this issue.

This incident has been resolved.

Tasks are processed again. We're monitoring the queue in the Japan cluster for the next hours.

The issue on the Japan cluster has been identified and the tasks should start to be processed.

Our data processing infrastructure is running behind in our Japan cluster which is causing delays in the project outputs. No data has been lost and the system should be caught up shortly as soon as we identified the issue.