Is Digital Pigeon Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved.

We have identified and mitigated the issue and are currently monitoring the situation.

We are currently investigating an issue affecting the ability to load upload details.

This incident has been resolved. Apple Push Notifications for events within Digital Pigeon are now functioning correctly.

A fix to the Apple Push Notification issue has been deployed and we are monitoring the results.

We have identified that the issue is only affecting customers that are using Apple Push Notifications, and are working on a fix. To mitigate the issue customers can temporarily disable 'Mobile Alert' notifications in Digital Pigeon | Management | Profile | Notifications.

We are currently investigating an issue that is impacting recipients from accessing files that are sent to them.

This incident has been resolved.

After further investigation, we have narrowed down the issue and understand it only occurs when file downloads have been disabled by the sender. We are aiming to deploy a fix shortly.

This incident has been resolved.

We are currently investigating this issue.

Closing the issue as monitoring confirms the issue have been solved.

We are currently investigating an where customers that have set up full custom domains are unable to access uploads.

We have now solved the issue, we are still monitoring for now. Some initial access to an upload pickup page may have failed but then worked after a browser refresh.

We are investigating an issue that's causing potential delays with accessing files

We have rolled back the update from this morning to resolve the issues. Reloading or restarting the desktop application will be required if you have issues forwarding files.

An update to the application this morning has caused issues which we are identifying now.

This incident has been resolved.

We have now been delisted by Google, we're still monitoring the situation but warnings have ceased.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Attempted access to files on Oceana servers are currently being blocked by Google chrome and incorrectly flagged as ‘phishing’ which is not the case. We are investigating with google and working on mitigation now. It is safe to use the 'visit website' under the 'More Details' section. Current workarounds right now are: - Upload to another region (ie US West) - Enable 'Cloudfront Optimized Routes Downloads' in settings if possible

Issues have been resolved, application functioning normally.

A fix has been implemented and we are monitoring the servers.

We are currently investigating the cause.

This incident has been resolved.

Servers have been updated with the fix for both email delivery and recipient pickup page time-out for users with iOS push notifications enabled.

We have a fix ready to deploy and will be ready within the hour

We have identified the issue and it's limited to accounts with Apple Push Notifications enabled. We are working on a fix now.

We are investigating an issue that's preventing email delivery to recipients from initial upload. WORKAROUND: If you have recipients who have not received a notification email you can click 'Resend' from the recipients tab and the email will be delivered. For new uploads you can upload and hit send first and then add recipients afterwards to ensure they are emailed.

We apologise for the inconvenience caused by this issue. ‌ We tracked the problem down to a configuration change that caused unexpected network issues that were intermittently affecting some customers. ‌ We have updated the configuration to prevent this issue going forward and will add further checks to prevent this situation happening again.

This incident has been resolved.

We are investigating an issue prevent access to the application

An incident at AWS resulted in the majority of the application servers to be taken out of action. While the remaining application servers took up the load and were able to response to most requests, with elevated response times, a configuration issue meant it took an 45 minutes before the replacement application servers were available. We will be deploying infrastructure fixes this weekend which will avoid this scenario from repeating again in the future.

Incident is resolved, we will provide a triage report soon.

We have identified and rectified a server issue that caused slow response times and in some cases loss of access to the application. We are currently closely monitoring the servers but access is restored.

We are currently investigating this issue.

This incident has been resolved.

The migration process is taking a little longer than expected. We expect it to be completed in the next 90 minutes.

One of our database servers is reporting hardware faults, to avoid issues we are migrating the database to a new server. The migration is not expected to disrupt the Digital Pigeon app and should take less than 2 hours to complete.

This incident has been resolved.

We have restored service to the application now and are monitoring the situation carefully.

We are currently investigating this issue.

We are closing the incident as the app has been stable for the last hour.

A fix has been implemented and we are monitoring the results.

The app is responding again. We are continuing to investigating the potential root cause of the issue.

We are currently investigating this issue.

This incident has been resolved.

We are continuing to monitor for any further issues.

We believe we have identified the source of the issue and we are monitoring the fix.

The app is responding again, we are continuing to investigate the underlying cause.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating issues with app stability.

This incident has been resolved.

Amazon AWS is reporting the restoration of services in it's Sydney data center after a major issue with a component that controls the data center networking layer. The file task backlog has been processed and all our system seem to be functioning as expected. We are continuing to monitor while Amazon AWS holds their incident open. Please revert any temporary changes you made to your accounts default file server region as a result of this incident. To switch regions head to 'Manage' -> 'Account' -> 'Defaults', select your desired region then click 'Save Changes'. Note that this change only applies to new uploads.

Amazon AWS has provided a brief update indicating that they expect the issue to be resolved within the next 2 hours. The Oceania Digital Pigeon file processing servers are still operating but at a reduced capacity which is causing delays processing previews, zip, etc.

Our hosting provider Amazon AWS is reporting wide spread issues with the Oceania (Sydney) data center which are affecting file related tasks like Previews, Zip files, etc. We are monitoring the incident with Amazon AWS (https://status.aws.amazon.com/) and will keep this incident updated as we get more info. In the mean time we recommend temporarily switching to the 'USA (West)' or 'S.E. Asia (Singapore)' file server regions which are unaffected. To switch regions head to 'Manage' -> 'Account' -> 'Defaults', select your desired region then click 'Save Changes'. Note that this change only applies to new uploads.

We are considering this issue resolved as we have had no new reports of problems in the last 12 hours.

The code level change we made seems as suggested by Amazon AWS appears to have mitigated the issue. However, Amazon AWS is still reporting that the issue is under investigation. We are considering all system as operational at this point, however we will continue to monitor the situation and provide updates. If you still experiencing issues please let us know.

We have deployed the code change as suggested by Amazon AWS. The change adds extra region information to all our Amazon S3 requests which according to Amazon will help mitigate the issue. We still recommend switching to either the Cloudflare or Google DNS servers which are confirmed working and are generally faster and apparently are a lot more reliable than those provided by your ISP. For more info on Cloudflare's DNS service check out https://www.cloudflare.com/learning/dns/what-is-1.1.1.1/. For setup instructions see: > Cloudflare MacOS: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/mac/ > Cloudflare Windows: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/windows/ > Google MacOS: https://developers.google.com/speed/public-dns/docs/using#mac_os > Google Windows: https://developers.google.com/speed/public-dns/docs/using#windows We will continue to provide updates as we get info from Amazon AWS.

We are deploying the code change suggested by Amazon AWS to help mitigate the issue.

Amazon AWS is continuing to implement fixes to mitigate the issue... Amazon have also suggested a code change to help mitigate the problem. We are currently working on implementing and testing that change and hope to have it deployed within the hour.

Amazon AWS reports that there service teams are actively investigating the issue. Unfortunately due to the complexity of the issue they are unable to provide an ETA. To illustrate the complexity of the issue check out: https://www.whatsmydns.net/#A/digitalpigeon-dp-us-west.s3.amazonaws.com

Continuing from the previous incident (https://digitalpigeon.statuspage.io/incidents/r886w5d2t611) which was prematurely closed based on feedback from Amazon AWS. We are continuing to work with Amazon AWS to resolve the issue. A minority of users report that uploads, download and previews are failing. Investigation suggests that the effected users are experiencing issues due to problems resolving Domain Name Server* (DNS) queries to Amazon S3 services (our hosting provider). DNS is a core component of the internet and is usually provided and configured auto-magically by your ISP when you connect to internet. Unfortunately being such a low level part of the internet puts it completely outside of our control. As a temporary work-around for the issue we recommend switching to either the Cloudflare or Google DNS servers which are confirmed working (and are generally faster than those provided by your ISP). > Cloudflare MacOS: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/mac/ > Cloudflare Windows: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/windows/ > Google MacOS: https://developers.google.com/speed/public-dns/docs/using#mac_os > Google Windows: https://developers.google.com/speed/public-dns/docs/using#windows * DNS converts human readable host names (e.g. digitalpigeon-dp-us-west.s3.amazonaws.com) into a series of numbers that computers understand. Whenever your computer attempts to make a connection on the internet the first thing it does is contact your configured Domain Name Server and attempt to resolve its host name into an actual network address. If this step fails then, which is what's happening in this case, the connection will fail.

Please see http://status.digitalpigeon.com/incidents/s3yhqmfk687j

A ticket has been raised with our hosting provider Amazon AWS. They are currently investigating the issue...

We have had small number of users report that uploads are failing immediately with the message 'Upload issues detected'. Investigation suggests that the uploads are failing due to problems resolving Domain Name Server (DNS) queries. A Domain Name Server is a core component the internet and is usually provided and configured auto-magically by your ISP when you connect to internet. The Domain Name Server converts human readable host names (e.g. digitalpigeon-dp-us-west.s3.amazonaws.com) into a series of numbers that computers understand. Whenever your computer attempts to make a connection on the internet the first thing it does is contact your configured Domain Name Server and attempt to resolve its host name into an actual network address. * Temporary Workaround * As a temporary work-around for the issue we recommend switching to either the Cloudflare or Google DNS servers (which are confirmed working). Instructions: Cloudflare MacOS: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/mac/ Cloudflare Windows: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/windows/ Google MacOS: https://developers.google.com/speed/public-dns/docs/using#mac_os Google Windows: https://developers.google.com/speed/public-dns/docs/using#windows

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Under certain scenarios the application APIs are returning errors when accessing uploads. We have identified the cause of the issue and are deploying a fix.

This incident has been resolved.

Fix has been applied to the South East Asia and Europe regions. Oceania is coming up now...

The fix has been applied to the US West transcoding servers.

We've identified an issue that is causing previews for certain video file formats to fail. We are currently applying a fix...

This issue has been resolved. Unfortunately any uploads that still appear to be stuck at 100% will need to be restarted. It should be sufficient to just re-add any files that have not transitioned to the completed state.

Switching your accounts 'Default File Server Region' under 'Manage' -> 'Accounts' -> 'Defaults' to 'USA West' servers provides a temporary work-around for the issue.

We are investigating reports of significant delays in completing uploads of small files (less than 50MB) to the Europe File Server region. Initial indications are that the uploads will eventually complete. Larger file upload do not seem to be effected.

This incident has been resolved, unfortunately our automated monitoring systems didn't alert us to this issue which meant it took longer than usual for an ops engineer to intervene. The automated monitoring system was configured to raise alarms when it detected long queue times but it was not configured to raise an alarm for missing queue stats (indicating that the process manager had stopped reporting). We've extended our monitoring alarms to alert us if this situation occurs again in the future. Our apologies for the inconvenience.

It appears that there was an issue with the process that manages the workers in the transcoding cluster which caused it to stop raising alarms as the 'time on queue' grew. This in turn stopped it from adding workers to the cluster as needed which slowed down the preview process. We are continuing to investigate the issue with the management cluster to prevent this from happening again in the future, but for now a reboot of the cluster seems to have resolved the issue.

We are investing reports that its taking longer than usual for previews to appear.

The incident has been resolved but we are closely monitoring the situation.

Access to the application servers has been restored. The situation is still being monitored and investigated.

The issue has been identified and we are working on correcting it.

We are currently experiencing a major service interruption that is preventing access to the application. Investigations are currently under way.

We experienced an issue with our app servers that left the app inaccessible for some users for about 15 minutes. The issue has since been fixed. Our apologies for the inconvenience.

It would appear that CNN hot-linked to a controversial video causing a huge spike (many times our previous peak load) in requests to our web servers. Unfortunately this saturated our servers and caused the app to become unresponsive. The app is fully operational again and we'll be making some improvements to make sure they next time this happens it doesn't drown the site in requests. Our apologises for the inconvenience.

We are continuing to investigate the cause of the issues. Our apologies for the inconvenience.

We are investigating issues with the app not responding to requests. Please stay tuned for updates...

This incident has been resolved.

The effected database instance has been restored and is currently running a full system verification scan. The app should be unaffected, although it is possible response times may be slightly higher than normal.

We are experiencing an issue with one of our database servers. Thanks to redundancy we don't expect this issue to impact customers much, if at all. Please stay tuned for updates...

This incident has been resolved.

The fix has been deployed. There may be some delays generating previews while the transcoding cluster is updated.

We have identified an issue with the task responsible for generating video previews. We are actively working on a fix and hope to have it deployed within the hour. The issue only seems to be effecting some videos.

This incident has been resolved.

The fix for the watermarking issues is now being deployed. There may be some short delays in generating previews as the transcoding images are updated..

We have identified an issue with the media watermarking feature that causes preview tasks to become 'stuck' generating previews. We're working on a fix and hope to have it live within the next 12 hours. The temporary work-around is to disable the watermarking feature until the issue is resolved. The issue was introduced as a result of a security update.

On October 7th at 2.30am through to 8am (UTC) we experienced an issue with our auto-scaling manager (used to determine the required capacity to encode web previews) that caused long delays generating previews in the US-West region. Unfortunately the scenario didn't trigger a monitoring alarm to wake the ops team. We have fixed the underlying issue and created an alarm to prevent the scenario from occurring again the future. Our apologies for the inconvenience.

This incident has been resolved.

The fix is now live. New uploads will appear as expected, uploads created in the last 12 hours will re-appear as they are indexed in the next few hours.

We have identified an issue that is causing new uploads to not appear in your upload list when you refresh the upload list (or restart the app). Uploads can still be accessed from the activity view, emails, etc. A fix is being implemented and uploads should re-appear in the search list over the next few hours.

The issue effecting uploads with a large number of files has been resolved. Our apologies for the inconvenience.

We are investigating issues initialising uploads that contain many files. The initial work-around is to zip up your files into a single zip file before uploading...

Desktop app upload issues have been resolved, re-starting the desktop app may be required. Our apologies for the inconvenience...

We're looking into reports of users experiencing errors trying to upload with the desktop app. The browser based app appears unaffected...

Issues loading the app have been resolved.

We are investigating issues loading the app. Please stay tuned...

This incident has been resolved.

We have identified an issue loading previews that is effecting some customers. A fix is inbound...

Amazon AWS has confirmed that the underlying network issues have been resolved.

The Digital Pigeon app infrastructure is partially restored and the app is responding again. We are still waiting for official word from Amazon AWS that the underlying issues have been resolved.

Amazon engineers have identified the root cause of the network connectivity issues. Some connectivity has been restored and the engineers are continuing to work on the issue.

Amazon engineers are working on the issue. Unfortunately the issue is out of our direct control, however we will keep you updated as more information become available.

We are being impacted by an issue with our underlying Amazon AWS infrastructure (status.aws.amazon.com). We are actively looking into mitigations and will update here.

We have narrowed down yesterdays app issues to a memory problem that effected our database servers for around four hours. Unfortunately the redundancy of three servers that usually keeps the app up and operational through all kinds of day to day issues, this time caught us out because the scenario wasn't detected by the monitoring software that raises alarms when things go wrong. We have implemented a fix on the database servers which seems to have stabilised the memory issues. We've also improved the monitoring software to better detect issues like this in the future. Our apologies for the inconvenience.

A fix has been implemented and we are monitoring the results.

We are investigating reports of users having problems connecting to our app servers hosted on the west coast of the USA.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified an issue that is causing intermittent failures when generating previews. We are working to mitigate the issue...

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue is being actively worked on by the sendgrid.com support team. All deferred emails will be delivered when connectivity is restored...

There appears to be a temporary internet connectivity issue between our mail servers hosted with sendgrid.com connecting to mail servers in Australia and New Zealand. All deferred emails will be delivered when connectivity is restored...

There appears to be delays delivering email notifications to certain email addresses or mail servers. We are currently trying to determine why this is happening.

We have resolved the issue that was causing delays for preview tasks.

We have identified an issue that is causing delays for preview tasks in the Oceania region. We are working to mitigate the issue...

Amazon AWS has resolved the issue with 'Cloudfront Optimized Routes' and we have reverted the temporary work-around. You may need to cancel and re-start uploads for the fix to take effect.

A fix has been implemented and we are monitoring the results.

The work-around is now live. For the work-around to take effect you will need to cancel and restart your upload.

We are currently deploying an emergency work-around to mitigate the issue. ETA 5 minutes...

We are working with Amazon AWS engineers to resolve the issue. A work-around will be available within the hour.

There appears to be a global Amazon AWS issue that is effecting all uploads and downloads that utilise the 'Cloudfront optimized routes' feature. We are working to mitigate the issue...

We are currently investigating issues uploading and downloading files.

This incident has been resolved.

An issue with the scheduled maintenance has been identified causing problems with the app. To resolve the issue please log out and log back in...

We have identified and fixed an issue with the load management system that was causing delayed previews for files uploaded to the Europe (West) region.

We're just rolling out a fix for a bug that was live for the weekend that prevented some uploads from completing showing the message "Waiting for upload to complete...". The issue effected files that were forwarded from previous uploads...

Late Sunday afternoon/evening through to early Monday morning the Sydney data center experienced connectivity issues which have now been resolved. During this time downloads and preview processing from the Oceania (Sydney) servers were intermittently unavailable.

We experienced an issue with the file processing servers in the US-West region that caused file tasks to be delayed. We have resolved the issue and are investigating why the scenario didn't trigger monitoring alarms.

This incident has been resolved.

Fix is now live in all regions. The file processing infrastructure will take some time to to reach capacity.

We are currently deploying an emergency update to address a recent security exploit found in ImageMagick (http://www.theregister.co.uk/2016/05/03/imagemagick/). There may be some delays in generating previews and zip files...

This incident has been resolved.

We have identified an issue with custom designs not being applied correctly.

This incident has been resolved.

We've deployed a fix. Restore tasks are processing but please allow several hours for the system to catch up.

We are experiencing long delays in restoring files. The delays seem to be caused by specific restore tasks getting stuck which in turn blocks the restore process for other files. We are working to mitigate the issue but at the moment have no ETA.