Historical record of incidents for Device Magic
Report: "Investigating API performance degradation"
Last updateWe're investigating reports of degraded performance with our Submissions and Dispatches APIs.
Report: "System Maintenance"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We will be performing system maintenance on 4/20/2025 from 8:00am EDT - 9:00am EDT. No service interruptions are expected during this time. We appreciate your patience and understanding. As always, if you have questions or issues please submit them to support@devicemagic.com
Report: "Degraded API Calls"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We're experiencing high load on the platform, as a result loading web stores and api calls to the dm database or related submission api calls are degraded.
Report: "Submission Delays"
Last updateThis incident has been resolved.
We are currently experiencing some delays in delivery processing. The issue has been identified and a resolution will be in place shortly.
Report: "Submission Delays"
Last updateThis incident has been resolved.
We are currently experiencing some delays in delivery processing. The issue has been identified and a resolution will be in place shortly.
Report: "Dropbox Destination Delivery Issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing an issue where Dropbox Destination deliveries are failing to complete, causing delivery errors. Our engineering team is actively investigating and working on a fix. Thank you for your patience.
Report: "Submission Delivery Delays"
Last updateThis incident has been resolved.
We are currently experiencing some delays in submission deliveries. The issue has been identified and a resolution put in place. Our apologies for any inconvenience this causes.
Report: "Page Loading Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently aware of an issue causing slower-than-normal page loading times for some users. Our team is investigating the cause and working to resolve it as quickly as possible. We apologize for any inconvenience and appreciate your patience. Please stay tuned for updates. This delay is NOT affecting the mobile app or submission deliveries.
Report: "Submission Delays"
Last updateThis incident has been resolved.
We are currently experiencing some delays in delivery processing. The issue has been identified and a resolution will be in place shortly. Thank you for your patience!
Report: "System Delays"
Last updateDelivery times have returned to normal.
Due to higher than normal volume the system is experiencing delays in deliveries. The team has added more horsepower to the system to work through the backlog as quickly as possible. We apologize for any inconvenience this may cause and thank you for your patience.
Report: "Missing data in submissions from iOS"
Last updateAll submissions have been processed behind the scenes and we have been monitoring with no new issues. We will be rolling back the mobile client changes in the next update so that local submissions are automatically removed as they were before.
We will continue to monitor submissions to ensure that this issue is resolved and will be releasing an update to the iOS client to remove the temporary change to ensure submissions were not removed from the device.
Our team have identified the root cause and we are able to fix this server side. We are actively reviewing for all instances where this occurred and are re-processing the submissions. We expect to have all submissions re-processed in the next 24 hours.
We believe we have found the core problem and are working on testing a fix. Fortunately, we appear to have all of the data from the submissions and should be able to recover and re-process any submissions with the issue. If you notice anything missing, please contact support to make sure that it is being tracked.
While we continue to investigate the issue, we have released iOS release 2024.4.3. This release will retain submissions past the "Days Submissions Stored" date to ensure there is no data loss.
We have released 2024.4.2 that should address this issue, and recommend all users on 2024.4.0 or 2024.4.1 update to this release as soon as possible. We continue to investigate the root cause and will update here when we have a full resolution. If you are experiencing data loss problems, the only place where the full copy of the data resides is on the device. This data is seen as completed on your device and will get automatically deleted based on the "Days Submissions Stored" value. Action: If you notice any problems with data missing, you can set the value for "Days Submissions Stored" to some larger value (for example 30 days) to ensure that the local copies are not deleted until you have verified that there was no data missing in the cloud copy.
We are investigating cases where submissions from iOS devices running builds 2024.4.0 or 2024.4.1 may have some of the data in the submission not be received in the cloud.
Report: "Resource Downloading and Dispatch Loading Issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The team has a temporary fix in place at the moment and users should see improved speed in downloading of resources/dispatches. Please recommend to all users to update their apps to the latest versions. - Device Magic Android = 2024.4.0 - Device Magic iOS = 2024.3.5 There is still an ongoing investigation into the underlying cause of the issue. We appreciate your patience and understanding and apologize for any inconvenience this may cause.
Some customers are currently experiencing mobile app issues with "Downloading Resources" and slow loading dispatches. Our team is actively working to resolve the problem as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience. Thank you for your understanding.
Report: "Website Performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently experiencing an issue with website performance and submission deliveries. The team is currently investigating.
Report: "Support email address not receiving emails."
Last updateThis incident has been resolved.
Contacting the Support Team via email at support@devicemagic.com is currently unavailable. To reach the support team please log into your Device Magic account and click on the 'Chat' icon located in the lower right corner of every page. Thank you for your patience and understanding as the team investigates the issue.
Report: "SSO Login via Mobile Device"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing an issue with using SSO to log in via a mobile device. The team is investigating the issue.
Report: "Database outage"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented
We experienced an issue with one of our databases. Our team is working on a fix.
Report: "Undelivered Submissions / Unknown Delivery Error"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue in which a small portion of our customers are seeing 'Undelivered Submissions' and a 'Delivery Issue' that has very limited information. The team is currently working to identify what is causing the issue. These deliveries will resolve themselves over time, and there is no threat of any data loss. We appreciate your understanding and patience.
Report: "Mobile Device Loading Delays"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently experiencing a delay in communication with mobile devices. This affects loading of forms, sending submissions, loading dispatches, and loading resources. The team is working as quickly as possible to identify the issue. We apologize for any inconvenience and appreciate your patience.
Report: "SMS Text Delivery Issue"
Last updateThis incident has been resolved.
SMS message sending should now be fully operational again
We are working with our provider on this issue and expect to have a solution soon. Note: This issue only impacts SMS deliveries to US phone numbers.
We are currently experiencing degraded performance with SMS text deliveries. 15-20% of deliveries are affected. The team has identified the issue and is currently looking into a resolution.
Report: "Mobile Device Loading Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented
We are currently experiencing a delay in communication with mobile devices. This affects loading of forms, sending submissions, loading dispatches, and loading resources. The team is working as quickly as possible to identify the issue. We apologize for any inconvenience and appreciate your patience.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved. System maintenance occurred during the night and one of the changes made had a negative affect causing the incident this morning. The changes have since been reverted and all systems a performing as normal now. We appreciate your patience.
A fix has been implemented and we are monitoring the results.
We are currently experiencing longer than usual submission delivery/resending times. We have identified the issue and are currently working to resolve.
Report: "Submission Delivery/Resending Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing longer than usual submission delivery/resending times. We are investigating the issue and will provide an update when there is one available.
Report: "Submission Delivery/Resending Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing longer than usual submission delivery/resending times. We are investigating the issue and will provide an update when there is one available.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing longer than usual submission delivery times. We are investigating the issue and will provide an update when there is one available.
Report: "Submission Resending Delays"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing an issue with Submission Resends being delayed. We are investigating the cause and will provide an update as soon as one is available.
Report: "Submission Resending Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing an issue with Submission Resends being delayed. We are investigating the cause and will provide an update as soon as one is available.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently experiencing longer than usual submission delivery times. We are investigating the issue and will provide an update when there is one available.
Report: "Submission Resending Delays"
Last updateThis incident has been resolved.
An issue was discovered that is affecting the timeliness of submission resend deliveries. A fix has been implemented and we are monitoring closely. The backlog of resent submissions will work through on it's own as quickly as possible.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
We have identified an issue with the 'Long Running' delivery queue. Submissions that typically take 1+ hours to process to PDF destinations are experiencing longer than normal processing times. A fix has already been put in place and these jobs will continue to process through as the system works through them. We apologize for the inconvenience and appreciate your patience!
Report: "Delivery Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently experiencing a delay with deliveries/dispatches. We are currently investigating the issue and will provide updates when available. Thank you for your patience! The Device Magic Team
Report: "AWS Internet Connectivity"
Last updateFrom AWS: [RESOLVED] Internet Connectivity 4:06 PM UTC-5 Between 2:34 PM and 3:51 PM UTC-5, customers experienced Internet connectivity issues for some networks to and from the US-EAST-2 Region. Connectivity between instances within the Region, in between Regions, and Direct Connect connectivity were not impacted by this issue. The issue has been resolved and connectivity has been fully restored.
AWS has reported an issue with possible internet connectivity between some customer networks and the US-EAST-2 Region. Status can be seen here: https://health.aws.amazon.com/health/status Possible issues with connecting to your Management Console could be noticed. We are currently monitoring the situation. All form submissions and deliveries are working as normal.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
New submissions are now delivering at normal rates. Submissions 'resends' are delivering at normal rates. The system has worked through a majority of the backlog and most deliveries should now have been made. We are continuing to monitor ALL systems and will update the status page to reflect any changes. We thank you for your continued understanding and patience!
Both Single and Multiple Dispatches are now fully functional.
Valued Device Magic Customers, As we approach the end of our business day here in the US we want to extend an apology for the inconveniences and frustrations that the delays in our systems over the last two days have caused. We understand how important our product is to your business and day-to-day operations. Multiple issues that led to this system outage have been identified and fixes have been put in place. Due to the complexity of the issues and the length of time to a full resolution our systems are working through an EXTREMELY large backlog at this time. Please know there have been NO data losses and ALL submissions will deliver as the backlog is processed but this will take some time. Attempting to ‘Resend’ submissions via the Management Console is not necessary and will only add to the backlog. Multiple dispatch requests are also experiencing delays as well at this time and the team is actively seeking a resolution. Our offices will be closed tomorrow (11-24-2022) in observance of Thanksgiving but the team will be actively monitoring the situation. We appreciate your patience and understanding as we work through this. For those of you in the US we wish you a Happy Thanksgiving!
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We thank you for your continued patience. A fix for the issue has been implemented. Due to the nature of the resolution and the length of the delay, submissions will take above average times to process as there is a large backlog for the system to work through. ALL data is safe and has been retained.
We have identified the issue and are have taken steps to remediate the problem and resume normal processing of submissions. All data is safe and should be processed within the day.
We are currently experiencing a delay in delivery of submissions. We apologize for this inconvenience and appreciate your patience. We expect destinations to be processed within the day.
Report: "Multi-Device Dispatches Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing an issue with the delivery of Multi-Device dispatches. Single device dispatches are working as normal. The issue has been identified and a fix is currently being worked on. Thank you for your patience!
Report: "Submission Delivery/Resending Delays"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are currently experiencing a delay in 'Resending' of Submissions and 'Retry' of Delivery Issues. New submissions are experiencing delays as well. We apologize for this inconvenience and appreciate your patience.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing a delay with dispatches and submission delivery. We are currently investigating the issue.
Report: "System/Delivery Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing system delays and are investigating the issue.
Report: "Submission Resending Delays"
Last updateThis incident has been resolved.
We are currently experiencing a delay in the processing of 'Resending' submissions. New submissions are not currently affected. We are looking into the cause of this incident and will work to resolve as quickly as possible.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating a delay in submission deliveries. Additional capacity has been added and delivery times should be back to normal shortly.
Report: "Partial outage due to AWS infrastructure network issues"
Last updateAWS outage has been resolved. All services are back online.
AWS is experiencing network connectivity issues in the same region as our servers. Our team is working with AWS Support to get all Device Magic services back to normal operations. For more information on the underlying AWS issue check - https://health.aws.amazon.com/health/status
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating a delay in submission deliveries. Additional capacity has been added and delivery times should be back to normal shortly.
Report: "Submission Delivery Delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating a delay in submission deliveries. Additional capacity has been added and delivery times should be back to normal shortly.
Report: "Submission delivery delays"
Last updateThe spike in background jobs has been processed. Deliveries are now under normal processing times.
We are currently investigating a delay in submission deliveries. Additional capacity has been added and delivery times should be back to normal shortly.
Report: "Submission delivery delays"
Last updateThe spike in background jobs has been processed. Deliveries are now under normal processing times.
Submission deliveries are delayed. We are currently investigating the issue.
Report: "Submission delivery delays"
Last updateThe spike in background jobs has been processed. Deliveries are now under normal processing times.
We are currently investigating a delay in submission deliveries. Additional capacity has been added and delivery times should be back to normal shortly.
Report: "Email and PDF delivery issues"
Last updateWe have identified the root cause and fixed it. All affected deliveries will get processed shortly.
We have detected a large volume of submissions deliveries failing due to email and PDF service issues. We are currently investigating the root cause.
Report: "Submission delivery delays"
Last updateDelivery times have returned to normal.
We are currently investigating a delay in submission deliveries. Additional capacity has been added and delivery times should be back to normal shortly.
Report: "Google Sheets API issue"
Last updateThe patch has been released and confirmed. The Google Sheets api issue is resolved.
We are manually processing any failed deliveries for now while we monitor the proposed patch. Once all deliveries are processed without further issues we will release the patch.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
A small number of customers may still experience intermittent delivery failures to Google Sheet destinations. We have identified and are testing a patch and will update when resolved. If you have any questions, please reach out to support@devicemagic.com for assistance. We sincerely apologize for any inconvenience this may cause.
Report: "Limited access to web console"
Last updateServers are back to normal. Access to web console restored.
A fix has been implemented and we are monitoring the results.
A software patch was applied on one of our databases which triggered the downtime. Our servers are getting back to normal. Our team is closely monitoring the situation.
Our engineers have been notified and are investigating the issue. We’re really sorry to be holding you up today! Please know our teams are working hard to get everything up and running
Report: "High Web Server Load"
Last updateThis incident has been resolved.
We're seeing an unexpected spike in web server load. We are investigating.
Report: "High Web Server Load"
Last updateWe're seeing an unexpected spike in web server load combined with out of memory errors on 2 of the 4 application servers. Additional servers are being added while we investigate.