Historical record of incidents for MileIQ
Report: "Degraded performance due to provider issues"
Last updateWe're currently experiencing processing issues with our application. This incident has been traced back to an ongoing situation with one of our providers. The team is working hard to fix this. We apologize to those impacted by the disruption.
Report: "Latency of MileIQ Services"
Last updateMileIQ applications are currently slow to respond. Our team is working hard to resolve this as soon as possible. We apologize for the inconvenience.
Report: "Creating Reports Temporarily Unavailable"
Last updateThis incident has been resolved and we are communicating to impacted users.
A fix has been implemented and we are monitoring results.
We are currently experiencing an issue with creating reports. Our team is working hard to resolve this as soon as possible. We apologize for the inconvenience.
Report: "Maps Temporarily Unavailable for All Drives Screen on Android"
Last updateMaps are operating, as intended, and displaying correctly for Android devices.
A fix has been implemented and we are monitoring results.
We have identified a solution, completed the fix and will begin rolling it out to users.
Maps are not displaying on the All Drives screen for Android devices. To access map information about drives, please use the web app (dashboard.mileiq.com).
Report: "Push Notifications and Email Delivery Issues"
Last updateOur third party provider has resolved the issue that impacted messages sent to users through push notifications and email. Messages through push notification and email are working, as expected.
Some users may not be receiving our messages through push notifications and email due to an issue with a third party provider. Our team is actively working with the third-party provider to resolve the issue and restore notifications as soon as possible. We apologize for any inconvenience this may cause. In the meantime, you can stay informed by: Manually checking the app for updates. We'll be providing updates on this status page. We'll update this message as soon as we have more information.
Report: "MileIQ Application Temporarily Unavailable in Google Play Store"
Last updateMileIQ is available for download on the Google Play Store. Thank you for your patience while we worked to resolve this confusing situation. We appreciate your continued support. Download/upgrade now, and hit the road with confidence!
We are still investigating why MileIQ is not currently available in the Google Play Store. Our team is continuing to work around the clock to resolve this as soon as possible. What does this mean for you? 1. If you already have the MileIQ app installed, it will continue to track your drives as usual. Your data is safe and secure. 2. If you want to upgrade your account to an Unlimited subscription, please log in here: https://dashboard.mileiq.com/user/subscription. We apologize for any inconvenience and will share updates when we have them. Thank you for your patience and understanding.
We are aware that MileIQ is temporarily unavailable for download on the Google Play Store. Our team is actively working to resolve this as soon as possible. If you already have the MileIQ app installed, it will continue to track your drives as usual. Your data is safe and secure. If you’re looking to upgrade your account to an Unlimited subscription, please head to our website, log in and upgrade in ‘Subscription Settings.’ We apologize for any inconvenience and will share updates when we have them. Thank you for your patience and understanding.
Report: "Push Notifications Delivery Issues"
Last updateOur third party provider is back online and notifications and emails are being sent. Message volumes have caught up for the day. Thank you for your patience with notification delays.
Our third party provider is starting to deliver messages to users. Some users are starting to receive push notifications and emails. Our team is actively working with the third-party provider to resolve the issue completely. We apologize for any inconvenience this may cause. We'll update this message as soon as we have more information.
Some users may not be receiving our messages through push notifications and email due to an issue with a third party provider. Our team is actively working with the third-party provider to resolve the issue and restore notifications as soon as possible. We apologize for any inconvenience this may cause. In the meantime, you can stay informed by: Manually checking the app for updates. We'll be providing updates on this status page. We'll update this message as soon as we have more information.
Report: "Error Rates"
Last updateToday we experienced a brief technical service disruption due to issues with our cloud provider. As a result, a small group of Android users could have been logged out of the app and could possibly be missing drives. To make sure your app is automatically tracking your drives, please follow the steps below: 1. Open the MileIQ app on your phone and make sure you are logged in. Once you’re logged in, all future drives will be automatically tracked. 2. Check your drive history and make sure no drives were missed. If you do not see your drive history, please check that you are signed in to the correct account. 3. If business drives were missed or deleted, you can manually add them following these steps If you have questions, please reach out to support@mileiq.com. We sincerely apologize for this interruption and will take continuous measures to make our service even more resilient.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We're currently observing elevated error rates on all of our apps which you may be experiencing. No drive data is being lost.
Report: "Unexpected force logout"
Last updateSome Android and iOS users may have experienced a logout with the the inability to log back in. We have corrected the issue that caused this and users should be able to login again.
Report: "Drive Processing Delays"
Last updateThis incident has been resolved.
Our data processing infrastructure is running behind which is causing a backlog of unprocessed drives. No data has been lost and the system should be caught up shortly.
Report: "Delay Processing delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We're currently experiencing a short delays with processing the drive data for a small subset of users. All pending drives will be processed shortly.
Report: "Users are experiencing issues accessing the MileIQ App and Dashboard"
Last updateWe understand that the recent service disruption on September 16, 2023, had a significant impact on your business operations, and we want to provide you with a detailed account of what transpired during this incident. **Chronology of Events:** * **07:33 UTC:** An issue surfaced that affected a large portion of customers using a [Azure’s mission-critical database service](https://app.azure.com/h/2LZ0-3DG/652dcd). * **Immediate Response:** Our internal monitoring systems alerted us to this issue, triggering an immediate investigation to assess the situation. * **Root Cause Identified:** After thorough investigation, we discovered that the core issue was linked to an unexpected power disruption in the underlying network infrastructure with our cloud provider. This disruption led to the temporary unavailability of certain compute nodes, which in turn caused failures and timeouts for SQL Database operations. * **Mitigation Steps:** To mitigate the initial impact and restore functionality as swiftly as possible, we took the following actions: * **Alternative steps of communicating:** Attempted vendor suggested workaround of connecting via a new tunnel - operation timed out due to the incident. * **Disaster Declaration:** We started executing on a previously prepared plan and restoring DB replicas in alternate regions, our backups are geo-replicated to account for this exact usecase. Due to the dataset size restore would have taken 15-20 hours from that point * **Complete Service Recovery:** Azure team confirms restoration of the service. We abandon the efforts to bring the service up in the alternate location and focus on restoration of services in the primary location. * **21:38 UTC:** We achieved complete service recovery, resolving the issue and restoring normal operation for all new sessions **Impact to our Service:** * Most of the application clients were able to automatically re-connect upon service restoration. * A small subset required users to re-login. * Most drives captured during the downtime were cached on the device and posted to the platform upon service restoration. * A small subset of drives during the incident became irrecoverable. Customers are advised to use manual entry. **Our Commitment to You:** * Our choice of infrastructure is our business, not yours, and we take full responsibility for this disruption. * We’re working with individual users who may have remnants of the issue or have questions via our support team * We’re pursuing investigation with our provider to understand why cross-AZ failover didn’t occur as planned. * We’re fixing all the reasons we have not persisted drives when the service was not available and ensuring high durability on -device. * We’re pursuing code improvements that would allow for us to buffer incoming data upstream of DB even with the loss of authentication.
We observe full service restoration.
A fix has been successfully implemented by Microsoft Azure, and our system is currently in the recovery phase. Users will regain access shortly, and we anticipate that drives will be posted within the next few hours.
A fix has been successfully implemented by Microsoft Azure, and our system is currently in the recovery phase. Users will regain access shortly, and we anticipate that drives will be posted within the next few hours.
Azure continues their mitigation efforts. We aim to provide the next update by 07:40PM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 06:00PM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 05:00PM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 04:00PM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 03:00PM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 02:00PM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 01:00PM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 12:00PM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 11:00AM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 10:25AM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 10:05AM UTC. Please check back for more information.
Azure continues their mitigation efforts. We aim to provide the next update by 09:45AM UTC. Please check back for more information.
Update: We're currently experiencing connectivity issues with our application. This incident has been traced back to an ongoing situation with our cloud provider, Azure. They have informed us of an issue impacting their services in our region, which has directly affected our application's ability to connect to its database. Impact: Users might encounter difficulties or be unable to access our application and related services until the problem is resolved. Rest assured, drives captured will be posted to your account once MileIQ is back online. We aim to provide the next update by 09:25 AM UTC. Please check back for more information.
We are continuing to investigate this issue.
Our team is investigating the issue.
Report: "incident management trial"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Subset of Android users is experiencing issues with drive detection functionality."
Last update**Lead-up:** * Google Play introduced a new mandatory version requirement * We released compliant version 1.64.xx in early Jan and have been observing normal operations for 3 weeks and several minor release iterations **Incident:** * On Feb 1 we started observing early signs of deteriorating detection on Android for a very small subset of user and started investigation * On Feb 4 volume of impacted users started approaching ~10% with disproportionate bulk of the affected users being on Samsung devices running Android 12 and 13 * On Feb 6 impacted users count escalated to ~15% and thorough our investigation efforts we have isolated the issue to be in the transport layer of the app * On Feb7 we started testing potential fix and created backend workaround to facilitate the drive capture * On Feb 8 we started seeing mitigation measures take effect and we rolled out an app fix, percentage of users started seeing recovery at that time * On Feb 9 we created additional app update and rolled it out to 100%, effectively resolving the issue **Learnings:** * We will be prioritizing transport layer robustness as a part of our ongoing product modernization * We were under-monitored in some areas and will focus on additional telemetry that would have sped up the detection and remediation * We will allow for more time to soak test mandatory Play Store target SDK updates
We've been observing normal operations for some time now on versions 1.64.6 and above.
We're observing normal operation and processing rates on the latest Android update available in Play Store. Please update to version 1.64.6 or above if you're still experiencing issues.
We've started rollout of a new version which includes improvements around drive detection stability and additional telemetry.
We have started putting remedial measures in place earlier today and observe a significant improvement in drive detection and processing operations.
We are continuing the investigation and are pursuing multiple paths in parallel. Our current belief is that the issue is related to the mandatory Google Play target version changes. We will provide further updates as they become available.
We see the problem to be prevalent on Samsung devices running Android 12 and 13. We're continuing investigation and will provide more updates as the become available.
Our team is actively investigating the issue.
Report: "Delays in receiving reports"
Last updateOur team has resolved an issue where report submission had issues. We've caught up processing all outstanding report requests and observe normal operation of reporting functionality.
Report: "We're currently experiencing drive processing delays."
Last updateDrive processing latencies have returned back to normal levels. All drives captured during this incident have been fully processed.
We're continuing to go through the outstanding backlog and are currently processing drives taken less than two hours ago.
Our processing of backlog steadily continues with earlier drives showing up in the app. We anticipate to be close to real-time in 3-4 hours from now, all pending drives should process by then.
We're observing clearing of backlog accumulated during this incident. We anticipate it will take few more hours to process all outstanding drives and return to near real-time processing.
We have implemented a fix and continue to monitor the issue.
We're still working on resolution to drive processing delays. All drives are currently queued up will be posted once we fully resolve the issue.
We're still working on resolution to drive processing delays. All drives are currently queued up will be posted once we fully resolve the issue.
We are continuing to investigate this issue.
Due to technical issues with our processing infrastructure, we are experiencing delays with drives processing. All drives are preserved and will be processed once we resolve the issue.
Report: "Issue accessing Web dashboard and App"
Last updateBetween 07:00 UTC and until 08:00 UTC on 30 August 2022, our main API went down resulting an outage. Our customers may have experienced issues while accessing the App and dashboard. The issue is resolved now
Report: "We're currently experiencing drive processing delays."
Last updateThis incident has been resolved.
We're observing return to normal near real-time processing for drives.
All your drive data is saved and will be processed as we work with our maps provider to fix the issue.
Report: "Issue while manually adding Drives with MileIQ"
Last updateThis incident has been resolved.
We have identified that Users currently experience issue while manually adding Drives with MileIQ
Report: "Drive Processing Delays"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We're currently experiencing a short delays with processing the drive data for a small subset of users. All pending drives will be processed shortly.
Report: "Database failure"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
At 18:05 UTC our main API went down, we are currently investigating the issue.
Report: "Drive delay for small portion of users"
Last updateWe've experienced a delay in processing a subset of drives. As of 4:26 PM UTC all the delayed drives have been processed and posted to user accounts.
Report: "Intermittent errors logging into the UI"
Last updateThis incident has been resolved.
We're observing operations return to normal
The issue has been identified and a fix is being implemented.
We are investigating an issue that started at 8:04AM PST
Report: "Drive Processing Delays"
Last updateThis incident has been resolved.
We have implemented a fix and are observing a return to normal levels. Drives should start posting normally.
We are continuing to investigate this issue.
Some users may experience drive processing delays. The system will catch up.
Report: "Drive processing delays"
Last updateOur processing has fully caught up to normal speeds.
The processing times are returning to normal levels
We've implemented several fixes and observe the backlog rate decreasing
We're currently experiencing delays processing the drive data.
Report: "Elevated error rates using apps"
Last updateBetween 11:04 AM PST and 11:09 AM PST users experienced elevated error rates in Mobile and Web apps.