Historical record of incidents for Wakeo
Report: "Flat File Integration Outage"
Last updateThis incident has been resolved.
Following today's maintenance of our Flat file integration infrastructure our system is currently down. We are working actively to restore it and we expect to be back online by 08/11 EOD. We are really sorry for the inconvenience this issue is causing and we remain at your disposal.
Report: "Air China outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are encountering a complete outage on our connection with Air China. Our engineers are working on the topic to restore tracking as soon as possible.
Report: "Internal infrastructure outage"
Last updateThe incident has been resolved. We will return to clients impacted by 400 answers to assist them.
We are facing an outage of an internal infrastructure. Following consequences have been identified: - Latency in retreiving Track & trace data on Sea shipments - Trusted Routes complete outage - API V2 POST Shipment with Sea segment might respond a 400 instead of a 200 We are working on a fix to mitigate the impact.
Report: "Webapp - Major outage"
Last updateFrom 18:47PM to 18:55PM CET, we experienced a complete outage of the Webapp interface. The incident has been resolved.
Report: "Analytics - Data not refreshed"
Last updateThe incident has been resolved. All data in the Analytics section of the Webapp is now up to date.
Our provider Microsoft Power BI at the source of the incident has corrected the issue. We are now refreshing all data in the Analytics section of the Webapp.
The issue has been identified, it comes from our provider Microsoft Power BI which is encountering an incident itself.
The data in the Analytics section of the Webapp have not been refreshed since 02/07 10:00AM. We are currently investigating the issue.
Report: "Webapp and API v2 - Major outage"
Last updateFrom 11:37AM to 11:51AM CET, we experienced a complete outage Webapp interface and all API v2. The incident has been resolved.
Report: "API timeout"
Last updateThe incident has been resolved. All shipments created without providing a shipment_id have been removed from the database.
A fix has been implemented and we are monitoring the results.
Since 11:30AM timeouts (error 500) occur on API V2 Shipments routes. This results in the creation of shipments without providing a shipment_id in response. We have identified the issue and are working on a fix.
Report: "Trusted Routes - Web interface outage"
Last updateThe incident has been resolved.
We are encountering a complete outage of the Trusted Routes web interface. Search can be launched, however will not provide any response. Trusted Routes API is not impacted. The origin of the outage has been identified. We are working on a fix.
Report: "API timeout"
Last updateThe incident has been resolved. All shipments created without providing a shipment_id have been removed from the database.
Between 9:00AM and 5:00PM (CEST) timeouts (error 500) occured on API v2 routes. This resulted in the creation of shipments without providing a shipment_id in response. A fix has been implemented and we are monitoring the results.
Report: "GET outage on Locations API"
Last updateA release from Thursday Feb 29th at 4PM UTC caused an error on the Locations API, preventing users from getting all locations at once and returning a 500 error. No other API was impacted. The issue was resolved on Friday 1st of March, 11AM UTC and the Location API is now back to normal.
Report: "High latency on email delivery and webhooks retries"
Last updateAll delayed tasks have been processed by 8PM CET. All our systems are back to the normal, and we'll deploy improvements to prevent this issue in the next days.
We are continuing to work on a fix for this issue.
A release from this morning is causing high latency on the following scopes: - Alert email delivery - Webhooks retries - Position freshness on the homepage's map (caused by our mitigation actions) We have already updated parts of our infrastructure to increase our processing rate in the next couple of hours, and are currently deploying a fix to mitigate the latency. Improvements have already been identified and will be implemented in the next few days.
Report: "High response time"
Last updateThis incident has been resolved.
An internal operation is currently causing high response time on both API and Webapp. System remains operational, and the slow-down is expected to end in a few minutes.
Report: "POST/PATCH outage on Shipments API"
Last updateThe fix has been deployed.
The issue has been identified and a fix is being implemented.
Our Shipments API is currently facing an issue where POST/PATCH are returning 400 errors, even if data has been well integrated.
Report: "Webapp unreachable"
Last updateAn issue in our hosting system configuration has caused our webapp to be unreachable from 10:46 AM to 10:59 AM (the time to detect the issue and to redeploy the previous version). We do apologize for this unexpected failure, and are currently working on workflows improvements to prevent it in the future.
Report: "Features were temporarily unavailable"
Last updateDuring a sensitive release on our webapp, a few features have been unexpectedly disabled between 10:20 and 10:25 CET. This was only a display issue, and mainly affected embedded users. Our backend and APIs where fully functional the whole time.
Report: "Analytics Module data calculation issue"
Last updateAll Analytics Dashboards are now refreshed.
We have identified circumstances leading to memory exhaustion during analytics data refresh. Our system has been adjusted to prevent high memory usage. All Analytics Dashboards are currently being refreshed and should be up again in less than 2 hours. We'll closely monitor refreshes for a few days, to ensure this issue is clearly resolved.
We're facing an issue preventing data from being calculated in our Analytics Module. Investigation is ongoing to identify the origin and solve the issue.
Report: "Webhooks high latency"
Last update### Incident Cause After deeper investigation, we found that this latency was caused by a 3rd-party service’s disruption. This disruption suddenly caused longer processing time on many jobs on our side, and went up to a global high latency on jobs and webhooks calls in the end. ### Actions taken We already have taken immediate actions to mitigate the impact of future 3rd-party service disruptions on our system.
Our job scheduler suffered an outage this night. It caused no data loss but high latency on webhooks and several other scheduled actions. You may have experienced up to 4h latency on your webhook calls. All pending jobs have now been processed, the situation is back to the normal.
Report: "Webhooks outage"
Last updateFollowing an update to prevent performance degradation, we noticed that webhooks were not called anymore. The outage lasted from 1/13, 10:22 AM to 1/14, 09:58 (UTC). The cause has been identified and solved, and we'll be able to call back all missed webhooks without loss.
Report: "High API response time and Webapp outage"
Last updateA set of specific conditions led to very high memory usage from 6:48 PM to 6:57 PM CEST. For the duration of the incident, most of the Webapp components where not responding, and you may have experienced high API response time, depending on the route used. We are aware or the origin of this issue, and will soon plan an infrastructure update to prevent it.
Report: "Increased response time on Webapp and API"
Last updateNo timeouts have been detected over the last few hours. Although we consider this incident as resolved, database will remains under active monitoring.
API request rate throttling has been removed. Database remains under active monitoring.
Emergency actions have been taken to free database memory, timeouts should not occur anymore. Issue possible origins amount has been narrowed, but not clearly identified. We'll actively monitor our database metrics to gather more informations.
API is now impacted, and timeouts may occurs. Engineers are still working on identifying the issue.
We are currently facing a database performance issue impacting some Webapp pages response time. API request rate has been temporarily throttled to reduce database stress, but should not be impacted. Engineers are actively working on identifying the issue.
Report: "Global slow-down on API and Webapp response time"
Last updateAPI request rate limitation was set back to its original value. No other performance issue has been detected over the last few hours. This incident is resolved.
The performance issue has been identified by our engineers. A bugfix has been deployed, and we're still monitoring both API and Webapp to ensure there are no other hidden issues. Webhooks have been slowed-down, but will be rescheduled without any loss.
We are currently facing a database performance issue impacting both API and Webapp response time. API request rate has been temporarily throttled to reduce database stress. Engineers are actively working on identifying the issue.
Report: "Issues accessing common runtime applications"
Last updateEngineers have identified an issue for Common Runtime apps and actively worked on resolving the issue. This incident impacted both Web App and APIs. The outage duration was 1 hour 42minutes.