Historical record of incidents for Deliverect
Report: "Outage Due to Cloud Provider Issue"
Last updateWe are currently experiencing a service outage due to an incident with our cloud provider. This disruption is impacting multiple systems, and our team is actively working to assess the situation and restore services as quickly as possible
Report: "Slow order processing"
Last updateSome of our customers experienced temporary delays in order processing due to an issue with a new feature. This feature, designed to improve how we handle old data, unintentionally created extra work for our system during a busy period. While our team resolved the issue, we understand how important it is for your operations to run smoothly. To prevent this from happening again, we’ve made several improvements, including better monitoring, stricter testing for new features, and adjustments to how our system handles errors. We sincerely apologize for any inconvenience this may have caused and thank you for your patience as we continue to make Deliverect more reliable for your business.
This incident has been resolved
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue within our API that is causing delays in order processing. Our engineering team is actively working to identify the root cause and resolve the issue as quickly as possible.
Report: "Degraded Performance"
Last updateDeliverect experienced degraded performance between 20:08 UTC and 20:53 UTC. Most customers should not have been impacted, but a small number may have experienced dropped orders, delayed orders, or other issues with Deliverect functionalities. The issue has been resolved and we are working to prevent future occurrences.
Report: "Degraded API Performance"
Last updateThe overall degradation in API response times has been solved. The fix we implemented has fixed the issue.
A fix has been implemented and we are monitoring the results.
We are currently investigating the issue.
Report: "Deliverect API Degraded Performance"
Last updateStart Time: April 30, 2024, 06:41 AM CEST (04:41 AM UTC) End Time: April 30, 2024, 06:48 AM CEST (04:48 AM UTC) We are backfilling an incident that occurred earlier this morning where the Deliverect API experienced degraded performance for a short period of time (approximatively 7 minutes). This resulted in slower response times and potential delayed order processing. Deliverect API is now functioning normally and we apologize for any inconvenience this may have caused.
Report: "Order processing delay"
Last updateCustomers might have experienced delay on order processing from 18:12 to 18:36 UTC+0 The incident is now resolved and we are back to normal order processing activities.
Report: "Processing Delays"
Last updateBetween 00:18 and 00:27 UTC+1, our system degraded and performance slowly declined. The incident was caused by a third-party provider experiencing issues. We sincerely apologize for the inconvenience.
Report: "Slow order processing"
Last updateThis incident has been resolved.
We've identified the cause of the incident and applied a fix for it. We'll keep monitoring.
We experienced order processing delays, but we can confirm that now all processing activities are back to normal. As a way to improve order processing performance, the menu-push feature has been paused until we have further information about its impact on the order flow.
We are currently investigating this issue.
Report: "Slow order processing"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Service Unavailability"
Last updateOperations were restored completely for affected customers. We are currently actively monitoring our systems.
A fix has been deployed and we are monitoring the results
After a thorough investigation the issue has been identified and a fix is being implemented.
We are continuing to investigate this issue. The impact was that our queueing systems was struggling to process orders to the POS systems, and restarting them only partially helped recovery.
Our monitoring system detected a sudden traffic decrease on one of our systems, which was impacting order processing. Requests are not correctly arriving to our systems we are actively investigating and setting all efforts to quickly resolve it.
Report: "Degraded performance for DMA and KDS solutions"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are monitoring the fix that was applied, and are implementing changes to avoid this from occurring in the future.
Starting at 10:28 (UTC+0), we faced potential delays over a period of 45min. All orders were eventually processed and the incident has been addressed and fixed.
Report: "Customers and partners may experience API and Frontend slowness"
Last updateThis incident has been resolved
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
You may notice that our Frontend is slower than usual at certain times, when it's the case our API takes more time than usual to respond to requests. This can lead to some orders being rejected or duplicated. Our team is actively working on resolving the problem and restoring normal operations as quickly as possible.
Starting May 21, 19:00 CET time we have been experiencing degraded performance with our API, which is impacting the speed and responsiveness of our frontend.
Report: "Customers using Frontend and Deliverect API might have experienced slowness"
Last updateCustomers and partners might have experienced slowness on the Deliverect Frontend and non-critical API calls between 8:00 PM CET and 9:45 PM CET. A fix has been implemented and the incident is now resolved.
Report: "Tunnel service outage"
Last updateAn issue with a TLS certificate impacted customers using the Tunnel but did not affect customers using DMA. We experienced system instability for around 50 minutes. The issue is related to the expiry of a TLS certificate, which is normally automatically renewed by our processes. Our Infrastructure and Support teams were notified of the incident and immediately began investigating the cause. The automation responsible for renewing this specific certificate was identified as the root cause of the incident. The team worked to replace manually the expired certificate with a new one so customers can use the tunnel again without disruption. The code responsible for the certificate renewal is now being reviewed to avoid this bug from happening again.
Report: "Updates failed to reach DMA"
Last updateWe’ve noticed some order updates did not reach the DMA. This issue was noticed between 15:19:45 (UTC+0) and 18:40:00 (UTC+0). We’ve identified the issue and applied a fix to it. We will keep monitoring.
Report: "Degraded performance"
Last updateThis incident has been resolved. Users may have experienced slowness and duplicate orders from 10:30 to 10:58 UTC+2
A fix has been implemented and we are monitoring the results.
We have identified a performance issue causing slowness for some frontend users. As a workaround, users can do a hard refresh.
Report: "Connectivity issues"
Last updateA major cloud services provider experienced a global outage for approximately 30 minutes. Even if Deliverect critical systems were up and running during the incident, customers may have experienced service unavailability if one of their POS or Delivery Channel providers was impacted by the incident.
Report: "Degraded Performance on some Online Ordering stores"
Last updateWe've experienced degraded performance with order processing from 09:47 AM (GMT+0) to 11:08 AM (GMT+0). The only product impacted was Online Ordering, where some stores have faced service interruption.
Report: "Degraded performance"
Last updateThis incident has been resolved.
Between 22:20 - 22:40 UTC+2, customers might have faced delays with requests to our application.
A fix has been implemented and we are monitoring the results.
Deliverect is aware of and investigating an issue that potentially impacts some customers causing HTTP connection timeouts.
Report: "Experienced degraded performance on order processing"
Last updateFaced degraded performance with order processing from 5:48 PM to 5:55 PM (UTC+2). From 5:50 PM to 6:18 PM (UTC+2) only the DMA had degraded performance.
Report: "DMA delayed notifications"
Last updateThis incident has been resolved. DMA notifications have been restored to normal.
We identified the issue and re-triggered all notifications to the DMA. The service should be back to normal. We'll keep monitoring.
A restricted number of DMA customers might have faced delayed notifications on the DMA between 16:52 and 16:56 UTC+2 time
Report: "Service instability"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results
The issue has been identified and a fix is being implemented. We experienced degraded performance between 18:40 and 18:48 UTC.
Deliverect is investigating delays regarding order processing
Report: "Service Unavailability"
Last updateThis incident has been resolved. All services are back to normal. The impact was limited to requests between 13h03 to 13h12 (UTC)
We already implemented a fix and services are back up. We'll keep monitoring.
Some of our services were unstable between 13h03 to 13h12 (UTC).
Requests are not correctly arriving at the destination we are actively investigating and setting all efforts to quickly resolve it.
Report: "Service instability"
Last updateThis incident has been resolved. The impact was limited to requests between 21:37 and 21:52 (GMT).
We already implemented a fix and services are back up. We'll keep monitoring.
Some of our services were unstable between 21h37 to 21h52 (GMT).
Report: "Degraded performance on scheduled orders"
Last updateThis incident has been resolved.
A fix has been implemented and our systems got back to normal. We will keep monitoring its behavior.
We have identified the issue with our partners who are working to resolve the issue as soon as possible. We are applying a fix on our side to reduce the impact.
We are continuing to investigate this issue.
Some scheduled orders are getting delayed.
Report: "Connection issues with Deliverect-Tunnel"
Last updateThis incident has been resolved.
We found a fix. We'll keep monitoring.
Locations using the Deliverect-Tunnel might be experiencing connectivity issues. We are investigating.
Report: "Network connectivity incident"
Last updateDuring a period of time one our servers was experiencing network connectivity problems, this led to a small percentage of network requests to fail. A fix has been applied and all seems to be back to normal. We are continuing to monitor for any further issues.
Report: "POSist orders partial outage"
Last updateThis incident has been solved.
A fix is in place (and being propagated), we are monitoring the status of it.
We have authentication issues between Deliverect and POSist
We are currently investigating what's causing the issue. We are in contact with the POSist team to solve the issue.
Report: "Partial interruption in our orders operations"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
It looks like orders were delayed for around 40 mins since 12h03 CEST. A fix has been implemented and we are monitoring the results.
A fix has been implemented and we are monitoring the results.
The issue has been identified and we are currently working on a solution
We are continuing to investigate this issue.
We are currently investigating an interruption in our orders operations
Report: "Order inflow error rate increased"
Last updateAfter a planned weekly release we noted a small uptick in the failed order ratio. First we rolled out a fix but later realized the impact was deeper and rollbacked to a previous release. The whole incident took around 45 minutes and resulted in a number of cancelled orders for certain channels.
Report: "Message overflow caused degraded performance"
Last updateA misconfiguration in our background tasks was generating an increase in the number of messages which result in a high volume of traffic on our infrastructure. Some of the services were experiencing longer response times as a consequence of this event.
Report: "Possible loss of scheduled orders"
Last updateLoss of a very small amount of scheduled orders after a cluster update. Due to a problem in scheduled orders persistence when a cluster update was triggered some scheduled orders were not delivered to the POS. By the time the problem was reported, the team had the orders re-scheduled. During this window there were possibly more than one scheduled order lost.