Historical record of incidents for Paazl
Report: "Staging environment: update"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We will be performing an update on our Staging environment. This will take around 30 minutes to finish.
Report: "DHL Parcel PUDO keyword identifiers"
Last updateLast week, we deployed an endpoint migration at around 11:30 on Tuesday 13 May. On Thursday, we discovered that the migration had caused an unexpected response. As a result, PUDO shipments for DHL Express Ecommerce \(DHL For You and DHL Parcel Connect\) received an incorrect keyword identifier. The impact was that 3PLs received incorrect keyword identifiers and were unable to generate labels. Of the labels that could be generated, DHL was unable to recognise the identifier, resulting in shipments being re-routed to the recipient's home address, which caused potential delays. To resolve the situation, Paazl implemented a fix on Friday 16 May at around 15:30, additionally we updated committed orders that had not yet been shipped in the Paazl portal by overwriting the incorrect keyword identifier with the correct version.
This incident has been resolved.
We have created a fix for this. All orders that were already entered in Paazl but did not yet have a generated label now have a corrected keyword identifier, and all new orders will be processed as usual.
At this moment the DHL Parcel keyword identifiers for a PUDO delivery are not being processed correctly. This may affect the label generation via third-party logistics (3PL) providers. In cases where the label has already been created, affected parcels may be redirected to the recipient’s home address instead of the intended service point. Our team is actively working on a solution and will provide updates as soon as more information becomes available.
Report: "UPS: service not responding"
Last updateUPS confirmed that everything should be fully operational again.
We are noticing that the UPS web service is not responding properly. This may affect the printing of UPS labels. Example of an error code: "[server_error]" UPS is aware of the situation and is working on a solution.
Report: "Degraded performance in checkouts"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and repairs are ongoing.
We are seeing higher error-rates in our monitoring which is affecting our response times. We are currently investigating this issue.
Report: "Paazl Web Application"
Last updateThis incident has been resolved.
We have noticed that the response of our Web Application, Checkout Widget, and API had degraded performance. We have immediately identified and rolled out a fix. Our performance is back to normal at this moment. We are investigation the issue further and will come back with more information as soon as possible
Report: "Paazl Web Application: slow response"
Last updateThe fix of yesterday has resolved the issues. Please see the further timeline and impact below. - Timeline 18:57 Paazl monitoring picks up issue with cache cleaning mechanism resulting in partial outages 19:00 Customers begin reporting issues with labels and webapp login 19:20 IT Teams finds the issue 19:31 IT team implements fix 19:39 First customers notice improvements 19:51 IT team confirms that fix has worked for all customers 19:52 Statuspage updated to reflect fix - Technical impact The cache cleaning mechanism that we rely on throughout our platform is set to evict old cache entries whenever it’s full. In this instance it did not do so. We have identified the issue in this mechanism and fixed it. The load testing that we performed in preparation for Black Friday were volume tests, performed over a specific period of time. This kind of stress test does not replicate the cache filling of a full day of volumes. The most significant impact of this issue, related to the generation of shipping labels, and the logging into our webapp. During the affected period, we noticed a potential issue with labels that were requested. The barcodes created may not have been pre-registered at the carrier. We advise you to verify in your carrier portal whether the created barcodes are known to them. If they are not, then please consider re-labeling those packages. We remain confident in our preparation for this peak season and the steps we have taken to make it a success
We have noticed a potential issue with labels that were requested during the time period of this issue. The barcode may not have been pre-registered at the carrier. Please verify in your carrier portal whether the barcodes are known to them. If they are not, then please consider relabeling the packages.
A fix has been implemented and we are monitoring the results.
We are noticing issues with our Web Application. We have identified the issue and are performing corrections.
Report: "Carrier performance dashboard and shipment tracking not being updated with new data"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "GLS API is unresponsive"
Last updateThis incident has been resolved.
All connections to GLS are currently not operational. We are awaiting an update from GLS.
Report: "DPD webservice timeout"
Last updateDPD has resolved the issue
DPD is currently experiencing an issue which makes their services unavailable. The issue is on their side so we are awaiting updates.
Report: "DPD Parcelshops not being retrieved."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have found the cause of this issue and are implementing a fix.
We are currently not getting results when trying to fetch DPD Parcelshops locations. We are investigating the issue and will update the statuspage accordingly.
Report: "Paazl shipping labels not being generated"
Last updateYesterday we experienced degraded performance due to two different issues. First, A server upgrade ended up in us losing a previously configured time zone, which was essential for some of our platform functionality. This caused issues with printing labels. This was solved within the hour. This fixed the label issues for most customers. Setting our status page to monitoring. During the monitoring phase, a second issue was reported by a few customers around 13:15. The SOAP generateLabel response was changed to have a namespace prefix ns2:. The introduction of this non-blank namespace prefix in our SOAP responses caused issues with some integrations using XML parsers not equipped to deal with namespace prefixes. This was fixed with a rollback around 15:30. Unfortunately, this is an issue undetectable by heavy testing, as this is coming from OMS/WMS specific settings. We are currently working on a fix to suppress the namespace and intent to roll out this release today during the afternoon. However, to avoid similar issues we also heavily advise our customers to update their parsing mechanism for scalability.
This incident has been resolved.
We performed a rollback and are currently monitoring the performance.
The problem has been identified and we're currently initiating a rollback.
The issue has not been completely resolved in every instance. We're currently looking into it again.
We have identified the issue and rolled a fix.
Paazl is currently experiencing an issue with generating labels. We're looking into it and will post updates as soon as we have more information.
Report: "Colissimo Servicepoints unavailable"
Last updateThe issue has been resolved, the Colissimo locator is operational again.
We have noticed an issue with retrieving available servicepoints, and we are following up with the carrier for further support. In case there is more information known, we will update this notice.
Report: "Paazl Web Application: slow response"
Last updateThis incident has been resolved.
The issue has been resolved around 15:35. There was a temporary impact on the order overview which caused orders not to be shown properly. The orders were known to our software and pre-registrations were send properly. We will be monitoring this situation.
We have noticed that an issue with the connection between some of our servers. We have contacted our provider and are looking into the issue. This impacts the performance of our email notification, order overview and some of our pre-registration towards carriers.
Report: "PostNL SFTP server unavailable"
Last updateThis incident has been resolved.
We are currently experiencing issues with PostNL's SFTP server, and have reached out to the carrier for further support. At the moment, no preregistration files or status data are processed for our self-built shipping options. We will update this alert if more information is available on this matter.
Report: "DPD Meta API unavailable"
Last updateThis incident has been resolved.
Due to an outage at DPD, DPD META API is currently unavailable. We are reaching out to DPD for further support. As soon as the DPD web service is operational again, we will update this alert.
Report: "Mondial Relay: API Time out"
Last updateThis incident has been resolved.
We have noticed that services from Mondial Relay are currently unavailable. We are following up with the carrier in order to solve this issue as soon as possible.
Report: "Budbee: API Time out"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have noticed that services from Budbee are delayed and time outs are experienced. We are following up with the carrier in order to solve this issue as soon as possible.
Report: "TNT Express: API Time out"
Last updateThis incident has been resolved.
We have noticed that services from TNT Express are unreachable at the moment. We are following up with the carrier in order to solve this issue as soon as possible.
Report: "Brief maintenance live servers"
Last updateThis incident has been resolved.
We have implemented a fix for our platform status updates, due to an issue in our platform the status events of several carriers were not processed properly. With this fix the pending status events will be processed today and tomorrow. Please note, these could trigger the e-mail notifications to be send and may impact the performance of our PUSH API as the backlog is being processed. If you have any questions, please contact us by email: support@paazl.com.
Report: "UPS Servicepoint locator degraded performance"
Last updateThis incident has been resolved.
We have noticed that several postal codes cannot be linked to proper geo coordinates, and as a result, correct service points are not displayed in the proximity. The issue has been identified and a fix is being implemented.
Report: "No shipping options available"
Last updateAt around 09:55, we noticed the first signs of a degraded performance. Within 15 minutes, at 10:10, we managed to determine the root cause being related to the allocation of traffic per instance. We were able to implement a fix that was fully operational at 10:25. Immediately after the fix, the stability of our platform was restored with response times quickly returning to normal levels.
The incident has been resolved and our performance is back to normal.
We've implemented a fix and we will keep monitoring the situation
We are currently investigating an issue regarding the visibility of shipping options. We are looking into the matter with the highest priority.
Report: "Staging environment: inaccessible"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating reports about our Staging environment being inaccessible. We will post an update within an hour.
Report: "Bpost: offline"
Last updateThis incident has been resolved.
We've received an update from Bpost that the issue is resolved. We will monitor the situation.
The Bpost webservice is not responding. Because of this outage, you may experience issues when attempting to print shipping labels for this carrier. We've contacted Bpost and are awaiting a status update. As soon as we receive an update, we will post it on this page. We thank you for your patience.
Report: "Bpost: offline"
Last updateThis incident has been resolved.
Our monitoring is no longer showing any errors, we tested and are able to retrieve labels again. We will monitor the situation.
The Bpost webservice is not responding. Because of this outage, you may experience issues when attempting to print shipping labels for this carrier. We've contacted Bpost and are awaiting a status update. As soon as we receive an update, we will post it on this page. We thank you for your patience.
Report: "Staging webservices: no response"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating reports about our Staging webservices that are not responding. We will post an update within an hour. Our apologies for any inconvenience caused.
Report: "UPS : Authentication"
Last updateThis incident has been resolved.
UPS has notified us that they are investigating the issue. We will post further updates as they arise.
We have been notified that UPS is giving an Authentication error: Unable to validate shipping order at UPS: (250002) Invalid Authentication Information We have reached out to UPS for more information
Report: "Checkout widget: pickup location bug"
Last updateOn Thursday the 7th of September, we released a new version of the checkout widget. In this release there was a bug present which impacted the pickup locations. The bug altered the behaviour where the previously selected pickup location was saved in the checkout instead of the selected pickup location. If customers kept the default selected pickup location, then there was no impact. Today we released a fix for this bug at 15:30 and the widget has returned to normal functioning. We would like to apologize to our customers for the experienced inconvenience. If you have any further questions, feel free to contact us by email or phone at support@paazl.com or +31 (0)20 - 773 - 63 03.
Report: "Partial Outage: Web app, Widget and labels"
Last updateOn Friday the 8th of September, we noticed our webservice response for creating labels and orders slowed down. Our customers were experiencing issues with creating labels, committing orders, or loading of the widget services to our platform. After an internal investigation and technical support from our hosting provider, we discovered the root cause. Based on this investigation we can provide the following timeline. · 4:30 our monitoring provided the first internal messages · 5:30 the amount of monitoring messages increased · 6:30 our 24/7 support team was notified and started their investigation · 7:30 we found the root cause and our team started working on a fix · 8:00 a fix was deployed · 8:15 monitoring and testing showed all systems were clear and back to normal functioning. In order to avoid a similar occurrence we will increase the resources of our servers. We would like to apologize to our customers for the experienced inconvenience. If you have any further questions, feel free to contact us by email or phone at [support@paazl.com](mailto:support@paazl.com) or \+31 \(0\)20 - 773 - 63 03.
This incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and all services are performing fully again. We will continue to monitor performance.
We are continuing to investigate this issue.
We are currently seeing some partial outages that might impact ship environment, checkout Widget and label service. We are currently investigating this with highest priority.
Report: "Paazl Widget: unavailable"
Last updateThe incident has been resolved. We will work on a Post Mortem and will follow with details as soon as available.
A fix has been implemented and widget should be fully active again. We are monitoring performance.
Paazl Widget is currently unavailable. We are fixing the issue with the highest priority.
Report: "Paazl Data Dashboards : Shipment tracking not updating"
Last updateThis incident has been resolved.
Our systems show that the shipment tracking dashboards are currently not updating properly. This causes the information shown in those dashboards to not be current. We have identified the cause of this issue and working on a fix.
Report: "DPD Outage : webservice timeout"
Last updateThis incident has been resolved.
DPD responded that everything should be functional again. We will monitor the situation.
We are noticing slow response times again, we have reached out to DPD for more information
The response times have recovered to regular times. We will be monitoring the performance.
We have received note that DPD is currently experiencing an outage. We have notified the relevant parties. We will update our status updates as soon as we receive them.
Report: "PostNL : API Slow response times"
Last updateThis incident has been resolved.
Our monitoring shows that postNL API response times have improved. Whilst everything should be operational again, we continue to monitor the situation.
We are noticing an increasing in the response times of the PostNL API. They have been notified and are working on it. We will post an update as soon as we have one.
Report: "DPD : Tracking site seems unavailable"
Last updateThis incident has been resolved.
The tracking site should be functional again. We will be monitoring the performance.
We notice that the DPD track and trace site does not load and can give a HTTP 504 error. We have reached out to DPD for more information.
Report: "DPD Meta API unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Due to an outage at the DPD web service, DPD META API is currently unavailable. We are reaching out to DPD for further support. As soon as the DPD web service is back up and running, we will update this alert.
Report: "Cycloon: slow server response"
Last updateThis incident has been resolved.
Cycloon has implemented a fix for their production environment. We are monitoring the performance.
We have identified an issue where the Cycloon shipping options slow down the shipping option responses. We are reaching out to the carrier for a fix.
Report: "Correos Locator service impacted"
Last updateThis incident has been resolved.
We are currently investigating an issue in regards to the Correos Locator service. You may experience slow responses for Correos pick-up point requests, we advise to turn off Correos pick-up points until we confirm that performance is back at the correct level. We will post an update within 1 hour.
Report: "DPD Meta API unavailable"
Last updateThis incident has been resolved.
Due to a DPD web service outage, they are currently unavailable. We have reached out to DPD for further support. When the DPD web service is online again, we will update it here.
Report: "Shipment tracking dashboard under investigation"
Last updateThis incident has been resolved.
We have identified an updating issue in regards to the shipment tracking dashboard. We are currently investigating this matter, and will most likely be back up and running on 05/05/2023.
Report: "Out of Office 27th of April"
Last updateThis incident has been resolved.
Due to a public holiday our office will be closed on the 27th of April.
Report: "DPD Meta API unavailable"
Last updateThere have been no more errors since 15:00. The outage is resolved.
Our monitoring shows a decrease in errors, the labels should be available again. We will be monitoring the integration.
Due to an outage at the DPD web service they are currently unavailable. We are reaching out to DPD for further support. As soon as the DPD web service is online again, we will update it here.
Report: "DPD Meta API unavailable"
Last updateThis incident has been resolved.
We see in our monitoring that the errors have decreased. DPD should be up and running since 15:30
Due to an outage at the DPD web service they are currently unavailable. We are reaching out to DPD for further support. As soon as the DPD web service is online again, we will update it here.
Report: "Cycloon reported issues"
Last updateThis incident has been resolved.
We are still in contact with Cycloon regarding this issue. We will get back to you as soon as possible.
Cycloon reported issues on their side on and off since last night. You may experience not being able to print labels. We have asked them to keep us up to date. We advice you to also reach out to your contact at Cycloon and report issues if you experience these.
Report: "Production environment: update"
Last updateThis incident has been resolved.
The release has been completed and all services should be fully operational again. We are monitoring the situation.
Due to a production release issue we are experiencing issues with the Webservice and Web application. We have identified the issue and are working on fixing it.
Report: "Paazl Web Application: slow response"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are noticing that the response of our Web Application is slow. We are looking into the issue and will respond as soon as possible.
Report: "DPD: Read timed out errors"
Last updateThe issue has been resolved and no more time outs are being shown. The connection is fully operational again.
Our monitoring is showing no more time outs since 09:05. Therefore, you should be able to get labels again. We will be monitoring the DPD service for today.
We are currently investigating an issue regarding DPD giving read time outs when requesting a label. You will notice that you are not get a label for DPD shipments. We will reach out to DPD for help.
Report: "Paazl website down: use ship.paazl.com to log in"
Last updateOur website is up and running again. Our apologies for the inconvenience.
We are looking into the issue of our website being down. If you'd like to log into your Paazl environment, please use the link below: <a href="https://ship.paazl.com" target="_blank">https://ship.paazl.com</a>
Report: "Bpost: label timeouts"
Last updateThis incident has been resolved.
Bpost identified issues on their side and are currently on top of resolving this as soon as possible. As soon as we have another update we will post it.
We are currently investigating an issue where the Bpost labels are giving timeouts. You may experience issues when attempting to print labels as these will be temporarily unavailable. If you retry the label generation later on it may work again as the timeout has then been resolved. We are investigating the root cause of this issue and have prompted Bpost to the same on their side. We thank you for your patience.
The Bpost webservice is not responding. Because of this outage, you may experience issues when attempting to print shipping labels for this carrier. We've contacted Bpost and are awaiting a status update. As soon as we receive an update, we will post it on this page. We thank you for your patience.
Report: "DPD : Slow API response"
Last updateDPD has contacted us that the connection has improved and our monitoring shows that timeouts have decreased.
Our monitoring is showing a increased response times for the DPD Meta API. Therefore, you may notice 'read timed out' errors when generating DPD labels.
Report: "DPD: no shipping labels"
Last updateThis incident has been resolved.
The outage appeared between 09:15 and 09:30. All services are back online. We will monitor the current status.
We received reports of DPD labels not being printed. We have contacted DPD for a status report.
Report: "Paazl Web services: slow response"
Last updateThis incident has been resolved.
We've identified the cause of our degraded performance. We'll be monitoring the situation over the weekend.
We've performed a restart of our servers and our services are fully operational again. We will be monitoring the situation.
We are noticing that the response of our Web Services is slow. We will perform a fix as soon as possible.
Report: "DHL Deutschland: Packstations offline"
Last updateThis incident has been resolved.
We have received reports about DHL Packstations not being available, caused by a Webservice Timeout. We have contacted DHL for information. As soon as we receive an update, we will post it on this page.