Is TrekkSoft Down Right Now? Check if there is a current outage ongoing.

On June 11th, between 8:30 PM and 9:20 PM, our company experienced a targeted attack on several subdomains. During this time, key applications—including Admin Desk, POS Desk, Booking Widget, and Website Builder—were partially or fully unavailable. As a result, bookings were not being processed successfully. After an analysis by the engineering team, patterns for the attack as well as the source was identified and the IP address successfully blocked. Bookings originating from OTAs were subsequently reprocessed and delivered later that evening. As improvement steps, we plan to change our proxies used for subdomains and move them to a web application firewall. We apologize for any inconvenience this may have caused.

**Incident Date**: April 16th 2025 **Incident Duration**: Approximately 4 hours **Affected Services**: TrekkSoft Backoffice, TrekkSoft Website Builder ‌ **Incident Description:** On April 16th, around 18:00 CEST, our system was hit by an attack that caused unusually high network traffic. This led to about 4 hours of downtime, during which the system was unusable for those using a custom domain. **Impact**: The outage of approximately 4h affected the Backoffice and Website Builder sections of the system. **Resolution:** Once we identified the source of the attack, we blocked it to stop the issue. **Preventive Measures:** We are now working on strengthening our protection and improving how quickly we can detect and respond to similar problems in the future.

The incident has been resolved, and all TrekkSoft functionalities are operating as expected since Wednesday, 22:16pm CEST. Our investigation revealed that an attack happened affecting custom domains. We aim to publish a detailed post-mortem later today. We sincerely apologize once again for any inconvenience caused.

Custom domains are now accessible again. The root cause of the issue was an attack, which was now successfully being blocked. We will keep monitoring the situation.

We are currently experiencing issues with custom domains not being accessible. Therefore the option to book might be limited. Our developers are already investigating the root cause of the issue. We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve the issue. If you have any questions or concerns, please don't hesitate to contact us. We will make sure to keep you updated on the progress of this issue.

We're happy to inform you that the incident has been resolved. Payyo's platform is now stable, and customers can successfully proceed with payments again. We sincerely apologize for any inconvenience this may have caused.

We want to inform you about an ongoing issue with the Payyo payment gateway. Currently, multiple errors are occurring when customers attempt to make payments on our hosted payment page (HPP), which may prevent transactions from being completed. We are in close contact with Payyo and will keep you updated on the situation. You can also check for real-time updates at http://status.payyo.ch/. We apologize for any inconvenience and appreciate your patience as we work to resolve this. If you have any questions or concerns, please don’t hesitate to reach out.

**Incident Date**: December 6th 2024 **Incident Duration**: Approximately 4 hours \(intermittently\) **Affected Services**: TrekkSoft Backoffice, POS Desk, TrekkSoft Mobile App \(mPOS\) ‌ **Incident Description:** On Friday Dec 6th at 1:44 pm CET the system experienced a slowdown due to a combination of slow schedule queue messages and resource-intensive reporting queries. **Impact**: The slowdown affected TrekkSoft Back office, POS Desk and TrekkSoft mobile App \(mPOS\) which intermittently prevented bookings from being made. **Resolution:** Slow schedule messages were moved to a temporary queue to stabilize the system  **Preventive Measures:** To prevent further disruptions, we scaled up the system capacity to handle the increased load ensuring smoother operations over the weekend. The system has since stabilized, and further optimizations are being planned to mitigate future issues.

The incident has been resolved, and all TrekkSoft functionalities are operating as expected since Friday 7:38pm CET. Our investigation revealed that the system slowdown was caused by an increase in CPU usage on our database cluster. By scaling up the database cluster, we were able to stabilize the system. However, we will explore measures to prevent similar issues in the future. A detailed postmortem of the incident will be shared in the coming days. We sincerely apologize once again for any inconvenience caused.

A fix has been implemented and we are monitoring the results.

After implementing a fix, affected parts became functional again, unfortunately, we see a downgraded performance again preventing bookings from being placed. Our Engineering team keeps on investigating. The affected parts of the system including the Booking Widget, PoS desk, and the mPoS mobile app are currently inaccessible to take bookings. While channel manager bookings are still being taken via OTAs, they might not be processed on our system at the moment. Once again we apologize for the issues this might cause and we will keep you posted.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are currently experiencing an issue that is preventing bookings from being placed. The affected parts of the system included the Booking Widget, PoS desk, and the mPoS mobile app, which is currently inaccessible. Users who host their site via the Trekksoft Website Builder may not be affected. While channel manager bookings are still being taken via OTAs, they might not be processed on our system at the moment. Our developers are already investigating the root cause of the issue. We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve the issue. If you have any questions or concerns, please don't hesitate to contact us. We will make sure to keep you updated on the progress of this issue. We will keep you updated on our status page for any further information. If you haven’t subscribed to it yet, we encourage you to do it.

**Incident Date**: December 2nd 2024 **Incident Duration**: Approximately 5 hours \(intermittently\) **Affected Services**: TrekkSoft Backoffice, POS Desk, TrekkSoft Mobile App \(mPOS\) **Incident Description**: On Monday, Dec 2nd, at 12:13 PM CET, system slowness was reported, leading to an inability to process bookings. Earlier that morning, an alert indicated web response times exceeding 1.0 seconds for over 10 minutes. **Impact**: Investigation revealed it was caused by resource-heavy queries, making the system unresponsive. **Resolution**: This issue coincided with increased demand for year-end reports, likely contributing to the slowdown. Queues for reporting and scheduling were overloaded, with pending messages being gradually consumed after being moved to temporary queues. **Preventive Measures:** Further monitoring and optimization are ongoing to prevent similar issues.

The incident has been resolved, and all TrekkSoft functionalities are operating as expected. Our investigation revealed that the system slowdown, which caused error messages during booking attempts and prevented some bookings from being processed, was triggered by an unexpected increase in system requests (e.g. a high volume of reporting requests). We have implemented a fix that appears to have resolved the issue and are actively exploring measures to prevent similar issues in the future. A detailed postmortem of the incident will be shared in the coming days. We sincerely apologize once again for any inconvenience caused.

We are continuing to monitor for any further issues.

We have identified the root cause, applied a fix, and are currently monitoring the results.

We are continuing to investigate this issue.

We are currently experiencing a system delay that is causing a general slowdown of the system, this might result in error messages being shown and also might prevent some bookings from being placed. The affected areas include the booking widget, PoS desk and the mPoS mobile app. Our developers are actively investigating the root cause of this issue. We sincerely apologize for any inconvenience this may have caused and appreciate your patience as we work toward a resolution. If you have any questions or concerns, please feel free to reach out to us. We will keep you informed of any updates through our https://status.trekksoft.com page. If you haven't subscribed yet, we encourage you to do so to stay updated.

Saferpay informed that this incident has been resolved. Please be informed that due to a technical issue, some of your customers were unable to make payments online via Six Payment Services Today, (05/11/2024) between 20:12 and 21:42 CET. From 21:42 CET onwards, customers are able to make payments as expected. We apologize for any inconvenience caused.

We would like to inform you that there is an ongoing issue with Saferpay, one of the payment providers used by Payyo. This is currently preventing many of our customers from processing certain payments including credit card, Apple Pay, and Google Pay transactions. This does not impact our merchants processing payments via Stripe or Paypal. We are in close contact with Payyo and will continue to update our customers on the situation. You can check any update directly at http://status.payyo.ch/ We apologize for any inconvenience and appreciate your patience as we address this matter. If you have questions or concerns, feel free to reach out. If you have any questions or concerns, please don't hesitate to contact us. We will make sure to keep you updated on the progress of this issue.

Saferpay informed that this incident has been resolved. Please be informed that due to a technical issue, some of your customers were unable to make payments online via Six Payment Services Today, (05/11/2024) between 05:51 and 10:07 CET. From 10:07 CET onwards, customers are able to make payments as expected.

We would like to inform you that there is an incident with Saferpay, one of the payment providers used by Payyo. As a result many of our customers are currently unable to process some payments with credit card, apple/ google pay, etc. We are in close contact with Payyo and will continue to update our customers on the situation. Note: This does not impact our merchants processing payments via Stripe or Paypal. We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve the issue. If you have any questions or concerns, please don't hesitate to contact us. We will make sure to keep you updated on the progress of this issue.

We would like to inform you that there is an incident with Saferpay, one of the payment providers used by Payyo. As a result many of our customers are currently unable to process some payments with credit card, apple/ google pay, etc. We are in close contact with Payyo and will continue to update our customers on the situation. Note: This does not impact our merchants processing payments via Stripe or Paypal. We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve the issue. If you have any questions or concerns, please don't hesitate to contact us. We will make sure to keep you updated on the progress of this issue.

**Incident Date**: October 9th 2024 **Incident Duration**: Approximately 1 hour **Affected Services**: TrekkSoft Application, POS Desk, Backend Mobile Applications **Incident Description**: At approximately 03:30 PM CEST on October 9th, 2024, a prepared cyber attack was performed against our system. **Impact**: We experienced an attack that targeted parts of our system, causing our TrekkSoft Application, POS Desk, Backend Mobile Applications,  be unavailable for about an hour. This happened because the attack was sending fake information to a specific parts of our system. **Resolution**: After identifying where the attack was coming from, we blocked it and restarted parts of our system that had stopped responding, which allowed us to get everything back up and running. **Preventive Measures:** We are taking further measurements to prevent similar incidents in the future.

We registered a cyber attack towards our systems, that resulted in a brief downtime. We identified and quickly reacted to the issue by blocking the attackers. The incident has been resolved, and all Trekksoft functionalities are now operating normally. We will provide a postmortem of the incident in the coming days. We apologize once again for any inconvenience this may have caused.

We are continuing to monitor for any further issues.

We are currently monitoring the issue.

We are currently experiencing global issues in TrekkSoft's production functionalities (e.g. Backoffice, POS Desk). Our developers are already investigating on finding the root cause of the issue. We will keep you updated and apologize for the inconvenience caused.

**Incident Date**: October 3rd 2024 **Incident Duration**: Approximately 20 minutes **Affected Services**: TrekkSoft API, TrekkSoft Application, POS Desk **Incident Description**: At approximately 12:15 PM CET on October 3rd, 2024, the system went down. **Impact**: The redis node used for session storage from API was rebooted and came back approximately 20 minutes later.  The node went out of service outside the maintenance windows. We opened a support ticket with AWS to understand why it was rebooted.  **Resolution**: The incident was resolved due to the rebooted redis node \(used for session storage from API\). **Learnings:** API uses redis-core-production for session storage. This is a one node instance. **Preventive Measures** * Review AWS Fault Tolerance reference:[ Mitigating Failures - Amazon ElastiCache \(Redis OSS\)](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/FaultTolerance.html)  * Ensure proper configuration is used for the redis-core-production instance * Review and improve our API session handling logic, or even consider other types of persistence for session storage

The incident has been resolved and all TrekkSoft functionalities are operating as expected. We have determined that the issue originated from one of our infrastructure services unexpectedly stopping and restarting, triggered a cascading effect, leading to a brief system outage. The responsibility for maintaining this service lies with our cloud services provider, AWS, and we have reached out to them for further clarification. We will explore measures to mitigate this type of issue on our end and will provide a postmortem of the incident in the coming days. We apologize once again for any inconvenience this may have caused.

We are continuing to investigate this issue.

TrekkSoft experienced a downtime during the last hour, with the outage lasting approximately 10 minutes. Since then, all systems have come back online. We are actively investigating to determine whether this was due to a potential cyber attack, while also reviewing our infrastructure for other possible causes. We will provide further updates as soon as we have more information. We apologize for this inconvenience.

**Incident Date:** Jun 11, 2024 **Incident Duration**: Approx 4 hours **Affected services:** TrekkSoft API, TrekkSoft Application, POS Desk, Backend Mobile Applications, Channel Manager **Incident Description:** June 11 at around 4.45AM our system alert triggered a high error rate, which we identified as an attacking IP address.  **Impact:** The incident critically affected the functionality of most of our services \(which made it impossible to finalize any bookings\). The disruption caused a downtime of approximately 4 hours, being resolved at 9:30AM. **Resolution:** The incident was resolved by identifying  and blocking the attacking IP address.  We also identified expired SSL certificates that we migrated to a platform that secures the organization’s network. **Preventive Measures and Recommendations:** * Migrate custom domains with expired SSL certificates to security platform * WAF solution: Extension of our Web Application Firewall Solution to all our domains within TrekkSoft and of TrekkSoft merchants Rest assured that at TrekkSoft, maintaining a secure and reliable system is our highest priority. Over the upcoming days, we will implement the necessary measures to enhance our security. We apologize for any inconvenience this caused.

The incident has been resolved, and all Trekksoft functionalities are now operating normally. We will provide a postmortem of the incident in the coming days. We apologize once again for any inconvenience this may have caused.

We have been able to isolate and produce a solution for the issue. Bookings appear to be working as expected on all channels now. We continue to monitor the system to make sure all systems are performing well.

We are currently experiencing an issue that is preventing bookings from being placed. The affected parts of the system include the Booking Widget, PoS desk, and the mPoS mobile app, which is currently inaccessible. Users who host their site via the Trekksoft Website Builder may also be unable to access their site. While channel manager bookings are still being taken via OTAs, they are not being processed on our system at the moment. We have identified the point of failure in our system and are actively investigating the root cause to find the quickest solution to restore booking functionality We will keep you updated and apologize for the inconvenience caused.

This incident has been resolved and all the Trekksoft functionalities are again operational as normal. We want to apologize for any inconvenience this might have caused you.

May the 3rd we have detected some patterns of a DDoS attack, causing slow response time. This means that there might be a delay affecting the system and the bookings. We have blocked the identified IP addresses and are currently monitoring our system load.

**Incident Date**: March 10, 2024 **Incident Duration**: March 10th evening till March 11th early morning **Affected Services**: TrekkSoft backoffice, POS Desk, Booking Widget, Website Builder, mobile App **Incident Description**: At approximately 6:18 PM CET on March, 10, 2024, a SQL injection attack was conducted which affected our database. A SQL injection is a technique used to attack applications by inserting malicious statements in order to hinder normal workflows from happening. As a result, services dependent on this database were impacted and therefore not accessible.  **Impact**: The incident rendered it impossible to perform operations on the database, leading to a halt in functionality for the majority of our services \(not being able to log in or take bookings\) . This disruption caused three downtime periods of 45 minutes at 9.30PM,  and a 30 minutes disruption on March 11th around 5.30AM **Resolution**: The incident was resolved by blocking the IP address from where the attack was conducted and restoring the database service. **Preventive Measures and Recommendations:** 1. **WAF Solution:** Extension of our Web Application Firewall Solution to all our domains within TrekkSoft and of TrekkSoft merchants. The ones that use custom domains within TrekkSoft will receive an email over the next weeks with instructions to follow.    2. Reviewing the code base to make sure we find any possible vulnerabilities. Rest assured, that at TrekkSoft, a secure and reliable system has the highest priority. Unfortunately, these types of attacks became very common in the industry. Our system sent out alerts upon being attacked  and, our engineering team was right on spot and started to resolve the issue. Over the next week, we will put the mentioned measures in place.

The incident has been resolved and all the Trekksoft functionalities are again operational as normal. We will provide a postmortem of the incident in the following days. Once again we want to apologize for any inconvenience this might have caused you.

March the 10th around 9pm CET we registered a cyber attack towards our systems, that resulted in the brief disruption with bookings. We identified and reacted to the issue quickly, blocking the attackers using our Web Application Firewall. Currently systems are operational and bookings appear to be working as usual. We are continuing to monitor the situation and will follow up accordingly.

**Incident Date**: February 15, 2024 **Incident Duration**: Approximately 1 hour **Affected Services**: All services **Incident Description**: At approximately 11:30 AM CET on February 15, 2024, a bug fix caused an unexpected slowdown of our database. As a result, all services dependent on this database were impacted. **Impact**: The incident rendered it impossible to perform operations on the database, leading to a halt in functionality for all of our services. This disruption lasted for around one hour until our developers were able to mitigate the issue. **Resolution**: The incident was resolved by rolling back the release. **Preventive Measures and Recommendations:** 1. **Testing in our test environment:** Improve options to test database impacts on our test environment, incl. smoke/regression/load tests.  2. **Speed up our roll-back process** Despite our strict release policy which includes thorough tests on our test environment, for this specific bug fix, our test environment exceptionally didn't cover all implications. By implementing these measures, we can proactively address and mitigate similar incidents in the future, ensuring the continued reliability of our services.

The incident has been resolved and all the Trekksoft functionalities are again operational as normal. We will provide a postmortem of the incident in the following days. Once again we want to apologize for any inconvenience this might have caused you.

The issue has been identified and a fix has been implemented. All system components are operational. We'll keep monitoring it.

We are currently experiencing global issues in several Trekksoft production functionalities (a.o. Backoffice, POS Desk). Our developers are already investigating on finding the root cause of the issue. We will keep you updated and apologize for the inconvenience caused.

**Incident Date**: October 22, 2023 **Incident Duration**: Approximately 1 hour **Affected Services**: All services **Incident Description**: At approximately 9:30 AM CET on October 22, 2023, an incident occurred on our database due to a disk space issue. The disk associated with the DB instance reached full capacity, causing a disruption in our database operations. As a result, all services dependent on this database were impacted. **Impact**: The incident rendered it impossible to perform write operations on the database, leading to a halt in functionality for all of our services. This disruption lasted for around one hour until our developers were able to mitigate the issue. **Resolution**: The incident was resolved by leveraging auto-scaling capabilities and expanding the disk size, alleviating the disk full issue and restoring normal operations. **Preventive Measures and Recommendations:** 1. **Alert Monitoring**: Monitoring to prevent disk size reaching full capacity. 2. **Auto-scaling Configuration**: Enable auto-scaling function which helps in automatically adjusting resource capacity to accommodate increased demand. By implementing these measures, we can proactively address and mitigate similar incidents in the future, ensuring the continued reliability of our services.

The incident has been resolved and all the Trekksoft functionalities are again operational as normal. We will provide a postmortem of the incident in the following days. Once again we want to apologize for any inconvenience this might have caused you.

Trekksoft functionalities are again operational for the vast majority of our users, however, we keep working on it and monitoring the situation.

We are currently experiencing global issues in several Trekksoft production functionalities (a.o. Backoffice, POS Desk). Our developers are already investigating on finding the root cause of the issue. We will keep you updated and apologize for the inconvenience caused.

We are pleased to announce that the incident with our payment gateway service provider has been successfully resolved. Payment processing is now again fully operational.

We would like to inform you that there is an ongoing incident with our payment gateway service provider. As a result many of our merchants are currently unable to process payments. As this issue is not within our system we have no ability to remedy this. Nonetheless we are in close contact with the payment services and will continue to update our customers on the situation. Note: This does not impact our merchants processing payments via Stripe.

We are happy to announce that the issue with our payment gateway (Payyo) has now been fully resolved. We see that all the bookings processed trough our payment gateway are going trough without further issues. We want to apologise for any inconvenience this might have caused you.

We are seeing some limited disruption on some attempt to make purchases resulting in an error. Most bookings are working as expected and we are investigating the cause of the intermittent failed bookings.

**The root issue:** On the evening of January 2nd 2023 scheduled maintenance work was done on some of our domain names. As a result, suppliers and partners connected to the ExperienceBank channel manager, lost their connection to OTAs that were using the old [trekkconnect.com](http://trekkconnect.com) domain name. These include Viator, GYG and Expedia among some other smaller OTAs. Sales on those OTAs were no longer reflecting on the booking software \(including TrekkSoft\), and availabilities were not being updated to the OTAs correctly. In addition our URL Shortener solution for SMS messages stopped working, [getstarted.trecksoft.com](http://getstarted.trecksoft.com) redirect to signup page stopped working. Some long standing merchants started receiving booking notifications with links that are not working anymore. **Identification:** Our automated observability system detected and alerted on errors coming in. Our engineers quickly escalated the situation to critical. **Steps taken to solve the issue:** DNS addresses for [trekkconnect.com](http://trekkconnect.com) were reconfigured. These changes can often take up to 48 hours to take effect. In addition, the affected OTAs were contacted as some reconfiguration work was required on their side so that their connection to the ExperienceBank channel manager was reestablished. **Resolution:** While in most cases bookings began to reflect on the booking systems within a few hours, some OTAs took an additional 12 hours to get all suppliers reconfigured and running. By early morning on January 4th all connections to OTAs were back and bookings were being reflected normally. We worked with individual partners to reactivate those connections where necessary. The critical situation was deemed resolved and communicated to all stakeholders. Some follow up housekeeping work came in the form of fixes for minor issues on the WEBAPP related to legacy domain deprecation were released on 5th of January in the evening.

The incident has been resolved as all OTAs are again connected to our channel manager. Bookings from OTAs are coming trough and are reflected properly to the baskets in TrekkSoft system. If you observe any more issues with the activities on the OTAs please check their mapping configuration and if further help is needed, feel free to reach out to support@experiencebank.travel. We will provide a postmortem of the incident in the following days. Once again we want to apologise for any inconvenience this might have caused you.

The major OTAs (e.g. Expedia, GYG, Viator) have been notified about the changes made with domain names and are now making the necessary changes on their end to establish again the connection with our channel manager (ExperienceBank). Changes are rolling out gradually and will be applied to all our customers one by one. Some suppliers are already back on line with their respective OTAs while others may still need to wait while the OTAs take action. Please note that in order for the fix to be applied to all the customers it can take up to 72h. We apologize for any inconvenience this may have caused and appreciate your patience until this is resolved. In the meantime, we recommend that you keep checking if the connection has been reestablished already. We will make sure to keep updating you on further progress.

We are currently facing a partial outage in our channel manager ExperienceBank which is impacting bookings coming from the following OTAs: Expedia, GYG, Viator, Musement, Tiqets, Project Expedition and Grayline. In some cases bookings from these sources are not reflected properly in baskets in the TrekkSoft system. The issue arose after scheduled maintenance to our domain names, namely “trekkconnect.com” redirection which is used by ExperienceBank to handle bookings coming from the OTAs. We want to assure you that we are working diligently to resolve this situation. We have already identified the issue and restored the domain redirect and settings. However, it may take some time for these changes to take effect. We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve the issue. If you have any questions or concerns, please don't hesitate to contact us. We will make sure to keep you updated on the progress of this issue.

**What was the cause of the issue?** TrekkSoft was targeted by a DDOS attack, which affected the availability of the system to our users. DDOS is a type of cybercrime where an unknown attacker is flooding the servers with high internet traffic, that is preventing users from accessing our system. You can read more about what DDOS attack is and how it works [here](https://en.wikipedia.org/wiki/Denial-of-service_attack). This particular DDOS attack generated an abnormal spike \(1000x higher\) in HTTP requests, which overloaded our servers. **Impact** The impact was global and most of the Trekksoft services were affected for all our customers. **What did we do?** As soon as the attack was initiated around 8am CEST, our developers started working on identifying the root cause of the issue. Once the cause was identified as a DDOS attack, we started working on preventing it by blocking the source of abnormal activity, as well as improving the configuration of the firewall solution. The mentioned solution was fully configured at roughly 1pm CEST, when we started seeing improvements and our system was again accessible as well as bookings started coming in. Performance of features was still impacted until all the issues from the attack were resolved at around 7pm CEST. After that all the services were again performing as expected, and we continued to monitor the situation closely until the next day. We subsequently confirmed that the incident was resolved. **Learnings** DDOS attacks are well thought through cyber crimes, using new methods and technologies, which renders them unpredictable and challenging to neutralize. In the aftermath of the recent attack, we are further improving our firewall protection, as well as investigating and expanding into new 3rd party solutions designed for tackling external security threats. These extra layers of protection will further add to the resilience of our system in similar situations.

The incident has been resolved and all the Trekksoft functionalities are again operational as normal. We will provide a postmortem of the incident in the following days. Once again we want to apologize for any inconvenience this might have caused you.

Trekksoft functionalities had been stable since the last update and we are observing that the speed of services has stabilised as well. We will continue to monitor the situation closely.

Trekksoft functionalities are again operational for the vast majority of our users, however, you might be still experiencing some issues such as reduced speed. We keep working on this and monitoring the situation.

TrekkSoft is currently being targeted by a DDOS attack, which affects the availability of the system to users. DDOS is a type of cybercrime where an unknown attacker is flooding the servers with high internet traffic, that is preventing users from accessing our system. Please note that the DDOS attack affects the <strong>availability</strong> of the system to users. The attack does <strong>not</strong> compromise user data held in the system, i.e. there is no breach of private data. As we implement solutions, there are time frames where the system is operational. We recommend that, if you are able to access your account at any point, you download the data which you need to run your daily operations. We continue to work hard on addressing this malicious activity, and resolve it as soon as possible. We will keep you updated and apologize for the inconvenience caused.

Our developers identified the cause of the issue and are currently working on it's resolution. We will keep you updated and apologize for the inconvenience caused.

We are continuing to investigate this issue.

We are currently experiencing global issues in several Trekksoft production functionalities (a.o. Backoffice, POS Desk) Our developers are already investigating on finding the root cause of the issue. We will keep you updated and apologize for the inconvenience caused.

**What happened?** Our developers were working on booking widget improvements and along with that they released a new version which included infrastructure changes. Those changes unexpectedly impacted the main functionalities and prevented end customers from booking the activities. **What we did?** As soon as this issue has been detected, our developers started working on it’s resolution first by rolling back to the previous version and later on providing a new healthy release with re-caching solution. Issue was fixed 2h and 37 min after the initial detection. **Impact** Booking widget functionalities were impacted for some merchants and end customers: 1. merchant impact: All the accounts without “en\_GB“ language as default weren’t able to receive bookings. 2. end customer impact: Customers using safari browser / apple devices were unable to proceed with bookings on booking widget. **Learnings** In order to prevent similar issues in the future we introduced additional tests which are executed on different browsers and systems each time new booking widget version is released.

This incident has been resolved and <strong>booking widget is working as expected</strong> without issues. Our developers were able to clear the cache as well, which means that <strong>manual clearing of cache is no longer needed.</strong> Postmortem on the issue will be provided in the following days. Once again, we apologise for any inconvenience this might have caused you and thank you for your patience.

A fix has been implemented and the booking widget is working now as intended. We will keep monitoring the behaviour to make sure the issue is completely resolved and it doesn't appear again. NOTE If you are still experiencing issue with availabilities not showing, please clear the cache.

An issue has been identified with our booking widget which is not working properly for end customers trying to make bookings in Safari (either mobile or desktop) and Chrome (only mobile) web browser. Our developers already identified the root cause of the issue and are currently working on it's resolution. We will keep you updated and we apologize for any inconveniences this might have caused.

The incident has been resolved and TrekkPay payment gateway is fully functional again. We are continuing to investigate what exactly caused this issue and will provide postmortem once the investigation is finished. We apologize for any inconveniences this might have caused you.

Our developers identified the root cause of the issue and already provided a fix for it. TrekkPay payment gateway is fully functional again allowing payments and refunds to be processed. Our developers are continuing to monitor the issue closely to ensure there are no further performance issues.

We are currently experiencing an issue in TrekkPay payment gateway which is preventing some operations related to payments and refunds. Our developers are already investigating on finding the root cause of the issue. We will keep you updated and apologize for the inconvenience caused.

**What happened?** One of external services, our system is using, was experiencing issues. This affected our web servers which were overloaded and finally crashed as the result of it. **What we did?** When our developers identified the issue, they fixed it by restarting failed services and servers for which it took some time to get back to the normal operating state. **Impact** During the estimated 2h time frame \(12.30 to 14.30 CEST\) of the incident no bookings have been able to be processed as well as our customers had issues with accessing our web and mobile application. **Learnings** In order to prevent similar incidents in the future we added additional monitoring system to the before mentioned external service as well as servers that the service is used on. With this in place, we will have a better overview in case similar issue happens again and this will allow us to act faster and minimize the impact this has on our customers. We apologize for any inconvenience this might have caused you.

The incident has been resolved and all the functionalities are performing as expected. We are continuing to investigate what encountered this unusual spike in network traffic and will provide postmortem once the investigation is finished.

Issues with bookings and payments have been addressed as well. Trekksoft functionalities are again performing as expected. Our developers are continuing to monitor the issue closely to ensure there are no further issues.

We are still facing some issues with taking bookings and payments. Our developers are continuously working towards finding a fix for this issue as well.

Our developers identified the root cause of the issue and already found a fix for it. Issue occurred due to unusual spike in network traffic which affected performance of our servers. Trekksoft functionalities are again performing normally. Our developers are continuing to monitor the issue closely to ensure there are no further performance issues.

We are currently experiencing global issues in several Trekksoft production functionalities. Our developers are already investigating on finding the root cause of the issue. We will keep you updated and apologize for the inconvenience caused.

**Root cause** New release containing database changes was being pushed to production which locked access to some database tables. Due to high volume of requests to access those tables, and them being locked, requests for access were piling up in the queue which overloaded our servers. **What happened** For a certain amount of time \(estimated at 45 min\) users were not able to access their TrekkSoft sites due to our production servers being down. **What we did** When our developers identified the new release is impacting the performance of our services and overloading the servers they had to restart those and return them to their normal functioning state with a rollback. **The consequences** During the 45 min time window of the incident no bookings have been able to be processed. **Learnings** In order to prevent similar incidents in the future we updated our internal guidelines on new releases to avoid releasing critical changes during peek hours when high availability is needed. We apologize for any inconvenience this might have caused you.

Issue with the TrekkSoft servers has been resolved. All the affected services are back working as expected. Our developers are continuing to monitor the issue closely to ensure there are no further performance issues.

We are currently experiencing issues with Trekksoft servers. This is affecting most of our product functionalities. Our developers are investigating with highest priority to identify the root cause of the issue and it’s resolution. We will keep you updated and apologize for the inconvenience caused.

Issue with the TrekkSoft database has been resolved, all the affected services are back working as expected. Our developers are continuing to monitor the issue closely to ensure there are no further performance issues.

We are currently experiencing issues with Trekksoft database. This is affecting POS Desk, TrekkSoft API and Backend Mobile Applications. Our developers have already located the root cause of the issue and are working hard towards a resolution. We will keep you updated and apologize for the inconvenience caused.

The issue has now been resolved. For any questions or concerns, please contact Trekksoft Support Team.

The performance has now returned to normal. Our developers will continue to monitor the situation and we will update further once we can confirm the issue is resolved. If any issues continue to occur, please contact the Trekksoft Support Team.

We've had reports of further issues occurring which is now causing further performance issues. We are also aware of cases where websites cannot be accessed. Our developers are working to resolve these issues and we will continue to keep you updated. We again can only apologise for the inconvenience caused but rest assured we are doing everything we can to restore normal operations.

Trekksoft websites and the back end are now returning to their normal performance and are working as expected. Our developers are continuing to monitor the issue closely to ensure there are no further performance issues. Once we can confirm the issue is resolved, we will provide a further status update.

We are currently experiencing issues with Trekksoft websites loading slower than usual. This is affecting the back end system and POS Desk also. Our developers have already located the root cause of the issue and are working hard towards a resolution. We will keep you updated and apologize for the inconvenience caused.

Root cause: Memory shortage on one of the servers caused bookings from OTAs not to be created on the internal booking system. What happened: Bookings from OTAs were taken by the channel manager but were not translated into bookings on TrekkSoft. What we did: Our developers pinpointed the specific services that were affected and were not performing correctly. These services were restarted and this solved the issue. The consequences: Most bookings taken during the down time were created on TrekkSoft after the issue was resolved. Only bookings for activities that already took place, or have meanwhile run out of seats, were not created on TrekkSoft. Providers receive email notifications about these failed bookings and should check their email inboxes. Refunds for failed bookings through the OTAs should be made by providers through the OTAs. We apologize for the inconvenience.

Issue Resolved: The Experience/TrekkConnect channel manager is working again as expected. Bookings that were made during this partial outage were stored and are now being carried over to the booking system.

We are experiencing issues with bookings coming in through our channel manager: TrekkConnect / ExperienceBank. A proportion of these bookings are currently not being processed. The main affected channels are Get Your Guide and Viator. Our developers are looking into finding the root cause of this issue. We will keep you updated and apologize for the inconvenience.

## Post Mortem of the incident ## Summary: On the morning of February the 26th we migrated the TrekkSoft servers from our Cloudscale hosting provider in Zurich to the new Amazon Web Services in Ireland. After the migration was complete, usage of the system increased as the merchants began to take bookings and use the system. We spotted a major drop in performance. The root cause was one database host that was throttling under the amount of requests per second. This database slowdown caused a significant drop in performance to our applications \(Merchants landing pages - CMS, Backoffice, public and private API and mobile apps\), in some cases rendering them inoperable. ## What Happened 1. 6:45am - 8:29am **CET** - We completed the AWS migration. 2. We tested all the main cases and monitored all hosts and the preliminary results were satisfactory. 3. **9:00am CET** - Our applications began handling an increased amount of requests as the system came back online and usage of the system ramped up. 4. One of the main database hosts \(MySQL\) began struggling with the amount of requests. This affected the performance of our application, preventing normal functionality. ## Contributing Factors 1. Uncertainty regarding the performance of the new AWS infrastructure vs CloudScale. 2. We compared all hosts in CloudScale vs AWS to ensure the same hardware requirements. 3. The infrastructures are different. ## Steps Taken Phase 1: 1. Increase the size of the database in AWS to increase performance \(no downtime was required at this point\). 2. Contact AWS support to provide for more information about the resizing time. 3. The database resize was to take AWS too long to deploy, so we decided to apply another workaround, described below. Phase 2: 1. We put all the webapps in maintenance mode \(down time\). 2. We created a new, larger database \(downtime was required to avoid data loss\). 3. We extracted all data from one database to another, now using a migration system in AWS. 4. The new database created failed. 5. This required a new approach, described below. Phase 3 1. We created a new empty database \(again, downtime was required to avoid data loss\). 2. We proceeded with a manual dump of the data from the old database to the new one. The process took 4 hours and was successful. 3. **3:20PM CET** The new infrastructure was ready to be released at aprox. 4. We have been monitoring and tweaking the system over the last 24 hours to improve performance. ‌ ## Impact ## Low number of bookings from 7:00am to 4:30pm CET \(about 9 hours\). Some merchants were unable to process any bookings, while others still managed to take some. The impact here is financial loss to all parties. Benefits The objective behind the migration that caused the issue. 1. Overall long term increase in performance. 2. Up to date industry infrastructure. 3. More direct control over our infrastructure. 4. Infrastructure ready to apply autoscaling in case of a peak of request per second. ## Lessons learned 1. We will strategically time operation of this scale so that we have more time to react and avoid peak booking hours. 2. Triple-check hardware and settings specifications. 3. Bulletproof checklist. 4. Replicate the system and run stress tests. 5. Build our infrastructure with extra capacity and resources/have a larger infrastructure as a backup. We apologise deeply for this incident.

The issue is now resolved. Booking functionality has been resumed as well as access to all pages and accounts. We will continue to monitor the situation closely making sure that the system performs properly. Loading times might still be longer than you are used to while we continue to improve. We will be updating this status page accordingly.

As we continue to monitor and work to fix the issue, we have identified the source of the problem. The root cause lies within one of our service providers. We rely on this partner service in order to provide a solution and therefore it is difficult to give an exact time frame at this time. However we are doing everything in our ability to get our systems running as soon as possible. We will continue to update this page as more information becomes available. Apologies for the inconvenience caused.

As we continue to monitor the situation and take steps so provide a solution. We have identified multiple failed bookings. As a precaution we have temporarily disabled bookings in order to avoid failed attempts, until we can remedy the situation. Thanks for your patience.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

Trekksoft systems are currently experiencing performance issues, including difficulty accessing some pages and longer than usual loading times, timeouts and possibly other issues as well. The issues are a side effect of a scheduled maintenance operation to improve the overall reliability and speed of our services. We are monitoring the situation closely and taking steps to counteract the issue.

Summary: On the morning of February the 26th we migrated the TrekkSoft servers from our Cloudscale hosting provider in Zurich to the new Amazon Web Services in Ireland. After the migration was complete, usage of the system increased as the merchants began to take bookings and use the system. We spotted a major drop in performance. The root cause was one database host that was throttling under the amount of requests per second. This database slowdown caused a significant drop in performance to our applications (Merchants landing pages - CMS, Backoffice, public and private API and mobile apps), in some cases rendering them inoperable. What Happened 6:45am - 8:29am CET - We completed the AWS migration. We tested all the main cases and monitored all hosts and the preliminary results were satisfactory. 9:00am CET - Our applications began handling an increased amount of requests as the system came back online and usage of the system scaled up. One of the main database hosts (MySQL) began struggling with the amount of requests. This affected the performance of our application, preventing normal functionality. Contributing Factors Uncertainty regarding the performance of the new AWS infrastructure vs CloudScale. We compared all hosts in CloudScale vs AWS to ensure the same hardware requirements. The infrastructures are different. Steps Taken Phase 1: Increase the size of the database in AWS to increase performance (no downtime was required at this point). Contact AWS support to provide for more information about the resizing time. The database resize was to take AWS too long to deploy, so we decided to apply another workaround, described below. Phase 2: We put all the webapps in maintenance mode (down time). We created a new, larger database (downtime was required to avoid data loss). We extracted all data from one database to another, now using a migration system in AWS. The new database created failed. This required a new approach, described below. Phase 3 We created a new empty database (again, downtime was required to avoid data loss). We proceeded with a manual dump of the data from the old database to the new one. The process took 4 hours and was successful. 3:20PM CET The new infrastructure was ready to be released at aprox. We have been monitoring and tweaking the system over the last 24 hours to improve performance. Impact Low number of bookings from 7:00am to 4:30pm CET (about 9 hours). Some merchants were unable to process any bookings, while others still managed to take some. The impact here is financial loss to all parties. Benefits The objective behind the migration that caused the issue. Overall long term increase in performance. Up to date industry infrastructure. More direct control over our infrastructure. Infrastructure ready to apply autoscaling in case of a peak of request per second. Lessons learned We will strategically time operation of this scale so that we have more time to react and avoid peak booking hours. Triple-check hardware and settings specifications. Bulletproof checklist. Replicate the system and run stress tests. Build our infrastructure with extra capacity and resources/have a larger infrastructure as a backup. We apologize deeply for this incident.

All systems are working correctly and we are able to give service regularly again.

Our provider has advised they have addressed the issue which was causing a delay however, some issues could still occur. Our provider will continue to monitor the performance until the issue is fully resolved. We apologise for any inconvenience that this may cause.

Our provider for the software we use to answer support related question such as chat and email, is experiencing issues. At the moment we do not have an update on when the issue will be resolved. Customers reaching out to us for support should be prepared for a longer than usual wait.

This incident has now been resolved. We thank you kindly for your patience.

This incident has now been resolved. We thank you kindly for your patience.

We are currently experiencing an issue affecting our chat tool. This means that temporarily, messages you send to us via the chat tool might not work. We already informed our service partner Hubspot and they are working on solving this issue as quickly as possible. Further details you find here: https://status.hubspot.com/ In the meantime feel free to send us an email to support@trekksoft.com We are sorry for the inconvenience and will inform you as soon as this issue is fully solved.

This incident has now been resolved and all merchants should be able to take bookings normally. We thank you kindly for your patience.

As the fix we have implemented continues to work through our system, we are monitoring and updating as we go along. We sent out an email with instructions on how merchants can manually help fix the situation while wait wait for our automated script to do it's job. Some merchants should be seeing their booking system come back online already while other might still take a few hours to see correct functionality. We will change the status back to normal once the job is finished.

We are currently experiencing an issue affecting some but not all of our subscribers. Those affected will not be able to take bookings for a short time on certain activities that have been made inactive by the system. We have pinpointed the issue and have already deployed a correction that will reset the affected activities, making them available once again. As the script works through the system merchants will begin to see their activities become bookable again. Depending on the merchants this may take between a few minutes and hours. We are sorry for the inconvenience and assure you we are doing everything possible to remedy the situation quickly.

Dear users, Everything is stable and fully operational. Best regards, TrekkSoft Team

Dear users, Our hosting provider managed to stabilise everything for now, so everything is back to normal. Please keep in mind that further outages might happen in the short term. However, we will keep a close eye on the situation and are taking steps to avoid this situation hapenning again in the future. Thank you for your understanding, TrekkSoft Team

Our Hosting Provider is currently resolving the issue.

Dear users, After recaching all activities and availabilities in our database, we have been monitoring our performance for a while and everything is back to normal. We’ve learned a lot from this experience and we’re working on preventing these kind of issues in the future. Please contact support@trekksoft.com if you have any further questions. Best regards, TrekkSoft Team

We are continuing to monitor for any further issues.

Dear users, Most cases have been resolved already and we are working to get all remaining cases resolved too. Best regards, TrekkSoft Product team

Dear users, We are currently implementing the fix for the issue, as soon as this is complete we will let you know. Thank you for your understanding and once again apologies for the inconvenience, TrekkSoft Product Team

Dear users, We are experiencing some major issues with our booking widget. Our developers are already working on implementing a solution. We will keep you updated and apologise for the inconvenience, TrekkSoft Product Team

Dear users, The changes have been reverted and the widgets on the affected accounts are fully functional again. Once again, apologies for the inconvenience, The TrekkSoft Product Team

Dear users, For some accounts we are experiencing some issues with our booking widget and are already finishing reverting the changes that caused this problem. Apologies for the inconvenience, The TrekkSoft Product Team

Dear users, The changes have been rolled back and the booking calendar behaviour is back to normal. Once again, we apologise for the inconvenience caused. The TrekkSoft Product Team

Dear users, We are experiencing issues with the booking calendar for accounts that have an apostrophe character in their account name. We are currently rolling back the changes that have caused this. Our sincere apologies for the inconvenience caused. The TrekkSoft Product Team

Everything is stable now.

With further investigation it turns that one of the search engine's crawler went crazy and started to DDoS our system.

Everything is normalized. We'll continue to monitor the situation.

Hello, we have an ongoing password brute force attack on our servers. Because of that, performance of all systems is decreased.

The problem was solved and we back online.

Due to unexpected problem during the regular update maintenance time prolonged.

Our message queue is now low and back to normal.

Dear users, Our queues are decreasing steadily and within a few hours all the messages should go through, so everything will get back to normal. This pile-up was caused by the extremely large volume of messages going through our queues in a very short timeframe. Of course we are constantly monitoring this and working on mitigating the negative consequences. Everything is functioning, but with a long delay in things such as occupancy changes or booking deliveries. Once again, we apologise for any inconveniences and are still working on this. The TrekkSoft Product team

Dear users, We are experiencing some issues with our message queue, the current queue for messages is multiple times the normal and therefore there is an extreme delay with certain changes to get reflected across the various parts of the software. We are currently investigating what is causing it and how it might be resolved. Further updates will be posted here as soon as there are any. We sincerely apologies for the inconvenience. The TrekkSoft Product team

Dear Customers, Together with our service provider, we have been monitoring our server performance for the past few days and can confirm that everything is back to normal and 100% stable. To avoid future issues, our service provider and hosting provider will both continue to work on implementing solutions to further improve the network latency and stability. Thank you for your understanding, The TrekkSoft Product Team

Dear users, After the security update by our hosting provider yesterday , our servers had decreased performance and slower network latency. This was mitigated as much as possible by our server provider, but there is still work to do together with our hosting provider to resolve some outstanding issues. In the meantime, everything is stable and our performance is back to normal. However, we might experience temporary slowdowns again in the near future until this case is fully resolved. Thank you for your udnerstanding, TrekkSoft Product Team

Dear users, Due to the changes implemented by our service provider addressing the discovered vulnerabilities on their side, we are experiencing decreased performance. We are already in touch with them to get it resolved as soon as possible. Apologies for the inconvenience. Best regards, TrekkSoft Product Team

Dear users, Our service provider had to do a reboot on their side due to an emergency maintenance. They have discovered some vulnerabilities on their side that had to be fixed and deployed. That is why our main server was not available for a few minutes. We are in touch with them currently and if there any updates, we will share them here. Once again, apologies for the inconvenience and thank you for your understanding. Best regards, TrekkSoft Product Team

Dear users, Today at 10:48 CET we were unable to access our main server a few minutes. We are talking to our service provider to see what caused this exactly. Once we receive further updates, we will share it here. Right now everything is functioning as intended. Our sincere apologies for the inconvenience caused by this. TrekkSoft Product Team

Dear users, We have been closely monitoring this issue after implementing some changes and everything seems to be back to normal. These changes were: * temporary storage of messages in case of losing queues again * refactoring how messages are added to the queue to limit unnecessary messages During the maintenance next week at 6:00-6:30AM CET we will increase our memory capacity. This means that we have less messages to process, we have safety measurements in place and we will have more capacity to process these messages. We will keep on working on this to further improve performance over the course of the next few months. Thank you for your patience so far and we apologise for any inconvenience caused.

Dear users, Yesterday at 8:44PM CET we have lost the messages from our queues again, but this time it will have a much smaller scale of impact due to the low amount of messages. We are already working on recaching these. In the meantime we have implemented the temporary storage of messages, so they will not be lost again in case of a reboot. Also, we are in touch with our service provider to increase the priority of them checking the memory capacity issues we`ve experienced recently. We sincerely apologise for the inconvenience caused.

Dear users, We have recached and regenerated all missing messages, bringing them back to our queue once again. These are currently going through to the accounts, correcting the last bits of the negative consequences of losing our message queue. The remaining actions planned on our side are more preventative measures rather than correctional steps, which will take longer to implement and won`t have immediate consequences.

We are continuing to work on a fix for this issue.

Dear users, As we wrote yesterday our message queues got emptied yesterday at 8PM CET. This affected the followings: * availabilities across all channels such as POS desk, OTAs an the frontend (TrekkSoft websites and widgets) * basket and guest caching * communication messages such as Zapier tasks and emails triggered We`ve recahed all baskets and guest for this time period. Also, the booking creation messages were regenerated. The availabilities in the meantime got updated as they regularly do, so no manual involvement was necessary from our side. As the next step we will recache all addiotional messages for the queue and implement storing for these messages so we can avoid losing them in the future and they won`t have to be re-generated. Moreover, we are in touch with our service provider to avoid running out of memory again and losing all our message queues. As soon as all these remaining steps are complete, we will share another update. Once, we apologise for the inconvenience caused by this and thank you for your understanding.

Dear users, Today at 8:04pm our message queues (software that manages "messages" in order to process them asynchronously) got emptied. We already identified what caused this and are working on a fix. We will provide a more detailed overview and status update as soon as possible and apologies for the inconvenience caused.

Saferpay is now 100% operational. All payment systems are OK.

Currently, users may experience issues with online payments via TrekkPay payment gateway. This is due to an issue with Saferpay, which is part of the TrekkPay payment system and is currently being investigated. At present, Saferpay is unable to provide a timeframe for when this will be resolved. From our side we are working with TrekkPay for alternative options to minimise the impact of this issue.

Yesterday TrekkSoft was suffering from downgraded performance, so the usual tasks were much slower than usual. Now, we are in the process of adding new servers as a mid-term solution. These will be operational early next week.

We are continuing to monitor for any further issues.

TrekkSoft is experiencing delays due to the number of messages piled up in our API. Because of this, some of the information will take up to 2h (maybe less depending on what changed) to be visible on the website. The main thing what is affected is the booking calendar on the website. We added +20% resources for queue processing to help lean up our queue.

We are continuing to work on a fix for this issue.

TrekkSoft is experiencing delays due to the number of messages piled up in our API. Because of this, some of the information will take up to 2h (maybe less depending on what changed) to be visible on the website. The main thing what is affected is the booking calendar on the website. We asked our server team to add 20% more resources to this queue. In around an hour this services should be already up and running.

TrekkSoft is experiencing delays due to the number of messages piled up in our API. Because of this, some of the information will take up to 2h (maybe less depending on what changed) to be visible on the website. The main thing what is affected is the booking calendar on the website. We asked our server team to add 20% more resources to this queue. In around an hour this services should be already up and running.

VSHN Servers are suffering an outage, they are currently investigating but all TrekkSoft services are now back.

We are currently investigating this issue.

Detailed Incident Report from our Internet Service Provider: On 2019-01-07 at 15:35 CET, we were notified by our network monitoring system that we were facing partial packet loss to various destinations in the Internet. After short investigation we discovered that the BGP sessions with all of our upstream providers have been reset at the same time and were re-established after a few seconds. Between 15:33 and 15:50 CET, the BGP sessions were reset and re-established repeatedly which caused further partial packet loss for Internet-facing connections. After that, the situation was stable again. However, as a result of the numerous BGP resets in such a short period of time, our prefixes became the victims of BGP route dampening and therefore, our services may not have been available via certain Internet providers for a longer period of time. At 16:25 CET, we received confirmation from various customers and our network monitoring system that our services were reachable from all over again. Root Cause Analysis Shortly after 16:00 CET, we were already in touch with the vendor and discussed potential workarounds. It quickly turned out that we were by far not the only ones affected and that these BGP session resets were caused by the DISCO experiment, which triggered a bug in our routing stack. The bug was triggered by the use of a BGP attribute reserved for development in the virtual network control (VNC) code of our routing stack. VNC was using "255" as a development value for features that were never standardized. The intent was to disable this usage for non-development use, but this did not happen. Since "255" was a known attribute, the software tried to parse this attribute (that was generated as part of the experiment) -- and failed as it was of an unknown format. This failure in turn resulted in the common attribute parsing error behavior being triggered. RFC 4271 mandates a session reset in this case. Steps Taken Between 18:00 and 19:00 CET, in an emergency maintenance window, we installed a patch containing a workaround for the VNC issue mentioned above. The vendor is now working on a final solution to prevent this from happening again. We are considering sponsoring the implementation of RFC 7606 to contribute to BGP stability in our routing stack in the future. Please accept our apologies for the inconvenience this incident may have caused you and your customers. We keep doing our best to prevent such situations from happening. DISCO BGP Experiment - https://mailman.nanog.org/pipermail/nanog/2018-December/098458.html What is Route Dampening? - https://itknowledgeexchange.techtarget.com/network-technologies/what-is-route-dampening/#post-1482 RFC 4271 - https://tools.ietf.org/html/rfc4271#section-6.3 RFC 7606 - https://tools.ietf.org/html/rfc7606

We are in contact with our internet service provider and have requested a detailed incident report.

We have verified the issue with our hosting providers, internet service provider and have reported the fault.

Incoming traffic from outside of Europe is experiencing intermittent connectivity issues with our system. We are looking into the issue.

## Summary Our upstream payment provider experienced an issue which affected our ability to process SEPA direct debit payments for German companies. ## Impact From TrekkSoft, only one company was affected. However, all MyObis' companies suffered this outage. The downtime occurred on Friday 14th of Dec. ## Resolution We will use Stripe Connect to ensure there is no interruption to the service levels we offer you. We envisage this will be a seamless process for you. Where we will require your support is consenting for us to accept the Stripe terms & conditions on your behalf. It will be a once off process within your TrekkPay/Payyo account to complete the KYC process.

This incident has been resolved.

Issue with SEPA direct debit payments for German merchants. Our upstream payment provider, is currently experiencing an issue which is affecting our ability to process SEPA direct debit payments for some German merchants. We will resolve this incident once we have received confirmation the issue has been resolved from our upstream provider. -- DE: Problem mit Zahlungen per Elektronischem Lastschriftverfahren für Deutsche Händler. Bei einigen deutschen Händlern kann es aktuell zu Problemen bei Zahlungen mit ELV (Elektronischem Lastschriftverfahren) kommen. Wir sind bereits mit unserem Zahlungsanbieter diesbezüglich in Kontakt und beheben es, sobald wir seinerseits eine positive Rückmeldung erhalten haben.

Dear users, we’re now back up and all our systems are fully functional. We were down for 26 minutes approximately due to problems with our database. We’ve learned a lot from this experience and we’re working on preventing these kind of issues in the future. Everything should be working normally now. Please contact support@trekksoft.com if you run into any issues. Best regards, TrekkSoft Team

Dear users, we’re currently experiencing a downtime in our TrekkSoft dashboard. We’re working to bring it back as soon as possible. We calculate a downtime of 10 minutes. Sorry for the inconveniences and thank you for your patience.

New viator bookings are available in TrekkSoft again. This issue is resolved.

We are continuing to investigate this issue.

Viator sends currently bookings via email and not via TrekkSoft API. We're trying to solve this issue with the support of Viator.

We have received an official statement from Saferpay and would like to share it with you: "The partial failure of the Saferpay module on Tuesday, 7 August 2018 at 7.50 pm meant that transactions could largely not be processed. The internal analyses of the specialists revealed that a problem occurred in the database connection, which meant that the Saferpay database could not be accessed. From 22.06 h the system operation was completely available again. The cause of the fault is still under investigation with the manufacturer." We would like to apologize again for any inconvenience this caused.

We have now received confirmation from Saferpay that they are aware of the outage and actively working on a solution. In the meantime we were able change the payment service provider for all TrekkPay merchants, meaning payment processing through TrekkPay is back to regular operation status. Of course we're still monitoring the situation closely and we will update this incident as soon as we receive more information.

We are currently experiencing problems with online payments processing through SIX Saferpay. We are ourselves currently unable to access the Saferpay back office application. We will update this incident as soon as we have more information available and apologize for any inconvenience this may have cause in your operations.

According to our service provider they did some maintenance work (operating system patches) on their systems between 6 AM and 7 AM yesterday, that may have caused two short periods of unavailability. At this point that's all the information we were able to gather. All systems and payment processing have been running smoothly for the last 24 hours. We again apologize for any inconvenience this incident may have caused in your operations yesterday.

Since approximately 8:45 AM CET no more errors were reported. We have inquired with our service provider for more details.

We're investigating communication issues with one of our payment service providers. At this time we can say that only some payments are affected, even on the aforementioned service provider. We'll update this incident once more information is available. We apologize for any inconvenience this causes in your operations.

The outage yesterday was caused by a hardware failure on Visa Europe's part. "Visa has resolved a technical issue which occurred yesterday in Europe and prevented some consumers from using Visa for payments. The issue was the result of a hardware failure within one of our European systems and is not associated with any unauthorised access or cyberattack." The full statement is available here: https://www.visaeurope.com/newsroom/news/visa-service-disruption We again apologize for any inconvenience this may have caused in your operations.

Between 15:40 CET and 16:20 CET some payments on one processor were not going through. Throughout this time we saw successful payments. We are not sure yet if specific credit card brands or currencies were affected, or if it was a general issue. This provider has since recovered and we see no more failed payments. However, starting at 16:35 CET a different provider started having issues related to 3D Secure. In the meantime we have made sure that all our merchants are running through a working provider. We'll update this issue when more information becomes available.

We're investigating reports of payments failing due to communication errors with our payments provider. At this time we can say that only some payments are affected. We'll update this incident once more information is available. We apologize for any inconvenience this causes in your operations.

Today at 3:49 PM and 5:13 PM TrekkSoft application was down for about 5 minutes each time due to a glitch on our loadbalancers virtual IP registration (we use 2 loadbalancers with one virtual IP on them, so when one loadbalancer is down, our IP goes to second one). Our server provider has carried out a short maintenance work to solve this issue and prevent that from happening in future. We apologise for that small inconvenience.

Our hosting provider had networking issues and now everything is resolved. They will follow up to find out and resolve a root cause of this issue, but our website is up and running now.

We are up again. We are waiting for more information from our hosting provider about this incident and will be monitoring system health for some time.

We are currently experiencing full outage. There are probably some problems at our hosting provider. We are working on identifying and solving this issue.