Is Squiz Down Right Now? Check if there is a current outage ongoing.

We are continuing to investigate this issue.

We are experiencing service degradation on the DXP environment. Currently investigating

Our monitoring has detected a performance degradation affecting UK Funnelback customers only. We are currently investigating the issue and will provide updates as soon as possible.

Systems have now recovered, and analytics data has been restored. The incident is resolved.

The fix has been implemented and analytics services have been successfully restored. We are currently monitoring the system to ensure continued stability and performance.

We are continuing to investigate the issue and are actively working on a fix.

We have identified the issue and can confirm that it only affects UK customers. We are implementing a rollback to restore analytics services.

Our monitoring has detected an issue resulting in some Funnelback analytics data being unavailable. Our engineers are investigating the issue and updates will be made regularly to this status page.

**Summary** On April 24, 2025, Squiz experienced a service degradation affecting several Funnelback DXP customers in the UK. Squiz alerting detected an increase in traffic, which led to heightened monitoring and as a result a Major incident was declared at 09:01 BST. Prior to this declaration some customers may have experienced periods of slower than usual response. Whilst monitoring was in place, an unexpected spike in traffic from a small number of sources resulted in a strain on system resources, and led to a degraded service. Rate-limiting was applied, as well as blocks to specific traffic which was deemed to be non-human. The system began to recover and full restoration of services was confirmed by 09:59 BST. ‌ **Customer Impact** A subset of UK-based customers using Funnelback DXP services experienced slow response times, degraded search performance, and intermittent 504 service errors. The impact was localised to the UK region, with no disruptions reported in other regions. ‌ **Issue, Resolution, and Mitigation** Upon investigation, Squiz engineers identified unusually high levels of traffic from a small group of actors. This traffic surge led to system congestion, resulting in delayed responses and intermittent availability for some customers. The issue was addressed through the following actions: * Squiz detected early signs of degradation via system alerts. * Engineers investigated and declared a major incident at 09:01 GMT. * High-traffic sources were identified and rate-limited to ease the load on the system. * Following the implementation of rate-limiting, service performance began to improve steadily. * Engineers actively monitored the environment to ensure continued stability. * By 09:59 GMT, full service was restored and the system stabilised. * Bot management controls were enabled to mitigate excessive automated traffic, and remain in place to ensure continued stability.

Search performance has returned to normal, and the incident is now resolved. We will continue to monitor closely, but no further impact is expected. Thank you for your patience.

A fix has been implemented, and we are actively monitoring system performance to ensure the issue has been resolved. We will continue to observe closely and provide a final update once we confirm that everything is operating normally.

We have identified the issue and are now working on implementing a fix. Further updates will be provided shortly.

Our engineers continue to investigate the issue and are working to identify the source of the problem.

Our monitoring has detected a performance degradation affecting UK Funnelback customers only. We are currently investigating the issue and will provide updates as soon as possible.

**Summary** On the 2nd April at approximately 14:49 \(GMT\+10\) Squiz received indications of 500 Errors from customer pages. Squiz’s Support teams alongside our product team, quickly identified that the issue was induced by a recent DXP Upgrade version 6.61.0. Hot patches were released whilst in parallel Matrix DXP was rolled back to 6.60.1 restoring services. **Customer Impact** Incident Duration: 02 Apr 2025, 14:49 - 23:14 \(GMT\+10\) Impact: some customers experienced site page 500 errors. Impact times and service restoration times varied throughout the course of the incident duration. The effect of this issue was limited to clients who changed asset statuses during a specific period of time which meant that impact was only felt by some users who would have been editing assets at the time of the incident. **Root cause Analysis** An asset property was removed in Matrix Version 6.61.0. This impacted assets that were placed into Safe Edit as it resulted in errors when Matrix attempted to serialise objects. Resolution Actions 1. Identification Squiz Support Team identified a trend in logs when investigating reports of problems. Product teams were engaged with and quickly isolated the cause. 2. Hot Patch Squiz developed, tested and deployed hot-patches, whilst in parallel assessing version Rollback vs Roll forward. 3. Downgrade To fully resolve the issue a Matrix Version downgrade took place. **Follow-up Actions** Squiz has deployed monitoring enhancements to have the ability to detect/monitor for similar events including identification during testing - completed Squiz has rolled out Matrix Version 6.61.1 successfully, which introduced a change to circumvent this issue. - completed

Dear Customers, Following an extended period of monitoring, we are pleased to confirm that this issue has now been resolved. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

We are continuing to monitor for any outstanding issues. A further update will be provided once the rollback has been completed.

We are continuing to see recovery for more affected customers and believe we are nearing resolution. We are continuing to be vigilant for any further issues.

We are continuing to progress with the rollback of the changes. We are continuing to see recovery for some affected customers. We are actively checking reported outages for recovery and are continuing to monitor for developments.

Rollback of the changes continues, and we are now seeing recovery for some affected customers. We are actively checking reported outages for recovery and are continuing to monitor for developments.

Deployment of the rollback is now underway. We will continue to monitor for any issues during the deployment.

We are continuing to work on the rollback to resolve the issues.

We are continuing to work on rolling back the changes and are continuing to monitor the situation.

We have Begun rolling back clients with the affected version of DXP and have begun monitoring them to see if there are any further issues.

We are now implementing a Rollback for the affected version of DXP. We will have another update ready once the operation is complete.

Squiz are continuing to work on a Fix for this issue and will implement a fix as soon as we have positive test results.

We are continuing to investigate this issue.

Squiz has been made aware of an error affecting customers on the latest version of DXP has caused an error when assets have been placed into safe edit where it will cause an error 500 on trying to access the asset in question. Squiz is working on Rolling back affected clients now.

We believe that the issue is now resolved. However we will continue to monitor the affected instances to ensure that the issue does not reoccur. Feel free to raise an incident Via My.Squiz.Net if you believe you have a related issue.

We have implemented some performance fixes and have carried out actions to clean up the processes on the instance affected. We are monitoring to see if the performance of the searches affected are back to normal.

We are looking into the performance of the affected instances and are looking to implement further resources in order to resolve the issues. We will update with any findings from this action.

Our Team continue to look into the cause of the impact to Funnelback Performance. We will provide an update once we have any further information on the cause.

We are currently investigating errors with DXP Funnelback instances where some Australian instances are seeing Funnelback search results impacted. Our Product Development team are looking into this with our engineers to resolve this as soon as possible.

### Postmortem **Summary** On March 26, 2025, Squiz observed a service degradation affecting several Funnelback DXP customers in the UK. The incident began at 11:57 GMT and lasted for several hours, during which customers experienced slow search responses and degraded service. Engineers identified a potential cause of the issue at 12:24 GMT, and by 13:20 GMT on March 27, signs of recovery were observed. Full service restoration was confirmed by 16:12 GMT, and the issue was resolved. The degradation was primarily caused by unusually high traffic levels coming from a few specific actors, which overwhelmed the system and led to service disruptions. This traffic was subsequently rate-limited to mitigate the impact and restore service functionality. **Customer Impact** A subset of UK customers using Funnelback DXP services experienced slow response times and degraded search functionality. Some users also reported intermittent 504 service errors. The impact was localised to the UK, with no other regions affected. **Issue, Resolution, and Mitigation** Upon investigation, Squiz engineers identified unusually high levels of traffic coming from a small group of specific actors. This high traffic load caused congestion in the system, resulting in delays and intermittent service disruptions for affected customers. The issue was addressed in the following steps: 1. The traffic from the specific actors was identified and rate-limited to reduce the load on the system. 2. Once the rate limiting was applied, service performance began to improve, and the degraded functionality started to recover. 3. Engineers continued to monitor the system for further anomalies and ensured that the rate-limiting measures were effectively restoring normal operations. 4. By 16:12 GMT, the system was fully stabilised, and all affected services were restored.

The issue has been addressed, and services are fully restored. We will continue to monitor system performance, but the incident has now been resolved. A Post-Incident Review (PIR) will be available in the coming days.

Our engineers continue to monitor the situation, as we see signs of recovery.

We've started to see improvements in search functionality as we continue to monitor the situation.

Our engineers are continuing to work through the current incident and are actively testing solutions to resolve the issue.

Our engineers have identified a potential source of the issue and are working on implementing a fix.

We are continuing to investigate this issue.

Squiz has detected a degradation of service impacting some Funnelback DXP customers in the UK only. Some customers may experience slow response and or degraded service. We are working to mitigate this problem. More updates to follow shortly. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

The maintenance has now concluded and we do not expect any further outages at this time.

Our service provider has provided the following details of the US & EMEA outage, we expect similar behaviour during the AU maintenance window. During maintenance, all integration flows were temporarily suspended and resumed after the completion of the upgrades to process all received data. The user interface (frontend) was disabled ensuring a smooth transition and minimal disruption of services.

Squiz teams are currently monitoring for any outages caused.

Some customers may see a degradation of performance with connect flows. This is due to a third party provider carrying out essential maintenance. US and EMEA clients service is now performing as normal, as maintenance in this region is complete. AU maintenance will be taking place during the following window. AU Tenant Window: Start: 1:00 PM GMT End: 4:00 PM GMT

We are pleased to confirm that the previously reported issue affecting the performance of our Funnelback system has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

Our engineers have found and deployed a solution for this issue, we will continue to monitor for the time being. Performance is now returning to a normal level, further updates will be provided as they are available.

Our engineers have found and deployed a solution for this issue, we will continue to monitor for the time being. Performance is now returning to a normal level, further updates will be provided as they are available.

We believe the issue has been identified and we are implementing a fix, we are starting to see some service level improvements. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Our teams are continuing to investigate the issue at this time. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

We are continuing to investigate the issue. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service impacted some Funnelback Saas customers in the UK only. Some customer may experience slow response and or degraded service. We are working to mitigate this problem. More updates to follow shortly. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Dear Customers, We are pleased to inform you that this issue has now been resolved. We sincerely appreciate your patience and understanding as we worked closely with Microsoft and our partners to address the matter. If you encounter any further issues, please do not hesitate to contact our Customer Support team. Kind regards, Customer Care

Dear Customers, We are continuing to work with Microsoft and our partners to resolve the issue announced. We have observed improvements, however until fully resolved kindly regularly review https://my.squiz.net/ for Support Ticket updates. As reminder, If you are a customer impacted as a result of this, please whitelist our domain using the following method and this appears to be the quickest resolution method. - Check Junk mail folder - Report the emails as "Not Junk" - Add to safe sender list We apologise for any inconvenience and will continue to provide an update as the situation improves. Kind Regards Customer Care

Dear Customers, Unfortunately this issue still persist and we have observed instances indicating that the impact has extended to Support Email notifications, as a result we would kindly ask you to regularly review https://my.squiz.net/ for Support Ticket updates. We continue to work with Microsoft and our partners to resolve the issue announced. As reminder, If you are a customer impacted as a result of this, please whitelist our domain using the following method and this appears to be the quickest resolution method. - Check Junk mail folder - Report the emails as "Not Junk" - Add to safe sender list We apologise for any inconvenience and will continue to provide an update as the situation improves. Kind Regards Customer Care

Dear Customers, We continue work with Microsoft and our partners to resolve the issue announced. As reminder, If you are a customer impacted as a result of this, we recommend whitelisting our domain using the following method, - Check Junk mail folder - Report the emails as "Not Junk" - Add to safe sender list We apologise for any inconvenience and will continue to provide an update as the situation improves. Kind Regards Customer Care

Dear Customers, We've seen an increase in outbound emails and calendar invites from @squiz.net going to Junk in Microsoft 365 and Outlook based accounts. This does not impact notification emails for support tickets raised. This issue is external to Squiz and we are working with Microsoft to find and implement a solution. If you are a customer impacted as a result of this, we recommend whitelisting our domain using the following method, - Check Junk mail folder - Report the emails as "Not Junk" - Add to safe sender list Thanks for your patience in this matter and we will provide you with more information as soon as possible. Kind Regards Squiz Customer Care

This Issue has now been considered resolved after our observations. Feel free to get in contact by MySquiz.net to let us know if you have any questions.

The patch mentioned in our previous update has now been applied across production. We will continue to monitor to see if this has been resolved.

Our Team have found a Patch for this issue which has worked on a test instance. We are now working on applying this to a wider group.

Squiz has identified an error with Rest resource assets from an update applied overnight and are now investigating a fix for the issue. Feel free to get in contact via MySquiz if you believe you are affected.

**Summary** Several AU customers experienced website degradation between Jan 14, 2025, 23:47 AEST - Jan 15, 2025, 1:40 AEST. Squiz identified operational issues with one of our third party network providers. This had a negative effect on Matrix and Funnelback services in the AU, leading to search function disruptions, latency for several customers and some 504 errors. On January the 15th, at 19:51 AEST, during a scheduled maintenance window there was a further observed degradation of service. This was a re-occurrence of the same issue. ### **Customer Impact** A small subset of AU Customers experienced delays in search results when attempting to utilise the Funnelback search and web functions, as well as some more 504 service outages. ### **Issue, Resolution and Mitigation** We experienced concurrent intermittent traffic loss to/from NTT in both our Sydney and Melbourne DCs. The traffic loss was severe enough to trigger automatic rerouting of Ingress traffic to a different Transit provider. Because the packet loss was intermittent, this rerouting process resolved, then repeated several times. We intervened manually to force the exclusive use of a different Transit Provider in Sydney. This partially mitigated the issue, but it took some time for routing to fully recover. This also affected some of our internal observability systems. Once the NTT transit was stable, we reverted our mitigations to restore full redundant service.

This incident has been resolved and all services are up and running

A fix has been implemented and services are restored. We are currently still monitoring the situation.

We are continuing to investigate this issue.

We are currently witnessing intermittent network issues with one of our transit providers in the AU. We’ll continue to investigate the situation and provides updates as soon as possible.

09:57-10:13am GMT Excess traffic towards the UK DXP Search Admin API caused a denial of service situation where some users received 504 Gateway Timeouts. Service was restored by cycling services and reducing traffic.

### Summary Squiz identified performance issues with our Compute Service in the UK. ### Customer impact A subset of UK Customers may have experienced delays and timeouts when attempting to access Squiz UK hosted sites. ### Issue and Resolution The root cause of this incident was a DDoS attack. The majority of the attack was automatically blocked by our DDoS protection, but the remainder caused a performance impact to our Compute service in the UK. We performed additional manual intervention to completely mitigate the impact. ### Mitigation In light of this incident, We are investigating a number of approaches in order to reduce the effect of a similar event in the future.

We are pleased to confirm that the previously reported issue affecting the performance of our UK hosted systems has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

Following a thorough investigation from our engineering teams, we have performed mitigation and remediation actions. These actions have been successful as we are currently experiencing no significant performance impact. At this time we will continue to monitor the service.

We have identified this is as a result of DDoS attack which we have partially mitigated, however we are still experiencing issues at this time and our engineers are working to fully mitigate this. Further update to be provided as soon as there is more information available.

We are now monitoring the performance after our engineering have been able to see some results so far. An update shall be available once we identify no issues.

Squiz are continuing to look into the potential causes of the performance degredation at hand and will have an update as soon as pertinent news has arisen.

We are continuing to investigate this issue.

Hey Everyone. Squiz has identified issues with the performance of our UK servers and wish to let you know that our engineering team is looking into this matter with top priority and will have updates for you as soon as they are available. Feel free to let us know by our Service Desk if you have any concerns.

This issue has now been resolved. We have retested Connect AU and the UI errors are gone and the service is working as expected.

The SSL certificate has now been renewed and Connect in AU is working as expected. We will continue to monitor for any lasting issues.

Our network teams are engaged on resolving this issue. We will provide further updates as more information becomes available.

Our team continue to work on the SSL certificate renewal

Our team is actively working on the SSL certificate renewal Certificate update process is in progress

We are pleased to confirm that the performance of our UK Funnelback DXP system has been restored. Our team were able to apply a fix for the issue, which led to significant improvements in performance. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. We'll continue to monitor the situation and be quick to provide updates where necessary. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

We are now beginning to see service improvements in our Funnelback search performance after implementing a fix. We'll continue to monitor the situation and look to confirm resolution when possible.

Our engineering team has identified an issue with the DXP Funnelback and is in the process of implementing a resolution.

Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only. Some customers are experiencing slow response times and/or timeouts. We are currently working to find the root cause of this issue. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

This incident has been resolved

Squiz teams have implemented a fix that appears to have resolved this issue for now and we are now monitoring the situation to ensure stability is retained.

We are currently experiencing connectivity issues with non production services, hosted in our UK Datacenter. Our Network teams are currently investigating.

### **Summary** Squiz identified operational issues with Funnelback services in the UK, leading to search function disruptions and latency for several customers. ### **Customer Impact** A small subset of UK Customers may have experienced delays in search results when attempting to utilise the Funnelback search function. ### **Issue and Resolution** We observed a steady increase in traffic, which was able to bypass previous rulesets, by appearing to be from more valid sources. As a result, more targeted and bespoke rules and rate-limiting controls were put in place to prevent systems from being negatively impacted. The traffic these rules targeted were of an automated nature, and no "real person" traffic was impacted by them. We have found no evidence that this was an attack of any sort. In rare circumstances some customers were negatively impacted as a result of these actions. Squiz support teams have liaised with these customers directly to resolve these issues. ### **Mitigation** We have Identified additional protection measures that have now been implemented across our FB DXP Services. * This include enhancing caching rules * Enhanced rate limiting for specific automated traffic sources eg Facebook and Tencent \(WeChat\)

We are pleased to confirm that the previously reported issue affecting the performance of our UK Funnelback DXP system has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

We continue to monitor the situation and will confirm resolution as soon as possible.

Customers are now confirming that services are recovering and performing as expected. Our team continues to monitor the situation closely to ensure everything remains stable.

We are still seeing service improvements in some areas of our Funnelback search performance. We'll continue to monitor the situation and look to confirm resolution when possible.

We are noting sustained improvements in Funnelback search performance. We will maintain close monitoring of the situation and will provide confirmation of resolution as soon as it is feasible.

We are now beginning to see service improvements in our Funnelback search performance. We'll continue to monitor the situation and look to confirm resolution when possible.

Our engineering team has identified an issue with the DXP Funnelback and is in the process of implementing a resolution.

We are continuing to investigate the situation and our engineers are working to find the root cause of this issue.

Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only. Some customers are experiencing slow response times and/or timeouts. We are currently working to find the root cause of this issue. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

We are pleased to inform you that our support phone line has been successfully repaired and is now fully operational.

Please be advised we are currently experiencing issues with our Support Telephone lines. This issue is not affecting other services so please continue to contact us via email at support@squiz.net and through MySquiz. This is only affecting the below phone lines as far as we are aware: Squiz 24x7 Helpdesk: (+61) 1300 077 849 from Australia (+64) 4 805 0098 from New Zealand (+44) 808 267 7849 from UK / Europe (+1) 877 394 0640 from the US

This update should be out as the latest patch. Feel free to get in touch at support@squiz.net if you see any further issues.

The proposed patch has now been pushed into the release cycle. Next update here will be posted once this is complete.

We have further tested the patch in question and are looking to release to more test sites.

We have gone through a new stage of testing with no issues to the solution so far.

Further testing of a proposed solution are underway and we are awaiting more positive results before pushing.

We are looking at a possible solution and are being sure to test all facets to ensure a safe fix.

We are continuing to work on implementing a fix to compile previews into the component directories.

We are continuing to implement a solution that we believe will fix the issue.

We are continuing to implement a solution that we believe will fix the issue.

We have identified the issue and have deployed a fix to customers who have reported an impact. We will be deploying the fix to remaining customers soon. If you are affected and wish to be prioritised to have the fix deployed, please contact Squiz.

We identified an issue and applied patches to all known instances. We are currently monitoring the outcome of the patch.

We are aware of an incident affecting DXP customers when making changes when editing components, causing the updates not to appear in the admin interface. We are currently investigating the issue.

### **Executive Summary** On the 1st of August 2024 Squiz received client reports that Funnelback services had sporadically begun returning **504** errors whilst trying to access DXP hosted search services. ‌ Shortly after Squiz Engineers identified an increase in traffic to our DXP Funnelback infrastructure that led to search timeouts. After initially investigating the potential threat of malicious activity, the team identified this traffic was legitimate and scaled our infrastructure to meet the demands of this new traffic. ‌ **Timeline: All times 01/08/2024: \(BST\):** **08:49** Squiz receives reports of degradation of DXP Funnelback Services from UK clients. Squiz technical support team begin investigation. **08:50 - 09:48** Squiz Technical Support Engineers and Developers work to identify the cause of issues. **09:48** More clients report issues with the service and Squiz declares a Major Incident. **10:01** Squiz Cloud Engineering team identifies that Funnelback Query Processors are under heavy load. Work to identify the cause of the load. **10:11** Squiz identifies an increase in traffic levels has been caused by legitimate traffic. Decision is made to scale the infrastructure to meet demand. **10:27** The first of the two \*Query Processors is taken offline to double its capacity. **10:42** The first uplifted Query Processor is back online, serving traffic and marked as Healthy. The second Query Processor is taken offline to double its capacity. **10:43** Squiz Engineers begin to see recovery of client services. Monitoring phase begins. **10:48** The second uplifted Query Processor comes online, serving traffic and marked as Healthy. **11:10** Squiz monitoring sees all systems as healthy and officially resolves the incident. **\*Query Processor**: Server responsible for serving Funnelback search requests. ‌ **Customer Impact** Shortly before and during the incident clients saw an increase in 504 errors. The impact of these errors varied depending on clients' usage of Squiz Funnelback services. This included outages of systems reliant on Funnelback for login procedures and failure to customer search results ‌ **Root Cause** The root cause was identified as the servers being under-resourced for the amount of traffic, which was resolved by right-sizing for capacity. Additional mitigation was performed by increasing capacity further for additional head-room while we investigated the source of the traffic. We have since identified the source of traffic and taken steps to reduce malicious and low-value bot traffic. This has further increased head-room on the servers. Instead of scaling resources back, we've left them scaled up to ensure no issues during clearing ## **Mitigation and Follow-up Actions** **Increased Capacity** During the incident the capacity of the Query Processors were doubled in-order to stabilise the system. Post incident the servers have been uplifted and now have **eight times** the capacity of the original size. This action constitutes our main mitigation strategy and follow-up plan, offering ample headroom for Query processing. **Increased Monitoring** Squiz is currently enhancing our monitoring processes for the DXP Funnelback Service. This initiative encompasses a comprehensive review of our server load monitoring practices to proactively alert engineers about potential high load situations well in advance, preventing them from escalating into critical issues. **Improved identification of unoptimised code** Our Cloud Engineering team has dedicated efforts to enhance our capability in pinpointing collections that are not operating optimally. By effectively identifying these under-performing collections, we can optimise them to boost overall system performance.

We are pleased to confirm that the previously reported issue affecting the performance of our Funnelback DXP system has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

We are continuing to see service improvements across the board. We'll continue to monitor the situation and look to confirm resolution when possible.

We are beginning to see slight improvements in Funnelback search performance. Squiz will continue to investigate the root cause and provide regular updates via this status page.

We are continuing to investigate the situation and our engineers are working to find the root cause of this issue.

Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only. Some customers are experiencing slow response times and/or timeouts. We are currently working to find the root cause of this issue. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Our monitoring systems found slowness on some DXP Funnelback instances. Squiz is actively working on finding the root cause and fixing the issue.

### Summary Squiz identified operational issues with the query processing resources in the UK DXP, leading to queuing and slower response times on searches being processed in the UK region. ### Customer impact A subset of UK Customers may have experienced delays in search results and timeouts when attempting to utilise the search function. ### Issue and Resolution Squiz engineers identified an increase in query volumes on the processing compute layer. Whilst clients have independent query processing capabilities, the overall compute layer powering this has a finite ceiling. The increased traffic was identified as a potential DDOS masquerading as valid search traffic. At no stage was there any breach of our systems. Once this was identified, Squiz Cloud Engineering was able to pinpoint the pattern of traffic and our web application firewall \(WAF\) was reconfigured to stop the the negative affect on our compute layer. This had an immediate impact on search performance and resulted in restoration of service. The degraded service was restored at 11:08 GMT. ### Mitigation In light of this incident, we are reviewing our alerting thresholds to detect and prevent these issues sooner.

We are pleased to confirm that the previously reported issue affecting the performance of our Funnelback DXP system has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

We have identified the root cause of this issue and have implemented a fix. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service impacting some Funnelback DXP customers in the UK only. Some customers are experiencing slow response times and/or timeouts. We are working to find the root cause of this issue currently A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

The incident has now been resolved.

team believes that we have found the issue and are currently working on applying a fix.

The team still continues to investigate the issue

Our Team are still investigating the root cause of this issue.

Still investigating the issue We are still working with AWS to identify the cause.

We are in contact with AWS now and investigations are still underway

We are continuing to investigate this issue.

Still investigating the issue We are currently contacting AWS to get further support

Squiz continue to investigate a degradation of service for Squiz DXP customers in Australia and New Zealand. A further update will be provided in ~15 minutes.

This incident has been resolved.

We are declaring this incident closed The production pipeline will continue to run as we upgrade all other instances. We will monitor the pipeline throughout the afternoon

We are currently waiting on pipelines to run so we can release the fix.

We are currently waiting on pipelines to deploy the fix to develop so we can test

we are currently deploying the fix to a canary uat system to test it further before deciding if we release to production.

Squiz is aware of an intermittent issue affecting SaaS DXP users ability to save metadata values in the CMS. Our team is currently investigating this issue and will provide ongoing updates

Issues with DXP admin panel have been resolved.

The issue with intermittent 504s in the DXP admin panel has been identified

### Summary During routine monitoring, Squiz identified operational issues with Funnelback services in the US, leading to search function disruptions and latency for several customers. ### Customer impact A subset of US Customers may have experienced delays in search results and encountered 500 errors when attempting to utilise the Funnelback search function. ### Issue and Resolution Squiz engineers were alerted to errors and timeouts originating from our Squiz hosted Funnelback services in the US. This was isolated to the search sessions feature, which was subject to slow response times or termination due a build up of stored requests. In response, we disabled the session storing feature and alleviated the database usage. This will not cause any disruption to search traffic. As part of our standard process we initiated a period of heightened monitoring leading to resolution on May 13th at 17:20 BST ### Mitigation We have added new monitoring checks to flag excess database usage as well as utility scripts to help us debug slow performance in the future. Our Product team is investigating approaches to improve session performance in order to improve overall query performance going forward.

After identifying issues with sessions and taking steps to repair sessions database the server performance issues have now been resolved.

Changes have been implemented to the session database which should resolve the performance issues. We are currently monitoring to confirm.

We have identified issues with search sessions and are taking steps to address the issue.

We are continuing to investigate performance issues with Squiz hosted Funnelback in the US. Our team is actively troubleshooting server issues.

We are continuing to investigate performance issues with Squiz hosted Funnelback in the US. Other regions are unaffected.

Squiz monitoring has detected a degradation of service with Squiz hosted Funnelback. We are working hard to investigate the route cause of the issue and will provide further updates via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

This incident has been resolved.

Atlassian has notified on their status page that their Jira software has an ongoing issue affecting some users when it comes to editing the body text. Current workaround is to use an external Text editor and paste the information in. Feel free to check updates here. https://jira-software.status.atlassian.com/incidents/h6rv80btf5x0

### Summary During routine monitoring, Squiz identified operational issues with multiple Funnelback servers, leading to search function disruptions and latency for several customers. ### Customer impact A subset of UK Customers may have experienced delays in search results and encountered 500 errors when attempting to utilise the Funnelback search function. ### Issue and Resolution Squiz engineers were alerted to errors and timeouts originating from our Squiz hosted Funnelback services. Investigation by Squiz engineers revealed that certain filter configurations were resulting in background optimisation calculations taking an excessive amount of time to compute. This resulted in a large amount of computation resources being locked by these calculations for extended periods, reducing available computation resources to other main searches. In response, we reduced the timing thresholds and the amount of computing power these calculations are allowed to take, in order to streamline query performance. These restrictions only apply to optional background calculations and the limitations put in place will not cause disruption to search traffic. As part of our standard process we initiated a period of heightened monitoring leading to resolution on May 8th at 11:03 BST ### Mitigation We have added new monitoring checks to flag excess computation delay as well as utility scripts to help us debug slow performance in the future. Our Product team is investigating approaches to improve filter background calculation performance in order to improve overall query performance going forward.

We can now confirm that the issues affecting Squiz Hosted Funnelback services are now resolved. Our team monitored the situation carefully, and applied a fix for the issue. We are seeing system performance back to normal after this fix was applied. We will continue to monitor the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem with further information will be made available on https://status.squiz.cloud/ in the coming days.

Performance remains stable and we are continuing to monitor for any outstanding issues at this time. A further update will provided in 15 minutes, or earlier if the situation or information changes.

We are still observing stable performance and are continuing to monitor for any outstanding issues at this time. A further update will provided in 15 minutes, or earlier if the situation or information changes.

We have now deployed a fix for the issue and are seeing an improvement in performance. We are continuing to monitor for any outstanding issues at this time. A further update will provided in 15 minutes, or earlier if the situation or information changes.

We have now identified what is causing this issue and are deploying a change to rectify this. A further update will provided in 15 minutes, or earlier if the situation or information changes.

We are continuing to investigate this issue.

Our engineers continue to investigate the issue. We will provide a further update in 15 minutes, or earlier if the situation or information changes.

Our engineers are currently investigating the issue. We will provide a further update in 15 minutes, or earlier if the situation or information changes.

We are continuing to investigate this issue.

Squiz monitoring has detected a degradation of service with Squiz hosted Funnelback. We are working hard to investigate the route cause of the issue and will provide further updates via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

### Summary During routine monitoring, Squiz identified operational issues with multiple Funnelback servers, leading to search function disruptions for several customers. ### Customer impact A subset of UK Customers may have experienced delays in search results and encountered 500 errors when attempting to utilise the search function. ### Issue and Resolution Squiz engineers were alerted to errors and timeouts originating from our Squiz Funnelback services. Subsequent investigation revealed that the search session functionality within Funnelback was causing slow or erroneous requests, leading to a build up of requests within the search query processing pipeline. Requests utilising the search session feature were subject to slow response times or termination. In turn this impacted performance and resulted in timed out searches. In response, we isolated the specific searches and their connection behaviour to our session feature and, where needed, disabled/paused the use of this feature temporarily to allow the query processing pipeline to recover. As a precautionary measure, resource allocation to the query processing pipeline was increased. As part of our standard process we initiated a period of heightened monitoring leading to resolution on April 25th at 13:00 BST ### Mitigation In light of this incident, Squiz support staff conducted a thorough review of our UK Funnelback systems to preempt future disruptions, including the expansion of memory resources. In addition, measures have been taken to enhance processes enabling fast-tracked resolutions to similar incidents in the future.

We are pleased to confirm that the previously reported issue affecting the performance of our Funnelback system has been successfully resolved. Our team closely monitored the situation, and were able to apply a fix for the issue, which led to significant improvements in performance. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused. A post mortem will be made available on https://status.squiz.cloud/ in the coming days.

We are now seeing systems recovering and are continuing to monitor in case of further issues. A post mortem for this incident will also be made available on https://status.squiz.cloud/ in the coming days.

We are now seeing an improvement in performance and are continuing to investigate the root cause of this issue.

Our engineers continue to address the issue and apply fixes where possible to improve service performance.

We are continuing to work on an effective fix for this issue.

The problem is receiving our full attention as our engineers work to improve service performance through applied fixes.

Our engineering team is actively engaged in resolving the issue and applying fixes to boost service performance.

The problem is receiving our full attention as our engineers continue to work on improving service performance through applied fixes.

We are continuing to work on an effective fix for this issue.

Our engineers continue to address the issue and apply fixes where possible to improve service performance.

Our fix resulted in initial performance improvements, however we are still seeing a performance degradation. Our teams continue to work on a full resolution.

Our engineers have identified the likely cause of the incident and have implemented an appropriate fix. We are now testing and monitoring search performance, and have started to see signs of recovery.

We are currently experiencing a degradation impacting Funnelback and continue to investigate the situation. We have announced a major Incident and have multiple teams engaged in resolution.

We are continuing to investigate this issue.

Squiz monitoring has detected a degradation of service in our Funnelback Pods Squiz is working hard to investigate the route cause of the issue and will provide further updates via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Dear Customers, We're pleased to inform you that this incident has been resolved. All systems are now back to normal operation. We will continue to closely monitor, however if you experience any further issues or have concerns, please do not hesitate to contact our support team.

All systems have now recovered and services returned to normal. Squiz will continue to monitor the situation and provide updates as required.

We are beginning to see multiple systems recovering. Squiz will continue to monitor the situation and provide updates shortly.

Squiz monitoring has detected a degradation of service in our UK Data Centres Squiz has identified the cause of the issue and is working to resolve as quickly possible. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

This incident has been resolved.

Restarted all paused VMs.

Only subset of clients is affected on a faulty node

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Only subset of clients is affected on a faulty node

Dear Customers, We are pleased to inform you that the service degradation that some may have experienced has now been resolved. All systems are now back to normal operation. We will continue to closely monitor, however if you experience any further issues or have concerns, please do not hesitate to contact our support team.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We have restored most services and are working on entering the monitoring phase soon.

We are still working on restoring services and continue to monitor services, that have already been restored.

We have identified the issue and are still working on bringing all service back to normal operation. For already restored services we are monitoring the situation.

The problem has been identified, and immediate measures have been implemented to mitigate its impact and prevent recurrence. Our team is actively monitoring the situation for any further issues to ensure a stable environment. Rest assured, we are committed to maintaining the highest level of service for our valued customers. We will continue to provide timely updates as the situation evolves. Thank you for your understanding and patience.

Squiz continue to investigate a degradation of service for Squiz Cloud customers in Europe. A further update will be provided in ~15 minutes.

We are continuing to investigate this issue.

We want to bring to your attention that we are currently experiencing degradation of sessions hosted in our Ash datacenter. This issue specifically impacts sessions originating from the northern hemisphere. Our dedicated team is actively investigating the root cause of this situation, and we are committed to providing regular updates as we gather more information. Please anticipate updates on this page every 15 minutes as we work diligently to resolve the issue and ensure the seamless functioning of our services. We appreciate your patience and understanding during this time.

### Summary Squiz identified that a query processor node was not operating optimally, which may have impacted some of our Funnelback clients in the UK. ### Customer impact Customers being served from the impacted nodes may have seen slow search results and 500 errors when trying access search. ### Issue and Resolution Squiz engineers were alerted to an increase in 500 errors and time outs being served from one of Squiz Funnelback search nodes. Upon investigation it was found that a process responsible for serving search result was running out of available memory. The available memory was increased and the issue resolved on the 2nd February 12:25 GMT. ### Mitigation Squiz support staff have taken a review of our UK Funnelback systems to prevent further issues including increasing memory.

This issue has now been resolved. We are no longer seeing a service degradation since the fix has been applied.

A fix for this has now been deployed. We will continue to monitor for further issues. Further updates will be provided via https://status.squiz.cloud

Our engineers are now working on deploying a fix for this. A further update will be provided via https://status.squiz.cloud when this has been deployed.

Our engineers are still working on resolving this issue. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Our engineers are still working on resolving this issue. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

The issue has been identified and we are currently working on a resolution. Further updates to follow.

Squiz monitoring has detected a degradation of service impacted some Funnelback customers Some customer may experience slow response and or degraded service. We are working to mitigate this problem. More updates to follow shortly. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

We are pleased to inform you that the major incident at our Ash Data Center has been fully resolved. All systems are now back to normal operation. Our technical teams have worked tirelessly to address the issue and have successfully implemented the necessary solutions to restore all services. We understand the disruption this may have caused and thank you for your patience and understanding during this time. If you experience any further issues or have concerns, please do not hesitate to contact our support team via support@squiz.net

We are pleased to report significant progress regarding the major incident at our Ash Data Center. Our technical teams have successfully identified the underlying problem. We are now in the process of implementing necessary measures to bring all systems back to normal operation. Our teams are working diligently to ensure a swift and efficient resolution. We will continue to provide updates every ~15 minutes on this page. We appreciate your patience and understanding as we work towards resolving this issue.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers hosted in Europe ONLY. Multiple Squiz teams are currently investigating. A further update will be provided in ~15 minutes.

# **Executive Summary** On October 23, 2023, Squiz received reliable intelligence suggesting an imminent threat to infrastructure and websites in Australia, New Zealand, and the United Kingdom. While no particular targets or detailed timelines were disclosed, an initial attack date of October 25, 2023, was indicated. # **Customer impact** On October 25, 2023, AEST, our monitoring systems detected a potential DDoS attack. Our engineers immediately implemented our planned response strategies, effectively mitigating a significant DDoS attack that affected only a few of our clients in the end. There was a slight degradation of service for a small number of other clients due to the nature of the DDoS tactics used, which required us to adjust our defences as the attack progressed. **At no stage was there any breach of our own systems, or those of client systems hosted by us.** # **Root cause** A hacktivist group with geopolitical motivation attacked infrastructure and websites in Australia, New Zealand and the United Kingdom. # **Mitigation** Squiz actively prepared to reduce client impact upon learning of potential attacks. We employed automated defences and other security protocols as the situation evolved. Additionally, Squiz held consultative meetings and provided updates to various government security partners throughout the incident. Squiz continues to be vigilant against any security threats and thanks you for your continued support in working with us as your trusted partner.

The previously identified issue has been successfully resolved by our engineering teams. All services are now fully operational and have been restored to normal functionality. We appreciate your patience and thank you for your continued trust in our platform.

The identified issues are still affecting services. Actions are underway to restore normal functionality. A further update will be provided in ~ 15 minutes.

Technical teams are addressing the issue to regain normal functionality. A further update will be provided in 15 minutes.

The identified problem continues to affect services. Our engineering teams are working to resolve it. A further update will be provided in 15 minutes.

Progress is being made on the identified issue; our engineering teams continue their work. A further update will be provided in 15 minutes.

While the identified issue is ongoing, we are advancing in our efforts to resolve it. A further update will be provided in 15 minutes.

The Issue is identified and still present. Our technical teams are acting to bring services back to normal. A further update will be provided in 15 minutes.

The identified issues persists. Our engineering teams are actively working to normalise all services.An update will be provided every 15 minutes.

Squiz continue to investigate a degradation of service for Squiz Cloud customers globally. A further update will be provided in ~15 minutes.

Squiz has identified an issue that is affecting our customers and we are actively working on restoring all services back to normal.

Squiz teams continue to investigate the degradation of service. We will provide with further updates once more information becomes available.

The Squiz Technical Engineering teams continue to investigate the degradation of service affecting global customers. Please bear with us. A further update will be provided in 15 minutes.

Squiz teams continue to investigate the degradation of service affecting global customers.

We are presently addressing a detected problem affecting the performance of our Edge system. Measures have been implemented, and we are noticing some improvements. For now, alerts are still consistent, and we are keeping a close eye on the situation. Please bear with us as we are still investigating this issue.

We are currently addressing an identified issue impacting our Edge system performance. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz continue to investigate a degradation of service for Squiz Cloud customers globally. A further update will be provided in ~15 minutes.

Squiz is experiencing high traffic load at the moment. Please bear with us as we are still investigating this issue.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

# **Executive Summary** Multiple Cloudflare services were unavailable for 37 minutes on October 30, 2023, between 19:54 UTC and 20:15 UTC. This was due to the misconfiguration of a deployment tool used by Cloudflare in their ongoing maintenance of Workers KV. # **Customer impact** On October 30, 2023, between 19:54 UTC and 20:15 UTC Cloudflare experienced an outage linked to their Workers KV. A number of Cloudflare services used by Squiz depend on Workers KV. These services instead received an HTTP 401 \(Unauthorized\) error when performing any get, put, delete, or list operation against a KV namespace. # **Root cause** A number of Cloudflare services used by Squiz depend on Workers KV. These services instead received an HTTP 401 \(Unauthorized\) error when performing any get, put, delete, or list operation against a KV namespace. Workers KV is a Cloudflare globally distributed key-value store. It is used by both customers and Cloudflare teams alike to manage configuration data, routing lookups, static asset bundles, authentication tokens, and other data that needs low-latency access. # **Mitigation** Squiz has actively engaged with our Vendor, Cloudflare, and they have noted that they are working on bettering their specialised tooling to ensure these events do not occur again. Squiz is also internally looking if we can assist Cloudflare in this work.

This incident has been resolved.

A fix has been implemented and we are monitoring the results. This incident has been resolved.

We are continuing to work on a fix for this issue.

Cloudflare is investigating an availability issues that impacts Workers KV, Access, Pages, Zero Trust WARP Posture, Images, Single file purge, Turnstile, and Waiting Room. We are working to mitigate this problem. More updates to follow shortly.

Squiz monitoring has detected a degradation of service incident from Cloudflare Cloudflare is investigating availability issues with Cloudflare Pages and Workers KV. Visitors may encounter errors when attempting to load Cloudflare Pages/KV based sites. We are working to mitigate this problem. More updates to follow shortly. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz has identified an issue that is affecting our customers and we are currently taking measures to restore all services to normal.

### Summary Squiz monitoring detected a degradation of service which affected Squiz Edge customers both in Europe and the ANZ region. Investigations identified an internal change that caused some Squiz Edge customers to observe SSL errors when accessing their sites or performing searches. ### ‌C**ustomer Impact** Between 12:11 AEST and 12:39 AEST on September 25 2023, some Squiz Edge customers may have had SSL errors when accessing their sites or performing searches. ### ‌**Root Cause** A change was implemented in a Squiz Edge testing environment that impacted some production servers. The change was distributed to a number of production services before it was detected and rolled back to the previous state. ### **Mitigation and Follow-up Actions** The Squiz teams are working on detecting these types of issues before a change can be distributed. The teams are also making improvements to our internal change process.

The fix has successfully been implemented and Squiz internal monitoring indicates no other issues. Our Technical Engineering teams are keeping a close eye on the environment for stability. A postmortem report will be published after further investigations.

Squiz Technical Engineering teams have identified the issue and a fix is being implemented. More updates to follow.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers hosted in Europe and in ANZ region. Multiple Squiz teams are currently investigating. A further update will be provided in ~15 minutes.

The defective hardware has been replaced in the Sydney Data Centre. No degradation to service was observed.

A hardware issue has been identified in the Sydney data centre. Replacement of this defective hardware is being arranged as soon as possible. Further updates will be provided accordingly. No degradation to service has been observed or is expected.

Troubleshooting has revealed a hardware issue in the Sydney data centre. Squiz has arranged the replacement of this defective hardware as soon as possible. No degradation to service has been observed or is expected.

An Engineer has arrived on-site at the Sydney data centre and has begun troubleshooting. There remains to be no customer impact

Squiz is currently responding to a hardware alert raised in our Sydney data centre. There is currently no customer impact and Squiz has sent an engineer to site to resolve the issue. Further updates will follow on this page.

The issue is resolved and same is reflected via our internal monitoring. No recovery actions were performed from our end as this was an external DNS issue. Squiz teams continue to monitor the situation closely.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

Our teams continue to monitor for any further issues, however this issue appears to be caused by DNS services external to Squiz

Our teams continue to monitor for any further issues, however this issue appears to be caused by DNS services external to Squiz.

We are continuing to monitor for any further issues, however this issue appears to be caused by DNS services external to Squiz. All Squiz-side services are up and running but unreachable due to these third party DNS issues.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

Some Australian sites are unresponsive due to Network DNS issues.

A workaround has been implemented and conducted tests are successful. Our support staff are able to receive calls via the Squiz (Australia) support number +61 1300 077 849 . Squiz infrastructure team continues to monitor the environment.

Dear Customers, We are experiencing intermittent connectivity issues with receiving calls via the Squiz (Australia) support number +61 1300 077 849. Our Infrastructure team is currently investigating this issue. Work Around We are reachable via other support numbers (+64) 4 805 0098 from NZ , (+44) 808 267 7849 from UK / Europe , (+1) 877 394 0640 from the US. Alternatively, please reach out to us via MySquiz or via email at support@squiz.net . We apologise for any inconvenience caused.

## **Executive Summary** On the 22nd of May 2023 at ~17:33 UTC, Squiz internal monitoring started generating alerts indicating a degradation of service for Squiz cloud customers hosted in the ASH Data Centre. Performed investigations by our Data Centre team indicated that one of the compute nodes experienced a brief network issue which in turn caused a stall on its network storage mount resulting in degradation of performance for only a few customers hosted in the ASH Data Centre. Five customer Virtual Machines needed to be restarted, with a complete recovery at ~18:16 UTC on the 22nd of May 2023. All other customer VMs auto recovered at ~17:42 UTC. ‌ ## **Customer Impact** Between ~17:33 UTC and ~18:16 UTC on the 22nd of May 2023, some Squiz cloud customers hosted in the ASH Data Centre may have experienced issues when performing search and/or were unable to access their websites. ‌ ## **Root Cause** This incident was caused by an unresponsive node which recovered without Squiz Intervention. ‌ ## **Mitigation and Follow-up Actions** The Squiz Data Centre team is currently carrying out a network improvement project to improve performance and increase network resiliency. If you have any additional questions arising from this post-incident report please contact your Squiz Service Experience Manager or Account Manager.

This incident has been resolved.

Squiz has identified the issue and we are now seeing all impacted services as recovered. We will begin monitoring the begin monitoring the situation closely.

Squiz monitoring has detected a degradation of service incident that is affecting some Squiz cloud customers hosted in our ASH Data Centre. Multiple Squiz teams are currently investigating.

## **Executive Summary** On the 17th of May 2023 at ~08:03 UTC, Squiz internal monitoring started generating alerts indicating a degradation of service for Squiz cloud customers hosted in the ASH Data Centre.  Performed investigations by our Data Centre team indicated that one of the compute nodes experienced a brief network issue which in turn caused a stall on its network storage mount resulting in degradation of performance for only a few customers hosted in the ASH Data Centre. After the issue resolved itself \(with no manual intervention\), we migrated the affected VMs to other nodes and fenced the malfunctioning compute server. This resulted in a complete recovery at ~08:21 UTC on the 17th of May 2023. ‌ ## **Customer Impact** Between ~08:03 UTC and ~08:21 UTC on the 17th of May 2023, Squiz cloud customers hosted in the ASH Data Centre may have experienced issues when performing search and/or were unable to access their websites.  ‌ ## **Root Cause** This incident was caused by an unresponsive node and restarting services on that node allowed it to recover. ‌ ## **Mitigation and Follow-up Actions** The Squiz Data Centre team is currently carrying out a network improvement project to improve performance and increase network resiliency. If you have any additional questions arising from this post-incident report please contact your Squiz Service Experience Manager or Account Manager.

This incident has been resolved.

Squiz has identified the issue and we are now seeing all impacted services as recovered. We will begin monitoring the begin monitoring the situation closely.

We are continuing to investigate this issue.

Squiz monitoring has detected a degradation of service incident that is affecting some Squiz cloud customers hosted in our ASH Data Centre. Multiple Squiz teams are currently investigating.

### **Executive Summary** On the 19th of April 2023 at ~23:11 BST, Squiz internal monitoring started generating alerts indicating a degradation of service for some Squiz Cloud customers hosted in our ASH data centre.  ‌ Performed investigations by the Squiz Data Centre team identified a network change that caused some of our customer Virtual Machines \(VM’s\) to be in an offline state. Multiple recovery actions were performed that resulted in a partial recovery for most of our customers at ~23:22 PM BST. This was followed by additional actions for the remaining customers, resulting in a full incident recovery at ~02:08 BST on the 20th of April 2023.   ‌ ### **Customer Impact** Between ~23:11 BST on the 19th of April 2023 and ~02:08 BST on the 20th of April 2023, some Squiz cloud customers hosted in the ASH data centre may have experienced a degradation of service upon accessing their website and/or upon performing a search. ‌ ### **Root Cause** An internal change being implemented impacted the live network traffic, thereby affecting some customer applications hosted in the ASH data centre.  ‌ ### **Mitigation and Follow-up Actions** In response to this incident, the Squiz Data Centre team is working on the below: 1. Update configuration settings in customer VM’s for better availability and reliability of network routing paths. 2. Fix any observed errors on network switches in the ASH data centre.

Services have been restored for all the impacted customers. We apologise for this degradation of service and thank you for your patience. A postmortem will be provided via https://status.squiz.cloud.

Services are restored for most of the impacted Squiz cloud customers hosted in the ASH data centre. Mitigation actions continue for the remaining affected customers.

Services have been restored for most of the impacted Squiz cloud customers hosted in the ASH data centre. Our teams are working to get the service restored for the remaining customers and in parallel continue to closely monitor the environment for any anomalies.

The service issue in the ASH data centre has been rectified and services are being restored. We are continuing to monitor the situation closely to ensure services remain fully operational.

The Squiz Data Centre team have identified the issue and are applying a fix. In parallel, we continue to monitor the situation closely.

Squiz monitoring has detected a degradation of service incident that is affecting some Squiz cloud customers hosted in our ASH Data Centre. Multiple Squiz teams are currently investigating.

## **Executive Summary** On 02nd of April 2023 at ~10:05 AEST, Squiz internal monitoring started generating alerts indicating a degradation of service for some of our Squiz Edge customers. Customers may have experienced an “SSL error” message upon accessing their website and/or upon performing a search. Performed investigations by the Squiz Technical Engineering team identified issues with an internal service on some of the customer Virtual Machines \(VM’s\). This service was restarted, promoting a complete recovery at ~11:30 AEST on the 02nd of April 2023. ‌ ## **Customer Impact** Between ~10:05 AEST and ~11:30 AEST on the 02nd of April 2023, some Squiz Edge customers may have experienced an “SSL error” message upon accessing their website and/or upon performing a search. ‌ ## **Root Cause** An internal backend service that is responsible for keeping Matrix and Funnelback systems up and running was indicating “down” for some Squiz Edge customers. This service was restarted, promoting recovery.  ‌ ## **Mitigation and Follow-up Actions** In response to this incident, the Squiz Technical Engineering team is working to improve the way this service is set up for our customer servers. This will have a significant improvement when we upload SSL certificates on our customer’s website.

Services have been restored for our impacted customers. We apologise for this degradation of service and thank you for your patience. A postmortem will be provided via https://status.squiz.cloud within 48 hours.

A fix has been implemented and we are monitoring the results.

Squiz teams have identified the cause of the current degradation of service Major Incident, and are currently working to resolve the issue. We will provide a further update in ~15 minutes, or once the incident is resolved.

Squiz teams continue to investigate degradation of service for some Squiz Cloud customers. A further update will be provided in ~15 minutes.

Squiz monitoring has detected a degradation of service incident that is affecting some Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided in ~15 minutes.

## **Executive Summary** On the 22nd of March 20223, Squiz received notification of an ongoing malicious campaign against Australian government and private organisations. Squiz confirmed we have the detection capability, and mitigation strategies in place if an event should occur. ‌ On 25th of March 2023 at ~19:49 AEDT, Squiz internal monitoring started generating alerts indicating a degradation of service / slow response times for some of our Squiz Edge customers. Additionally, some customers may also have experienced a "500 Internal Server Error” message upon accessing their sites.  ‌ After a thorough investigation conducted by our team of experienced System Engineers, we detected abnormal traffic patterns and promptly initiated remedial actions. We immediately implemented a block on suspicious traffic in conjunction with our automated DDoS protection system from our trusted DDoS mitigation provider. ‌ While carrying out the mitigation process, our Squiz Security team concluded that the suspicious traffic was directed only to a small number of Squiz customers.. The high volume of concurrent requests resulted in performance degradation of our Squiz Edge solution. Once we successfully implemented our mitigation measures, our systems regained their normal performance levels. ‌ We continued to monitor the situation very closely and declared the incident as resolved at 01:23 AEDT on the 28th of March 2023. ‌ Intelligence gathering and assessment done by our security team found that this was a targeted attack against a larger group of Australian infrastructure and facilities, and not directed at Squiz itself.  Our system and client systems remain uncompromised as the incident was restricted to a Distributed Denial of Service \(DDoS\) attack. No unauthorised access or data breaches have occurred, and the security of our infrastructure remains intact. ‌ ‌ ## **Customer Impact** During the attacks, customers may have encountered a temporary decline in service quality, such as slow response times or difficulty accessing their websites. In some cases, they might have seen a "500 Internal Server Error" message for brief periods of time. ‌ ‌ ## **Root Cause** A Distributed Denial of Service \(DDoS\) attack was launched against numerous Australian websites, including several hosted by Squiz, resulting in substantial load on Squiz's underlying infrastructure. This activity has been associated with a hacktivist group targeting multiple Australian organisations across numerous sectors. ‌ ‌ ## **Mitigation and Follow-up Actions** In response to the ongoing traffic anomalies, Squiz Security and Hosting teams have collaborated to implement additional security measures.  ‌ At Squiz, we take security very seriously, and our core security capability includes DDoS protections that are routinely deployed to mitigate such attacks. In this instance, our DDoS protection measures functioned as intended, and our impacted customers were able to resume normal operations. ‌ Our teams remain vigilant and are closely monitoring the situation to detect any future disruptions to our customers' websites. ‌ Should you require any further information, please reach out to your Account Manager.

We are pleased to confirm that the previously reported issue affecting the performance of our Edge system has been successfully resolved. Our team implemented corrective measures and closely monitored the situation, which led to significant improvements and the cessation of alerts. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

We are presently addressing a detected problem affecting the performance of our Edge system. Measures have been implemented, and we are noticing improvements. For now, alerts have ceased, and we are keeping a close eye on the situation.

We are currently addressing an identified issue impacting our Edge system performance. Steps have been taken and we are observing improvements. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes

We are currently addressing an identified issue impacting our Edge system performance. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Service is now fully restored. Squiz is continuing to closely monitor for issues.

We are seeing all our services up and healthy. Squiz is still monitoring the situation closely. Some AU clients may still be seeing degradation in performance.

Squiz services have been restored. Squiz is treating the incident as on-going and is continuing to monitor the situation. Note some AU customers may still be seeing degradation in performance.

Squiz monitoring has detected the issue has returned. Squiz currently investigating the situation

Squiz services are now restored and we are currently closely monitoring the issue. More information on this issue will be available in the coming days via a Post Incident Report.

A fix has been implemented and we are monitoring the results.

We are currently addressing an identified issue impacting our Edge system performance. Multiple Squiz teams are currently investigating.

We are currently addressing an identified issue impacting our Edge system performance.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. We are responding to a monitored event.

We are pleased to confirm that the previously reported issue affecting our system has been successfully resolved. Our team implemented corrective measures and closely monitored the situation, which led to significant improvements and the cessation of alerts. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

Squiz is seeing the majority of customers impacted by the incident as now resolved and we are continuing to monitoring the situation.

A fix has been implemented and we are monitoring the results.

Squiz has applied mitigating steps and services are now beginning recover. Squiz is continuing to monitoring the situation.

We are currently addressing an identified issue impacting our Edge system performance.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating.

We are pleased to confirm that the previously reported issue affecting the performance of our Edge system has been successfully resolved. Our team implemented corrective measures and closely monitored the situation, which led to significant improvements and the cessation of alerts. We will continue to keep a watchful eye on the system to ensure optimal performance and stability. We appreciate your patience and understanding during this time and apologise for any inconvenience caused.

We are presently addressing a detected problem affecting the performance of our Edge system. Measures have been implemented, and we are noticing improvements. For now, alerts have ceased, and we are keeping a close eye on the situation.

We are currently addressing an identified issue impacting our Edge system performance. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

Squiz monitoring has detected a degradation of service incident that is affecting Squiz Cloud customers. Multiple Squiz teams are currently investigating. A further update will be provided via https://status.squiz.cloud in 15 minutes, or earlier if the situation or information changes.

We are continuing to investigate this issue.

We are currently investigating issues at the Sydney datacenter

Services have been restored for our impacted customers. We apologise for this degradation of service and thank you for your patience. The Squiz Technical Engineering team continues to monitor the environment.

Squiz teams continue to investigate the degradation of service affecting some AU customers. We will provide with further updates once more information becomes available.

Some AU customers have reported a degradation of service incident that is affecting Matrix admin backend and/or generating timeouts upon logging to the intranet instance. Multiple Squiz teams are currently investigating.

Investigations performed have indicated an issue with our external provider. A workaround has been implemented and conducted tests are successful. Our support staff are able to receive calls via the Squiz (Australia) support number +61 1300 077 849 . The Squiz Infrastructure team continues to monitor the environment.

Dear Customers, We are experiencing an issue with receiving calls via the Squiz (Australia) support number +61 1300 077 849. Our Infrastructure team is currently investigating this issue. Work Around We are reachable via other support numbers (+64) 4 805 0098 from NZ , (+44) 808 267 7849 from UK / Europe , (+1) 877 394 0640 from the US. Alternatively, please reach out to us via MySquiz or via email at support@squiz.net . We apologise for any inconvenience caused.

#### **Executive Summary** At ~11:02 AEDT on the 8th of March 2023, Squiz internal monitoring generated SSL alerts for some Australia/New Zealand customers using [clients.squiz.net](http://clients.squiz.net) subdomains. Impacted customers may have received an SSL error message upon accessing their site or upon performing a search. ‌ The Squiz Technical Engineering team renewed the expired certificates used by Squiz Edge to communicate with web servers, followed by a restart of Openresty service, promoting recovery at ~11:30 AEDT on the 8th of March 2023.   #### **Root Cause** SSL certificates for some Squiz Edge customers using [clients.squiz.net](http://clients.squiz.net) subdomains had expired.  ‌ #### **Mitigation and Follow-up Actions** To prevent future recurrences, the Squiz Technical Engineering team will examine and update the procedures and processes involved with these internal certificate renewals.

Squiz monitoring detected an issue leading to a degradation of services for some customer endpoints. This service issue has now been resolved. A postmortem will be provided via https://status.squiz.cloud .

Squiz monitoring has detected a degradation of service affecting some customers. Squiz Technical Engineering team is currently investigating.