Historical record of incidents for Qwilr
Report: "Service Degradation - Cloudflare Outage"
Last updateCloudflare, a network and security service provider to Qwilr, experienced intermittent failures for some of their services. This had a flow on effect that degraded some of the Qwilr Platform.
Report: "Published Qwilr Pages outage"
Last updateThe cause has been identified, and a fix deployed.
We are currently investigating an issue with published Qwilr pages experiencing an outage.
Report: "Increase in PDF generation errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Some service degradation"
Last updateThis incident has been resolved.
Services are recovering and appears that everything is operational again.
The issue has been identified and we are testing solutions.
We are currently investigating a degradation in a few in-app features. Though editing and published pages are unaffected.
Report: "Outage in main qwilr app detected"
Last updateA deployment issue triggered a configuration mismatch in production. We've released a fix and should now be using the correct values.
We've identified the issue and should have a fix rolling out soon
We are currently investigating this issue.
Report: "Delayed notifications and automations"
Last updateThe issue causing notifications and automations to be delayed has been resolved so that this will not happen in the future.
The issue has been identified, and a fix is being investigated
We are currently investigating the issue
Report: "Outage with Qwilr app and Qwilr pages"
Last updateThe Qwilr app and Qwilr pages are now operating as usual. This incident is resolved.
We are still experiencing an outage with connections to the Qwilr app and Qwilr pages. We are continuing to investigate the issue.
We are continuing to investigate this issue.
We are experiencing a partial outage with connections to the Qwilr app and Qwilr pages. We are currently investigating this issue.
Report: "Search experience is degraded"
Last updateThis incident has been resolved.
A fix has been implemented and we're monitoring the results. Any new changes made to Qwilr pages and blocks will show up in the search now. Some of the changes that were made in the last 12 hours may not appear in the search yet.
We are implementing a fix currently for this and expect it to be fully resolved for new pages and blocks within the next few hours.
Dashboard search is currently experiencing issues. Newly added pages and blocks are not being indexed and available in your results. Our team is investigating.
Report: "Qwilr custom domain page links unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Customers using custom domains will see their Qwilr public page links not working. Our team is currently investigating the issue.
Report: "Search results for some content aren't displaying"
Last updateThis incident has been resolved.
Our team has released a fix and all edits are appearing in the dashboard searches now. We are monitoring the results.
Our team is working on implementing a fix for search results. We will update this page as soon as it's live.
Edits made to new Qwilr Pages, new blocks, and edited blocks within the last 12 hours are not currently appearing in dashboard searches, but the edits themselves are saving normally. Our team is working on this now. This fix could take some time, but we will update this page as soon as the issue is resolved. Existing Qwilr Pages will show up normally in searches.
Report: "Qwilr public pages unavailable"
Last updateThis incident has been resolved.
A fix has been released for this. We're still closely monitoring.
We are continuing to investigate this issue.
We've still investigating the causes of these issues. We'll update as soon as we have a solution.
Report: "Service disruption to Qwilr"
Last updateThis incident has been resolved.
System has appeared to recover and we're continuing investigation into root cause for the unavailability
An issue affecting Qwilr services availability has been detected and we're working to resolve
Report: "Hubspot integration degraded experience"
Last updateThis incident has been resolved.
A fix has been implemented and deployed. We are now monitoring.
Issues have been reported with our Hubspot integration for users of Google Chrome and Chromium-based browsers. Users are currently unable to create Qwilr pages from Hubspot in those browsers. As a workaround, we'd recommend using other browsers like Firefox. For Safari and Brave, you'll need to enable third-party cookies for the integration to work.
Report: "Hubspot integration degraded experience"
Last updateThis incident has been resolved.
A fix has been implemented and deployed. We are now monitoring
We are continuing to work on a fix for this issue.
The issue has been identified, and a fix implemented. The fix will be released in the upcoming days. As a workaround in the meantine, please log into the main Qwilr app directly and try to create the page again from HubSpot.
Some users are experiencing issues when trying to create a Qwilr page from our Hubspot integration app. We're still investigating the causes of this issue with Hubspot. We'll update as soon as we have a solution. As a temporary workaround, please ensure that you are logged into the main Qwilr app and try to create the Qwilr page from Hubspot again.
Report: "Partial outage of some Qwilr services"
Last updateA problem with our internal server routing was identified and fixed. All systems are back up and operational!
We are currently investigating a partial outage that is affecting: * The public API service * Public pages using custom domains
Report: "Investigating outage"
Last updateThis incident has been resolved.
It appears that some additional load on our database caused degraded performance. We are investigating this to determine the root cause, but are not currently seeing issues in app performance.
We are currently investigating this issue.
Report: "Very slow app response times"
Last updateService should be functioning as normal again.
We identified a bottleneck in our database disk resources, and have allocated more resources to remove the bottleneck. We will continue to closely monitor the database for any further problems
We are continuing to investigate this issue.
We have identified an issue that is causing the app to be very slow, and some users may not be able to log in. The team is urgently working on a fix now.
Report: "Outage of main app"
Last updateApp appears to be functioning as normal again.
Some long running db queries were preventing normal performance within the app, resulting in the degraded performance. We have terminated the queries and are continuing investigation into root cause, which appears to be some code changes added in the rolled back changes. Not expecting any further degradation.
Continuing investigation. It appears that Qwilr is accessible although we are still experiencing some degraded performance in the app.
Some additional issues have been identified causing much slower requests than usual. Proceeding with further investigations.
It appears that a recent deployment caused some outage of the app where the servers became unresponsive. We are currently doing a deeper investigation but rolling back to an earlier version appears to have brought the server connectivity back online.
Monitoring indicates inaccessibility to the app and public pages. Currently investigating the cause of the issue.
Report: "Login issues affecting some customers"
Last updateAWS is reporting that the server issues are resolved and everything appears to be stable. If you're experiencing any issues, please reach out to help@qwilr.com
AWS is experiencing intermittent DNS errors, which is blocking login for some customers. AWS is working to resolve this now. We are monitoring the situation.
Report: "Server issues affecting Qwilr.com"
Last updateThis incident has been resolved. If you run into any issues, please email us at help@qwilr.com
The server issue have been resolved. We will continue to monitor the situation.
It looks like we're currently having some server issues affecting Qwilr.com. We're investigating the cause. The Qwilr App and qwilr pages are working normally.
Report: "Degraded response times"
Last updateThis incident has been resolved.
A number of our servers became unresponsive which caused a degradation of our service. The issue has been fixed and we are currently monitoring the situation but don't expect further issues.
Report: "Server issues affecting the Qwilr app"
Last updateOn Tuesday June 11th at approximately 10.30am AEST, Qwilr experienced serious issues with site reliability and many users experienced failures in using the application and delivery of content to customers. Qwilr’s engineering team investigated the issue and observed spikes in CPU on some of our webserver instances, but nothing that should cause the 502 and 504 errors customers reported. Eventually we could observe that some of our NodeJS docker Pods \(we run in Kubernetes\) were hitting 100% CPU and with further investigation could see that these processes were taking up to 30 minutes to process a single request. The cause of this turned out to be a very large payload sent to our API, causing that request to take up to 30 minutes. Part of this was a result of having code that was designed to run fast for small payloads but didn’t handle this large payload. It filled up the memory allocated to the Pod and caused the CPU to go to 100%. Combined with this, as a consequence of recently moving infrastructure from Rackspace to AWS, our Kubernetes Pods lacked readiness checks that would ensure traffic not be routed to them when not responsive. This meant requests to these Pods would time out and return 502 or 504s. By 6pm AEST on the 11th, we deployed a code fix to resolve the root cause and ensure that these Pods could process such a large payload in approximately 1/10th of the time and also set up a readiness check to ensure our system is more robust. We are also working with our API customers to find a sensible limit to payload sizes. As a result of this issue we are confident that our system has been made more stable and resilient for the future.
The server issue has been resolved.
We are continuing to work on a fix for this issue.
We're currently experiencing server issues, and we're investigating. You may see errors loading the app or slow load times.
The server issue have been resolved. We will continue to monitor the situation.
The server issue have been resolved. We will continue to monitor the situation.
It looks like we're currently having some server issues and we're investigating the cause. You may see errors loading the app or slow load times.
Report: "Server issues affecting the Qwilr app"
Last updateEverything has been stable for the last day and we're not expecting any further problems now that the root cause has been fixed. If you do see any issues with your Qwilr account, reach out to help@qwilr.com
We are continuing to monitor for any further issues.
The server issue has been resolved and a permanent fix has been implemented. We will continue to monitor the situation.
We've identified the core problem causing the server issues over the last few days, and are working on a fix.
It looks like we're currently having some server issues again and we're investigating the cause. You may see errors loading the app or slow load times.
Report: "Server issues affecting the Qwilr app"
Last updateThe server issue has been resolved.
It looks like we're currently having some server issues and we're investigating the cause. You may see errors loading the app or slow load times.
Report: "Service restored"
Last updateThe server issues have been resolved.
Report: "Service Restored, Monitoring"
Last updateService has been restored.
We are continuing to monitor for any further issues.
The server issue have been resolved. We will continue to monitor the situation.
Report: "App experiencing networking issues"
Last updateThis incident has been resolved.
The networking issues have been resolved. We will continue to monitor the situation.
Network requests to the Qwilr servers are intermittently failing, causing an unstable user experience in the app. We are investigating the cause of this
Report: "Very slow app response times"
Last updateThis incident has been resolved.
The problematic cloud servers have been removed, and app response times have stabilised. We will continue to monitor performance and add more cloud resources as necessary
We are continuing to work on a fix for this issue.
We have identified the issue with the cloud servers and are implementing a fix
We are currently investigating an issue with our cloud provider. Load times in the app will be quite slow until this is resolved
Report: "Intermittent errors in the Qwilr app"
Last updateEverything is looking good now. Reach out to our team at help@qwilr.com if you're continuing to see any issues.
Everything is still looking good, but we're continuing to monitor to ensure it stays that way.
We've made some updates and everything should be loading fine now. We're continuing to monitor the situation.
We've tracked down the cause and are making some changes to improve performance.
We're still investigating the cause of the errors. They're intermittent so Qwilr should still be working, but if you're seeing them try refreshing the screen.
We're currently seeing intermittent loading or "something went wrong" errors in the app. Our team is investigating the cause.
Report: "Maintenance being performed on the website"
Last updateThis incident has been resolved.
The qwilr.com website is currently having a security upgrade and will be intermittently down. The published pages, and the app itself, are unaffected.
Report: "app.qwilr.com isn't currently loading"
Last updateEverything is stable. Get in touch with our team at help@qwilr.com if you're still seeing any issues.
Service has been restored. We're still keeping an eye on things to make sure all is well.
We've narrowed down the cause of the loading issues to a recent code deploy. We're rolling back those changes and the app should be back up in a few minutes.
We're seeing reports of Qwilr not loading and we're currently investigating.
Report: "Qwilr login page is down"
Last updateThis incident has been resolved.
The qwilr.com website is back up :) We will continue to monitor the situation and diagnose exactly what went wrong.
We are currently working to get the website back online. If you have previously logged in, you can still browse directly to https://app.qwilr.com to use the app
Report: "Can't open error when editing your Qwilr Pages"
Last updateWe've identified the issue and projects should be loading again. Please get in touch with our team (help@qwilr.com) if they're not.
We're currently seeing an issue where a "Can't Open Project" error is displayed while trying to edit your Qwilr Pages. Our team is investigating.
Report: "Servers are experiencing slow response times"
Last updateThere was an issue with a particular database field not being indexed, resulting in long query times. The field is now indexed and response times are back to normal
Users may experience pages loading slowly. We are currently investigating a fix to get response times back to normal
Report: "Servers not available."
Last updateThis incident has been resolved.
Service has been restored and we're monitoring the database to make sure everything is stable.
Update: We have identified the issue and it is being caused by a third party service we rely on. We are working with their team to fix the issue.
Some customers are unable to access their account. We’re aware of the issue and are working on it urgently. We’re really sorry to be holding you up today! Please know our engineering team is working hard to get everything up and running and we will update you ASAP with the latest information.
Some customers are unable to access their account. We’re aware of the issue and are working on it urgently. We’re really sorry to be holding you up today! Please know our engineering team is working hard to get everything up and running and we will update you ASAP with the latest information.
Report: "Some pages are showing an error on opening"
Last updateThis incident has been resolved.
The fix has finished deploying to all the servers. We will continue to monitor the situation - if you are still experiencing issues opening projects, please let us know through our support channels!
A hotfix is in the deployment pipeline now
The problem appears to be isolated to non-admin users. A fix has been identified and is being worked on.
We're looking into an issue where some projects are not able to be opened for editing. We are working on a fix now
Report: "Unable to accept quotes"
Last updateAll the servers have been updated with the fix
A fix has been implemented and is in the deployment pipeline
A bug has been identified where the 'accept' button is not functioning. We've identified the cause and are working on a fix
Report: "Troubles loading Qwilr"
Last updateEverything is stable and working well now. Please get in touch at help@qwilr.com if you're still experiencing any problems.
Service has been restored and we're monitoring the database to make sure everything is stable.
We've identified the cause and we're working with the database provider to restore service to that database. We don't have an ETA yet, but we're getting closer.
We've identified some database issues, but we don't yet have an ETA for a fix. We're still working to identify the right course of action.
We're still investigating and don't have an ETA at this time.
Some customers are seeing an error loading a qwilr or while logging in. We're investigating the cause.
Report: "Media browser currently isn't allowing you to browse or search for new images"
Last updateThe Unsplash servers are back online.
Our integration with Unsplash is currently offline as they're experiencing issues. We're monitoring the problem and waiting for their servers to come back online. Images already in use in your projects will remain visible, but while creating new projects you won't be able to browse the Unsplash servers for new images
Report: "Media browser currently isn't allowing you to browse or search for new images"
Last updateThe issues with Unsplash appear to be resolved.
The Unsplash servers are recovering now and the search, browse, and shuffle functions in the media browser are loading images, but a bit slower than normal. We're keeping in touch with Unsplash and monitoring the situation.
Our integration with Unsplash is currently offline as they're experiencing issues. We're monitoring the problem and waiting for their servers to come back online. Images already in use in your projects will remain visible, but while creating new projects you won't be able to browse the Unsplash servers for new images.
Report: "Errors while saving or logging in"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've pushed a fix for this issue live. Anyone experiencing errors will want to refresh their browser and that should clear them up. We're keeping an eye on everything to ensure no more errors pop up. If you're still seeing errors after a refresh please contact help@qwilr.com.
The issue has been identified and a fix is being implemented.
We've been able to identify the cause of these issues. It's related to cached versions of the app being returned when editing projects for a small subset of customers. We're working on a solution.
The error messages are unfortunately still persisting. We're still investigating the cause but don't yet have an ETA.
We're still investigating the cause of the errors while editing, but the errors while logging in seem to be clearing up. Please contact our support team (help@qwilr.com) if you're still seeing an error while logging in.
We're seeing reports of errors while saving content or when trying to log in. Our team is investigating the cause.
Report: "Custom styles and quote block rendering issues"
Last updateThis incident has been resolved.
The database restoration is complete, but we are keeping a close eye on any further issues. Contact us if you are still experiencing problems with your qwilr pages
A solution has been found and tested. A database update is currently rolling out that will restore affected project and block settings to their original state
Custom styles and quote block settings are not being applied to blocks, causing rendering issues on qwilr pages. We are urgently working on restoring functionality
Report: "PDF Export Issues"
Last updateThis incident has been resolved.
We are currently having PDF export issues for certain clients. Our engineering team is investigating this now.