Qwilr

Is Qwilr Down Right Now? Check if there is a current outage ongoing.

Qwilr is currently Operational

Last checked from Qwilr's official status page

Historical record of incidents for Qwilr

Report: "Service Degradation - Cloudflare Outage"

Last update
resolved

Cloudflare, a network and security service provider to Qwilr, experienced intermittent failures for some of their services. This had a flow on effect that degraded some of the Qwilr Platform.

Report: "Published Qwilr Pages outage"

Last update
resolved

The cause has been identified, and a fix deployed.

investigating

We are currently investigating an issue with published Qwilr pages experiencing an outage.

Report: "Increase in PDF generation errors"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Some service degradation"

Last update
resolved

This incident has been resolved.

monitoring

Services are recovering and appears that everything is operational again.

identified

The issue has been identified and we are testing solutions.

investigating

We are currently investigating a degradation in a few in-app features. Though editing and published pages are unaffected.

Report: "Outage in main qwilr app detected"

Last update
resolved

A deployment issue triggered a configuration mismatch in production. We've released a fix and should now be using the correct values.

identified

We've identified the issue and should have a fix rolling out soon

investigating

We are currently investigating this issue.

Report: "Delayed notifications and automations"

Last update
resolved

The issue causing notifications and automations to be delayed has been resolved so that this will not happen in the future.

identified

The issue has been identified, and a fix is being investigated

investigating

We are currently investigating the issue

Report: "Outage with Qwilr app and Qwilr pages"

Last update
resolved

The Qwilr app and Qwilr pages are now operating as usual. This incident is resolved.

investigating

We are still experiencing an outage with connections to the Qwilr app and Qwilr pages. We are continuing to investigate the issue.

investigating

We are continuing to investigate this issue.

investigating

We are experiencing a partial outage with connections to the Qwilr app and Qwilr pages. We are currently investigating this issue.

Report: "Search experience is degraded"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we're monitoring the results. Any new changes made to Qwilr pages and blocks will show up in the search now. Some of the changes that were made in the last 12 hours may not appear in the search yet.

identified

We are implementing a fix currently for this and expect it to be fully resolved for new pages and blocks within the next few hours.

investigating

Dashboard search is currently experiencing issues. Newly added pages and blocks are not being indexed and available in your results. Our team is investigating.

Report: "Qwilr custom domain page links unavailable"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Customers using custom domains will see their Qwilr public page links not working. Our team is currently investigating the issue.

Report: "Search results for some content aren't displaying"

Last update
resolved

This incident has been resolved.

monitoring

Our team has released a fix and all edits are appearing in the dashboard searches now. We are monitoring the results.

identified

Our team is working on implementing a fix for search results. We will update this page as soon as it's live.

identified

Edits made to new Qwilr Pages, new blocks, and edited blocks within the last 12 hours are not currently appearing in dashboard searches, but the edits themselves are saving normally. Our team is working on this now. This fix could take some time, but we will update this page as soon as the issue is resolved. Existing Qwilr Pages will show up normally in searches.

Report: "Qwilr public pages unavailable"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been released for this. We're still closely monitoring.

investigating

We are continuing to investigate this issue.

investigating

We've still investigating the causes of these issues. We'll update as soon as we have a solution.

Report: "Service disruption to Qwilr"

Last update
resolved

This incident has been resolved.

monitoring

System has appeared to recover and we're continuing investigation into root cause for the unavailability

investigating

An issue affecting Qwilr services availability has been detected and we're working to resolve

Report: "Hubspot integration degraded experience"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and deployed. We are now monitoring.

investigating

Issues have been reported with our Hubspot integration for users of Google Chrome and Chromium-based browsers. Users are currently unable to create Qwilr pages from Hubspot in those browsers. As a workaround, we'd recommend using other browsers like Firefox. For Safari and Brave, you'll need to enable third-party cookies for the integration to work.

Report: "Hubspot integration degraded experience"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and deployed. We are now monitoring

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified, and a fix implemented. The fix will be released in the upcoming days. As a workaround in the meantine, please log into the main Qwilr app directly and try to create the page again from HubSpot.

investigating

Some users are experiencing issues when trying to create a Qwilr page from our Hubspot integration app. We're still investigating the causes of this issue with Hubspot. We'll update as soon as we have a solution. As a temporary workaround, please ensure that you are logged into the main Qwilr app and try to create the Qwilr page from Hubspot again.

Report: "Partial outage of some Qwilr services"

Last update
resolved

A problem with our internal server routing was identified and fixed. All systems are back up and operational!

investigating

We are currently investigating a partial outage that is affecting: * The public API service * Public pages using custom domains

Report: "Investigating outage"

Last update
resolved

This incident has been resolved.

monitoring

It appears that some additional load on our database caused degraded performance. We are investigating this to determine the root cause, but are not currently seeing issues in app performance.

investigating

We are currently investigating this issue.

Report: "Very slow app response times"

Last update
resolved

Service should be functioning as normal again.

monitoring

We identified a bottleneck in our database disk resources, and have allocated more resources to remove the bottleneck. We will continue to closely monitor the database for any further problems

investigating

We are continuing to investigate this issue.

investigating

We have identified an issue that is causing the app to be very slow, and some users may not be able to log in. The team is urgently working on a fix now.

Report: "Outage of main app"

Last update
resolved

App appears to be functioning as normal again.

monitoring

Some long running db queries were preventing normal performance within the app, resulting in the degraded performance. We have terminated the queries and are continuing investigation into root cause, which appears to be some code changes added in the rolled back changes. Not expecting any further degradation.

investigating

Continuing investigation. It appears that Qwilr is accessible although we are still experiencing some degraded performance in the app.

investigating

Some additional issues have been identified causing much slower requests than usual. Proceeding with further investigations.

monitoring

It appears that a recent deployment caused some outage of the app where the servers became unresponsive. We are currently doing a deeper investigation but rolling back to an earlier version appears to have brought the server connectivity back online.

investigating

Monitoring indicates inaccessibility to the app and public pages. Currently investigating the cause of the issue.

Report: "Login issues affecting some customers"

Last update
resolved

AWS is reporting that the server issues are resolved and everything appears to be stable. If you're experiencing any issues, please reach out to help@qwilr.com

monitoring

AWS is experiencing intermittent DNS errors, which is blocking login for some customers. AWS is working to resolve this now. We are monitoring the situation.

Report: "Server issues affecting Qwilr.com"

Last update
resolved

This incident has been resolved. If you run into any issues, please email us at help@qwilr.com

monitoring

The server issue have been resolved. We will continue to monitor the situation.

investigating

It looks like we're currently having some server issues affecting Qwilr.com. We're investigating the cause. The Qwilr App and qwilr pages are working normally.

Report: "Degraded response times"

Last update
resolved

This incident has been resolved.

monitoring

A number of our servers became unresponsive which caused a degradation of our service. The issue has been fixed and we are currently monitoring the situation but don't expect further issues.

Report: "Server issues affecting the Qwilr app"

Last update
postmortem

On Tuesday June 11th at approximately 10.30am AEST, Qwilr experienced serious issues with site reliability and many users experienced failures in using the application and delivery of content to customers. Qwilr’s engineering team investigated the issue and observed spikes in CPU on some of our webserver instances, but nothing that should cause the 502 and 504 errors customers reported. Eventually we could observe that some of our NodeJS docker Pods \(we run in Kubernetes\) were hitting 100% CPU and with further investigation could see that these processes were taking up to 30 minutes to process a single request. The cause of this turned out to be a very large payload sent to our API, causing that request to take up to 30 minutes. Part of this was a result of having code that was designed to run fast for small payloads but didn’t handle this large payload. It filled up the memory allocated to the Pod and caused the CPU to go to 100%. Combined with this, as a consequence of recently moving infrastructure from Rackspace to AWS, our Kubernetes Pods lacked readiness checks that would ensure traffic not be routed to them when not responsive. This meant requests to these Pods would time out and return 502 or 504s. By 6pm AEST on the 11th, we deployed a code fix to resolve the root cause and ensure that these Pods could process such a large payload in approximately 1/10th of the time and also set up a readiness check to ensure our system is more robust. We are also working with our API customers to find a sensible limit to payload sizes. As a result of this issue we are confident that our system has been made more stable and resilient for the future.

resolved

The server issue has been resolved.

identified

We are continuing to work on a fix for this issue.

identified

We're currently experiencing server issues, and we're investigating. You may see errors loading the app or slow load times.

monitoring

The server issue have been resolved. We will continue to monitor the situation.

identified

The server issue have been resolved. We will continue to monitor the situation.

investigating

It looks like we're currently having some server issues and we're investigating the cause. You may see errors loading the app or slow load times.

Report: "Server issues affecting the Qwilr app"

Last update
resolved

Everything has been stable for the last day and we're not expecting any further problems now that the root cause has been fixed. If you do see any issues with your Qwilr account, reach out to help@qwilr.com

monitoring

We are continuing to monitor for any further issues.

monitoring

The server issue has been resolved and a permanent fix has been implemented. We will continue to monitor the situation.

identified

We've identified the core problem causing the server issues over the last few days, and are working on a fix.

investigating

It looks like we're currently having some server issues again and we're investigating the cause. You may see errors loading the app or slow load times.

Report: "Server issues affecting the Qwilr app"

Last update
resolved

The server issue has been resolved.

investigating

It looks like we're currently having some server issues and we're investigating the cause. You may see errors loading the app or slow load times.

Report: "Service restored"

Last update
resolved

The server issues have been resolved.

Report: "Service Restored, Monitoring"

Last update
resolved

Service has been restored.

monitoring

We are continuing to monitor for any further issues.

monitoring

The server issue have been resolved. We will continue to monitor the situation.

Report: "App experiencing networking issues"

Last update
resolved

This incident has been resolved.

monitoring

The networking issues have been resolved. We will continue to monitor the situation.

investigating

Network requests to the Qwilr servers are intermittently failing, causing an unstable user experience in the app. We are investigating the cause of this

Report: "Very slow app response times"

Last update
resolved

This incident has been resolved.

monitoring

The problematic cloud servers have been removed, and app response times have stabilised. We will continue to monitor performance and add more cloud resources as necessary

identified

We are continuing to work on a fix for this issue.

identified

We have identified the issue with the cloud servers and are implementing a fix

investigating

We are currently investigating an issue with our cloud provider. Load times in the app will be quite slow until this is resolved

Report: "Intermittent errors in the Qwilr app"

Last update
resolved

Everything is looking good now. Reach out to our team at help@qwilr.com if you're continuing to see any issues.

monitoring

Everything is still looking good, but we're continuing to monitor to ensure it stays that way.

monitoring

We've made some updates and everything should be loading fine now. We're continuing to monitor the situation.

identified

We've tracked down the cause and are making some changes to improve performance.

investigating

We're still investigating the cause of the errors. They're intermittent so Qwilr should still be working, but if you're seeing them try refreshing the screen.

investigating

We're currently seeing intermittent loading or "something went wrong" errors in the app. Our team is investigating the cause.

Report: "Maintenance being performed on the website"

Last update
resolved

This incident has been resolved.

investigating

The qwilr.com website is currently having a security upgrade and will be intermittently down. The published pages, and the app itself, are unaffected.

Report: "app.qwilr.com isn't currently loading"

Last update
resolved

Everything is stable. Get in touch with our team at help@qwilr.com if you're still seeing any issues.

monitoring

Service has been restored. We're still keeping an eye on things to make sure all is well.

identified

We've narrowed down the cause of the loading issues to a recent code deploy. We're rolling back those changes and the app should be back up in a few minutes.

investigating

We're seeing reports of Qwilr not loading and we're currently investigating.

Report: "Qwilr login page is down"

Last update
resolved

This incident has been resolved.

monitoring

The qwilr.com website is back up :) We will continue to monitor the situation and diagnose exactly what went wrong.

investigating

We are currently working to get the website back online. If you have previously logged in, you can still browse directly to https://app.qwilr.com to use the app

Report: "Can't open error when editing your Qwilr Pages"

Last update
resolved

We've identified the issue and projects should be loading again. Please get in touch with our team (help@qwilr.com) if they're not.

investigating

We're currently seeing an issue where a "Can't Open Project" error is displayed while trying to edit your Qwilr Pages. Our team is investigating.

Report: "Servers are experiencing slow response times"

Last update
resolved

There was an issue with a particular database field not being indexed, resulting in long query times. The field is now indexed and response times are back to normal

investigating

Users may experience pages loading slowly. We are currently investigating a fix to get response times back to normal

Report: "Servers not available."

Last update
resolved

This incident has been resolved.

monitoring

Service has been restored and we're monitoring the database to make sure everything is stable.

identified

Update: We have identified the issue and it is being caused by a third party service we rely on. We are working with their team to fix the issue.

investigating

Some customers are unable to access their account. We’re aware of the issue and are working on it urgently. We’re really sorry to be holding you up today! Please know our engineering team is working hard to get everything up and running and we will update you ASAP with the latest information.

investigating

Some customers are unable to access their account. We’re aware of the issue and are working on it urgently. We’re really sorry to be holding you up today! Please know our engineering team is working hard to get everything up and running and we will update you ASAP with the latest information.

Report: "Some pages are showing an error on opening"

Last update
resolved

This incident has been resolved.

monitoring

The fix has finished deploying to all the servers. We will continue to monitor the situation - if you are still experiencing issues opening projects, please let us know through our support channels!

identified

A hotfix is in the deployment pipeline now

identified

The problem appears to be isolated to non-admin users. A fix has been identified and is being worked on.

identified

We're looking into an issue where some projects are not able to be opened for editing. We are working on a fix now

Report: "Unable to accept quotes"

Last update
resolved

All the servers have been updated with the fix

identified

A fix has been implemented and is in the deployment pipeline

identified

A bug has been identified where the 'accept' button is not functioning. We've identified the cause and are working on a fix

Report: "Troubles loading Qwilr"

Last update
resolved

Everything is stable and working well now. Please get in touch at help@qwilr.com if you're still experiencing any problems.

monitoring

Service has been restored and we're monitoring the database to make sure everything is stable.

identified

We've identified the cause and we're working with the database provider to restore service to that database. We don't have an ETA yet, but we're getting closer.

investigating

We've identified some database issues, but we don't yet have an ETA for a fix. We're still working to identify the right course of action.

investigating

We're still investigating and don't have an ETA at this time.

investigating

Some customers are seeing an error loading a qwilr or while logging in. We're investigating the cause.

Report: "Media browser currently isn't allowing you to browse or search for new images"

Last update
resolved

The Unsplash servers are back online.

investigating

Our integration with Unsplash is currently offline as they're experiencing issues. We're monitoring the problem and waiting for their servers to come back online. Images already in use in your projects will remain visible, but while creating new projects you won't be able to browse the Unsplash servers for new images

Report: "Media browser currently isn't allowing you to browse or search for new images"

Last update
resolved

The issues with Unsplash appear to be resolved.

monitoring

The Unsplash servers are recovering now and the search, browse, and shuffle functions in the media browser are loading images, but a bit slower than normal. We're keeping in touch with Unsplash and monitoring the situation.

identified

Our integration with Unsplash is currently offline as they're experiencing issues. We're monitoring the problem and waiting for their servers to come back online. Images already in use in your projects will remain visible, but while creating new projects you won't be able to browse the Unsplash servers for new images.

Report: "Errors while saving or logging in"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

monitoring

We've pushed a fix for this issue live. Anyone experiencing errors will want to refresh their browser and that should clear them up. We're keeping an eye on everything to ensure no more errors pop up. If you're still seeing errors after a refresh please contact help@qwilr.com.

identified

The issue has been identified and a fix is being implemented.

identified

We've been able to identify the cause of these issues. It's related to cached versions of the app being returned when editing projects for a small subset of customers. We're working on a solution.

investigating

The error messages are unfortunately still persisting. We're still investigating the cause but don't yet have an ETA.

investigating

We're still investigating the cause of the errors while editing, but the errors while logging in seem to be clearing up. Please contact our support team (help@qwilr.com) if you're still seeing an error while logging in.

investigating

We're seeing reports of errors while saving content or when trying to log in. Our team is investigating the cause.

Report: "Custom styles and quote block rendering issues"

Last update
resolved

This incident has been resolved.

monitoring

The database restoration is complete, but we are keeping a close eye on any further issues. Contact us if you are still experiencing problems with your qwilr pages

investigating

A solution has been found and tested. A database update is currently rolling out that will restore affected project and block settings to their original state

investigating

Custom styles and quote block settings are not being applied to blocks, causing rendering issues on qwilr pages. We are urgently working on restoring functionality

Report: "PDF Export Issues"

Last update
resolved

This incident has been resolved.

investigating

We are currently having PDF export issues for certain clients. Our engineering team is investigating this now.