Is UserVoice Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we're monitoring. Please let us know if you continue to experience issues.

Thanks for your patience while we continue to investigate.

When creating a profile with Email & Password auth option, after receiving the verification email, when users click 'create account', the application loads continuously. We've identified the request to our email verification provider, Magic.link, is hanging and not responding. We're continuing to investigate the issue.

This incident has been resolved.

The issue was identified by engineering team and a fix has been deployed.

We are currently investigating an issue with Public Status Updates emailed to subscribed users via our Quick Actions panel.

This incident has been resolved.

CloudFlare has applied a fix and we are monitoring to confirm full performance and stability.

CloudFlare has identified the issue and is working on resolution: https://www.cloudflarestatus.com/incidents/55w67hz7kjw5

We are investigating an issue where some small portion of requests are failing to resolve.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating some issues with search on recently created suggestions.

This incident has been resolved.

UserVoice went down due to a MySQL crash at 11:52 AM EST. A fix has been implemented and we are monitoring.

This incident has been resolved.

We are continuing to monitor for any further issues.

CSS and javascript assets were failing to load causing intermittent blank pages and errors across the site. The error has been resolved and we are monitoring

This incident has been resolved.

We have received reports and are investigating with Microsoft a higher volume of UserVoice notifications with delayed delivery or being quarantined when the recipient uses Microsoft365 for Email.

A security hardening aspect of our cluster infrastructure inadvertently prevented a healthy restart of our load-balancing services. This caused a propogation failure that prevented traffic from being served. We have resolved the issue and are investigating additional testing resources to verify it can't reoccur.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

We have mitigated DB issue and are monitoring to confirm.

We are currently investigating an infrastructure issue causing errors and slow performance.

This incident has been resolved.

A fix has been implemented and we are confiming no existing issues remain.

We are continuing to investigate this issue.

We are currently investigating an issue resulting in a significantly increased error rate accessing the UserVoice application.

Functionality has been restored. Please contact support with any issues or concerns.

Searches for ideas are showing partial results. Ideas which have been recently created or updated are showing up in search results. We are back-filling the data and will update you when search functionality is completely restored.

This incident has been resolved.

Search has been fully recovered. Our engineers are currently verifying all search queries are behaving as expected.

Search for suggestions, tickets, and users is returning partial results as we recover from our outage. We will update you when search is again fully operational.

We are experiencing an issue where search is currently unavailable.

We deployed a major upgrade to our backend about 20 minutes ago. This upgrade was reverted after about 5 minutes due to extra cautious error detection. After reverting, the backend was having issues rendering its already generated cached pages. This has now been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified an issue with our upstream email provider which is delaying outgoing notifications for parts of our app.

There was an issue between 9:00-9:40 EST that would have seen new user sign ups and password resets fail to send verification emails and would have blocked this functionality. The issue has been fully resolved.

The cause was identified and the issue has been resolved. Issue: The email template configured for these types of notification emails had an error during editing. Fix: The template has been reverted back to a working version.

The following notification emails are affected (Legacy HelpDesk/Ticketing features are not impacted): - New Comment - Supporter Message - Feature Update - Suggestion AutoResponse - Public Status Update - Internal Status Update - Discovery Account Create - Validation Account Create - Validation Invitation - One:One Comms - Outreach The provider we use to send these emails is currently investigating the issue.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

The issue at cloudflare has now been resolved and all services are operational

We are seeing timeouts getting to the portal, admin console, and the api. This is due to an outage in one of our upstream providers. Will update shortly

This incident has been resolved.

Provider has recovered, and we are monitoring to ensure stability.

Our primary application e-mail provider (Mailgun) is currently experiencing a partial outage. This is causing delays in the processing of incoming and outbound UserVoice application emails. Emails are being cached and should not be lost with this incident.

After monitoring the fix, we have confirmed that the incident has been resolved and the service is fully operational.

We're continuing to monitor the fix. We're aware that this is still impacting a handful of customers. Please reach out to support (at) uservoice.com if this is still affecting your ticketing service.

We have implemented a fix for this issue from our side, and are monitoring for any additional reports. Please reach out to support (at) uservoice.com with any additional questions.

We have identified an issue where some ticketing emails are bouncing with an error message. Our email service provider is investigating a change on their end that has caused this problem. We will continue updates here on our status page throughout the day, and please write in to support(at)uservoice.com with any additional questions.

Full service has been restored and verified.

Thanks all for your patience. We have made a fix, and are monitoring the performance of the application. Please write in to support@uservoice.com with any questions.

We are currently experiencing an outage, will update shortly

Our provider has resolved their issues and all backlogged emails have been delivered.

Our email provider extensively uses AWS for their infastructure. Beginning earlier today, AWS started experiencing intermittent errors https://status.aws.amazon.com/ that are causing delays in the delivery of our Feedback related emails. Helpdesk emails appear not to be affected by this issue.

This incident has been resolved.

Our engineers have implemented a fix and are monitoring to verify that everything has appropriately recovered.

We've identified an issue regarding the creation of new users, which is causing performance issues and error messages in the admin console. We will continue to post updates today as they come available. Please reach out to support(at)uservoice.com if you have any questions.

Our testing is showing that this is resolved. Users will have to refresh the page to see the changes, but if you continue to experience any problems or have questions please reach out to support (at) uservoice.com.

We have applied a fix and are monitoring the results. Please reach out to support (at) uservoice.com if you are continuing to experience problems.

Our engineering team is investigating an issue where some users are unable to create new ideas directly from the portal. (Admin Portal/API/Widget/Sidebar idea entry is not impacted)

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Our engineering team is investigating an issue where some users are unable to create new ideas directly from the portal. (Admin Portal/API/Widget/Sidebar idea entry is not impacted)

All systems should be back and responsive now.

We have identified a linked issue regarding helpdesk ticket creation and are working to resolve.

Datastore is recovered, though we are operating with some additional latency as we reprocess older data. Engineers are continuing to monitor and attempting to scale up some additional resources to reduce measured latency.

Engineers are continuing to work to resolve corruption in a data store table to fully restore service.

Our engineers are continuing to work to repair the data store issue that is causing this issue.

We have been notified and identified an issue with our data store that is currently causing some production slowdowns and errors. Engineering are currently working to mitigate and resolve.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are experiencing an outage in a key component of our infrastructure. Will update you shortly

We have verified root cause of the incident and are implementing some testing changes to ensure such an incident does not reoccur.

We are currently investigating an issue that has caused some customers with their Language set to a Non-English variant to have difficulties getting the Admin Console to fully load. We have identified the code-change that initiated this issue, and have reverted it while we continue to investigate the root cause.

Cloudflare believes a bad router announcement caused massive backend failures on their system. This caused DNS to fail for multiple regions (explicitly North America). The issue has been fully resolved and tested.

CloudFlare appears to have restored service, resolving our downstream issues.

Our DNS and DDoS provider, CloudFlare is having a significant issue right now that is impacting our site availability. We are working with them to try and mitigate/resolve the issue.

We've resolved this issue. Thanks for your patience, and please write into support@uservoice.com if you have any questions.

The issue should be resolved, but we are monitoring for any lingering problems.

Errors would be present in both our admin console and API for the last 30 minutes. We have found the cause of the issue and are working on a fix. Thanks for your patience and we will keep you updated on our progress in further status updates.

UserVoice experienced slowness and unavailability for users attempting to access the UserVoice Site/Portal. API/Ticketing/Emails where unaffected.

We have resolved the issue. Please reach out to support@uservoice.com if you notice any issues or questions.

We've fixed performance issues and are monitoring the application.

We are currently investigating a database problem that is preventing suggestions and users from loading in the admin console. Thanks for your patience while we investigate the issue.

This issue is now resolved. Please reach out to support@uservoice.com if you have any questions.

Everything has caught up now, but we are continuing monitor the performance of search.

We have identified the issue and will provide another update about a fix later today.

We are investigating an issue that is preventing recently created ideas from appearing in search results. More updates will be posted as we continue our investigation. Thanks in advance for your patience, and please reach out to support@uservoice.com if you have any questions.

This incident has been resolved.

We identified an issue with admin licensing and permissions that would cause error messages regarding access to the admin console and specific feature sets. We have pushed a fix and are monitoring the results. Please reach out to us at support@uservoice.com if you are having any issues.

On August 5, 2019, from 5:49PM to 6:23PM EDT, the admin console and UserVoice API were down. **Business Impact** * Admins saw 503 errors in the admin console, or noticed excessive slowness when loading parts of the application. * The UserVoice API returned errors. * Users on the web portal or with the widgets may have briefly seen errors or slowness at the start of the incident, but only for a few moments. **Root Cause** * A configuration change during a routine Kubernetes Cluster upgrade caused an unexpected restart for key operational services. One of these services had trouble recovering, which caused the downtime. **What we are Doing to Prevent This** * Our team identified and fixed the issue that caused the trouble during restart. * Our team also identified the setting that triggered the unplanned update, and how best to ensure that does not happen again. We do apologize for the downtime. This interrupted your usage of UserVoice, and for that we truly are sorry. If you have any questions or concerns, please, don’t hesitate to reach out! Claire Talbott Support Manager claire.talbott@uservoice.com

We have pushed a fix for the issue and things are stable. Let us know at support@uservoice.com if you have any issues.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue that is causing 503 errors and loading issues in the admin console. We will keep you updated as we investigate further.

All services have been confirmed to be performing as expected.

Cloudflare has implemented a fix for the issue, and we are monitoring to ensure all services have been restored.

We are continuing to work on a fix for this issue.

We are observing issues related to availability of our services. It appears to be due to a Network Issue with our CDN provider CloudFlare. Affected users are reporting both timeouts and 502 Error Pages when attempting to use the service. CloudFlare and UserVoice Engineers are both investigating to mitigate and resolve. https://www.cloudflarestatus.com/

This incident has been resolved.

We believe network congestion issues have been mitigated, but have not received full confirmation or analysis form Google Engineering. We are continuing to monitor system performance.

The UserVoice system is encountering higher than normal error and lowered availability due to apparent issues with Google Cloud Network beginning at 11:13 US/Pacific. UserVoice and Google engineers are both working to fully resolve this issue.

This issue is resolved. We have implemented filters to catch the spam, and are seeing it successfully catch these emails. We have also cleaned up the emails that made it past our filters earlier today. We are also investigating ways to continue to update and improve our spam filters so we can address these types of spammers quicker and more efficiently. We do apologize for the pain point this caused for you and your team, and if you have any questions, please, don't hesitate to reach out!

We've updated our spam filters and new, incoming spam is being caught now as expected. We are also working to clean up the spam that made it past our filters last night. We will keep you updated on our progress.

We are currently investigating reports of customers who use our support tools getting a high amount of ticketing spam. Our engineers are working to address, and we will keep you updated through our status page.

This incident has been resolved.

We have observed a few instances of slow performance and failed api requests this morning. It appears to be related to a flapping backend service. We believe we have mitigated the issue and are monitoring as we continue to investigate root cause.

This incident has been resolved.

UserVoice appears to have been the target of a Distributed Denial of Service (DDoS) attack using the HTTP HEAD method. We have temporarily blocked this method while we investigate the source more fully. The application functionality should be fully restored at this time.

We are currently investigating reports of 500 errors. We will keep you updated as we work to get this resolved.

On November 13th between 13:38 and 14:14 PT, UserVoice experienced a networking infrastructure issue that caused a sitewide outage and system unavailability. **Business Impact** * During the outage end users and admins would have been unable to load or interact with UserVoice sites, widgets or the API. * Email would have been delayed, but no emails were lost. **Root Cause** In the process of cleaning up unused resources in the UserVoice infrastructure an old kubernetes cluster was removed from production. The automated cleanup of this cluster unintentionally removed a networking firewall rule that allowed our active application cluster to communicate with our backend infrastructure. Initial debugging was incorrectly focused around in-cluster symptoms and we did not immediately determine proper cause of the issue. Manual restoration of a proper firewall rule allowed the service to be fully restored. **What we are Doing to Prevent This** * Proper failover firewall rules are now being controlled via our infrastructure-as-code system preventing automated cleanup of old rules. * Infrastructure cleanup tasks will be scheduled during maintenance windows going forward. We didn’t meet our own or your expectations for using UserVoice with this outage. We do apologize for the pain points this caused for you and your team. If you have any questions or concerns, please reach out and let me know. Claire Talbott Support Manager claire.talbott@uservoice.com

Root cause of issue (an automated firewall rule that was automatically removed incorrectly) has been discovered and fixed. A full post-mortem will be forthcoming.

We are seeing the application back up and working again. We are monitoring things closely. Our engineers are still digging into the root cause, and we will keep you updated. If you use our support tools, incoming emails were delayed, but none were lost, and you will see those tickets being created over the next little bit.

We want to keep you all updated while we work to resolve this issue. The application is down, and we are all hands on deck to get this issue resolved and everything back up and working for you and your customers. We will post our next status update by 2:30PM PST.

We are investigating 503 errors being returned in the UserVoice admin console and on web portals.

This incident has been resolved.

The application is available and we are monitoring performance.

We are continuing to investigate this issue.

We are currently investigating unexpected downtime, resulting in 500 errors in the application. Thanks for your patience.

On Friday 11/2/18 from 5AM to 5:13AM PDT, UserVoice experienced downtime. ## Business Impact During the time of the incidents end users and admins would have seen 500 errors. They wouldn’t have been able to load the admin console, use the API, interact with ideas on the front end or use the widget or Contributor Sidebar. Email would have been delayed, but no emails were lost. ## Root Cause We saw an issue similar to last Friday’s incident where one of the servers in Uservoice's database cluster experienced an application stall event.  This caused a pause in database writes. Our engineering team manually removed the affected node to allow the cluster to resume operation. ## What we are Doing to Prevent This Our team has been focused, since last week, on finding the root issue that is caused one of our database clusters to stall. This work is still in progress. Once the root issue is identified, we will be implementing a fix and updating this report with the information discovered. In the meantime, we have put increased alerting in place so that should the issue repeat, we will identify it immediately. We understand UserVoice being down is an interruption to you and your team, and impacts your workflows! We take this downtime seriously, and are all hands on deck to get this issue fully addressed, so we can prevent it happening again. If you have any questions or feedback for us about the incident please don’t hesitate to contact me at [claire.talbott@uservoice.com](mailto:claire.talbott@uservoice.com). Claire Talbott Support Manager

This incident has been resolved.

A temporary fix has been implemented and we are monitoring the database cluster

On October 26th between 10:20 and 10:35 PDT UserVoice experienced an infrastructure issue that caused intermittent system unavailability. ## Business Impact During the outage end-users and admins may have been unable to load or interact with UserVoice sites or widgets. ## Root Cause One of the servers in Uservoice's database cluster experienced an application stall event.  Due to misconfiguration of our cluster, this caused a pause in database writes. Our engineers were required to manually remove the affected node to allow the cluster to resume writing. ## What we are Doing to Prevent This We have updated our database cluster configuration to more aggressively monitor and remove cluster members that are non-performant. This caused an interruption for your team in UserVoice and your end users trying to view and submit feedback. We are sorry for the pain point this caused. We have already put improvements in place to prevent this type of issue from happening again. If you have any questions, don’t hesitate to reach out at [claire.talbott@uservoice.com](mailto:claire.talbott@uservoice.com). Claire Talbott Support Manager

This incident has been resolved.

The system has recovered and operating normally. We'll be checking to determine root cause and follow with a post mortem.

We are continuing to investigate this issue.

We are investigating 520 and 502 errors in UserVoice admin consoles and on web portals.

On October 19th between 10:00 and 11:30 PDT UserVoice experienced two approximately 10 minute infrastructure outages that caused site-wide outages and system unavailability. **Business Impact** During the outage end users and admins would have been unable to load or interact with UserVoice sites or widgets. Email would have been delayed, but no emails were lost. **Root Cause** UserVoice uses an in-memory data-store cluster \(Redis\) to handle asynchronous job management and transient data storage.  A recent change to one of the libraries that use this service caused a very sudden increase in its usage. The sudden usage increase caused a system failure and prevented failover to like-sized standby services. **What we are Doing to Prevent This** * Increased sizing of our Redis cluster and added additional alerting to allow us to more quickly detect usage spikes * Fixed the library that wasn’t properly interacting with Redis

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating 503 errors on UserVoice web portals and the admin console. This also affects the API and widgets.

This incident has been resolved

We are seeing everything working as expected again, but will continue to monitor closely.

We are continuing to investigate this issue.

We are investigating database issues, and will keep you updated as we work to resolve.

This incident has been resolved.

A fix has been implemented and we are monitoring the results

We are currently investigating reports of 500 errors showing in the UserVoice admin console and web portals.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are investigating reports of tickets not loading in the admin console for some accounts.

All systems have recovered appropriately.

Our provider believes they have fixed their api issues. We are observing expected volumes of application email being processed. We are continuing to monitor to ensure this is a full provider recovery.

Our email provider is currently experiencing api issues that are causing delays in processing inbound emails and delivering outbound emails.