Groove

Is Groove Down Right Now? Check if there is a current outage ongoing.

Groove is currently Operational

Last checked from Groove's official status page

Historical record of incidents for Groove

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results - backlog is currently clearing over the next 30-60 minutes. Expected delays for incoming mails, outgoing mails being sent, rule processing, realtime, and other features during this time.

identified

The issue has been identified and a fix is being implemented.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"

Last update
resolved

This incident has been resolved.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"

Last update
Resolved

This incident has been resolved.

Monitoring

A fix has been implemented and we are monitoring the results - backlog is currently clearing over the next 30-60 minutes.Expected delays for incoming mails, outgoing mails being sent, rule processing, realtime, and other features during this time.

Identified

The issue has been identified and a fix is being implemented.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"

Last update
resolved

We've identified and corrected a rogue process putting elevated load on the search cluster.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates)"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates)"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

investigating

We are currently investigating this issue.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates)"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

We have implemented a fix to address the issue. Our team is actively monitoring the results to ensure system stability and performance. We will continue to observe the situation closely and provide further updates as needed. Thank you for your patience.

Report: "Live Chat delayed messages"

Last update
resolved

We have successfully deployed a fix for the issue causing delays in some live chats. We want to reassure you that all delayed chats have successfully reached Groove and no messages were lost. Thank you for your patience. If you experience any further issues, please contact our support team.

identified

A fix for delayed messages from Live Chat was rolled out - we're seeing intermittent issues since deploying the fix. The teams are working on resolving the issues still being seen with delays for Live Chat messages. Note: no other areas of Groove are affected.

Report: "Live chat delays"

Last update
resolved

We have successfully deployed a fix for the issue causing delays in some live chats. We want to reassure you that all delayed chats have successfully reached Groove and no messages were lost. Thank you for your patience. If you experience any further issues, please contact our support team.

identified

We have identified an issue where some live chats are delayed. We are in the process of deploying a fix. This has no impact on any other channels or features within Groove. We will update this status page when a fix has been deployed.

Report: "Service outage"

Last update
resolved

An issue with one of Groove's database hosting providers caused a portion of Groove users to be unable to access Groove from certain devices for approximately 17 minutes. No data was lost during this time, and service has been completely restored for all Groove accounts.

Report: "Attachment issue - between 12:10 UTC and 4:10 UTC"

Last update
resolved

One of our upstream providers experienced an issue that corrupted some attachments before the mails reached Groove. This issue has been resolved. Some attachments can still be retrieved from your mail provider.

Report: "New slack connections restricted"

Last update
resolved

This incident has been resolved. Slack has re-approved our integration.

investigating

Our slack app is currently being reviewed for new scopes according to Slack's app policy. During this time, creating new Slack connections may be restricted.

Report: "Meta Outage - affecting Instagram and Messenger connectivity"

Last update
resolved

Meta (Facebook Messenger and Instagram) currently have an outage affecting API endpoints from connecting. Users may have disconnections from their Social Chats while the outage is happening. https://metastatus.com/

Report: "Degraded service on Groove inbox (delay on incoming tickets, logging in, live chat, ticket updates)"

Last update
resolved

Full service has resumed.

investigating

Users may see degraded performance in some areas in Groove.

Report: "Meta Messenger API Outage"

Last update
resolved

Meta has resolved the incident on their end. We are continuing to monitor any impact to affected Instagram Messenger users in Groove.

investigating

We are continuing to investigate this issue.

investigating

Meta is currently experiencing a major outage of the Instagram Messenger API. Unfortunately, no additional information has been provided by Meta regarding the cause or expected resolution time. While not all Instagram channels on Groove are affected, some are experiencing disruptions. Our team is actively monitoring the situation.

Report: "[Outage] Groove Web App Unavailable"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating this issue.

Report: "[Outage] Delays with processing outbound emails"

Last update
resolved

This incident has been resolved. Our email delivery partner inadvertently flagged our account for review this morning. Their action placed all outbound emails from Groove mailboxes that do not have custom email server settings and are not connected via Gmail into a temporary hold. All delayed emails have since been delivered. **To be clear, no emails were lost during this outage, they were simply delayed in their delivery.** We have been in contact with our email delivery partner regarding this incident and are awaiting addition information from them so that we can prevent this issue from occurring again.

identified

The issue has been identified and a fix is being implemented.

Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates)"

Last update
resolved

This incident has been resolved.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and a fix is being implemented.

Report: "Knowledge base display / slow loading issue"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Knowledge bases hosted by Groove may be experiencing slow loading or display issues. Our team is actively investigating the issue.

Report: "Connectivity issues"

Last update
resolved

We have completed the fix and the incident is resolved.

monitoring

We have identified the source of the issue and implemented a fix - Groove is back online and we're actively monitoring the situation.

investigating

We are currently experiencing major connectivity issues related to an Amazon Web Services issue. We are investigating ways of coming back online as soon as possible.

Report: "Service Outage"

Last update
resolved

This incident has been resolved.

monitoring

A solution has been implemented and we are monitoring the results.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are working with our upstream provider to resolve the issue as quickly as possible.

identified

We are continuing to work on a fix for this issue.

identified

We are in working with our upstream provider to resolve the issue as quickly as possible.

monitoring

A solution has been implemented and we are monitoring the results.

investigating

We are in working with our upstream provider to resolve the issue as quickly as possible.

investigating

We're continuing to monitor the elevated load and seeing more slow requests

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

We are seeing degraded performance with an upstream provider, which impacts our ability regarding server requests - we're currently investigating

Report: "Service slowdown"

Last update
postmortem

A subsystem for ensuring the accurate delivery of emails caused a gradual backup in job scheduling calls, which adversely affected the performance of our web workers.

resolved

This incident has been resolved.

investigating

A slowdown in service is currently being investigated. Groove may load slowly for customers at this time.

Report: "Groove slow to load in app and mobile."

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating an issue with speed to load with the groove app and mobile.

Report: "Groove connectivity issues"

Last update
resolved

This incident has been resolved.

identified

We are currently investigating an issue where mailboxes aren't loading.

Report: "Groove connectivity issues"

Last update
resolved

This incident has been resolved.

monitoring

We've identified the issue and deployed a fix

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "DNS outage causing Groove outage"

Last update
resolved

This incident has been resolved.

monitoring

Here is the status page to monitor the issue with the DNS provider: https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj You can subscribe for updates as well.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating an issue with our DNS provider outage.

Report: "Email receive/send issues related to Google Outage"

Last update
resolved

This incident has been resolved.

monitoring

Our monitoring reveals that the situation on Google's side is resolved, we are seeing mails coming in via Gmail API correctly. We will continue to monitor.

investigating

We are still seeing errors getting returned from the Gmail API, we continue to investigate for a possible solution

investigating

We are continuing to investigate this issue.

investigating

We are investigating an issue with the Google Suite of products, which is limiting our ability to retrieve and send email messages using the Gmail API. More information can be found at - https://www.google.com/appsstatus#hl=en-GB&v=status

Report: "Gmail connectivity issue"

Last update
resolved

This incident has been resolved.

identified

We are deploying a temporary workaround that will allow gmail syncing to continue while Google addresses the core problem on their servers.

identified

Google is currently experiencing a temporary api outage that affects Groove's ability to retrieve emails from gmail accounts. Google is addressing the issue internally, and a fix should be issued shortly. No messages will be lost, but you may experience delays in receiving messages in your Groove inbox during this time.

Report: "Connectivity issue"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Groove application is not functional because of connectivity issues - we are investigating.

Report: "Email processing delayed"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are backed up in processing incoming emails. No emails are lost. They will be processed once we are back online.

Report: "Server overload"

Last update
resolved

This incident has been resolved.

monitoring

We have resolved the issue and are monitoring the situation

investigating

We are continuing to investigate this issue.

investigating

We are investigating a case of server overload. More info to follow.

Report: "Outage with our upstream provider"

Last update
resolved

This incident has been resolved.

identified

We are experiencing a full outage due to issues with our upstream provider. Our team of engineers is working closely with them to bring our services up as soon as possible.

Report: "Partially degraded performance"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been deployed, and we are currently monitoring memory usage.

identified

A recent deploy has triggered an excessive use of memory on our servers. Some users may experience slowness when using the application. All emails are still processed and stored properly, and no customer communications will be lost. A fix for this issue is currently being tested.

Report: "Intermittent outages on Groove Inbox & Knowledge base products"

Last update
resolved

This incident has been resolved.

investigating

We have received multiple reports of degraded performance and outages. We are currently investigating the issue.

Report: "Inbox Authorization Issue"

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Blog down"

Last update
resolved

This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating our non-responsive Blog site.

Report: "Slow app performance"

Last update
resolved

This incident has been resolved.

monitoring

Traffic has decreased and operations are returning back to normal. We are still in the process of adding application servers to handle any future spikes and we will continue to monitor the situation.

identified

Our ops team has identified the issue as a sharp increase in traffic. We are in the process of adding more application servers to handle the increased load.

investigating

Our ops team is currently investigating slowdowns across the Groove app.

Report: "Technical incident"

Last update
resolved

This incident has been resolved.

investigating

App access and performance have been restored.

investigating

Currently investigating a service interruption which may be causing issues loading the app for some accounts.

Report: "Degraded Performance"

Last update
resolved

Our background processing queue is caught up and our search cluster has properly rebalanced. This incident has been resolved.

monitoring

A fix has been implemented and our search cluster and background processing jobs are recovering. We will continue to monitor the situation as the recovery progresses.

identified

At 12:30 UTC our background processing queue began backing up jobs. At 14:00 UTC we received a notification of degraded performance in the web app, caused by the growing backlog of unprocessed jobs. We have identified an issue with our search cluster's rebalancing behavior as the source of the backlog and are currently implementing a solution.

Report: "Intermittent API request failures"

Last update
resolved

This incident has been resolved.

investigating

We're experiencing intermittent access problems to Groove Application and Knowledge base pages. The issue being investigated and we will report back when it's resolved.

Report: "Errors in reports loading"

Last update
resolved

A fix has been applied and Errors rates have dropped to normal.

identified

We are continuing to work on a fix for this issue.

identified

We have received reports of errors in Reporting and are working on a fix. The problem is intermittent and refreshing the page after a short wait will normally delivery the numbers you are after.

Report: "Delays in new ticket notifications.."

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

Some customers might see a delay in seeing new ticket notifications appear in Groove. We're investigating a larger than usual load on our realtime cluster.

Report: "Delays in reports processing"

Last update
resolved

This incident has been resolved.

monitoring

Some customers might see a delay in data arriving in reports, we've applied fixes and are monitoring the situation.

Report: "File attachment downloads being flagged by Google"

Last update
resolved

This incident has been resolved and files can be downloaded.

identified

Google is currently showing a warning when downloading file attachments from Groove. We have submitted a request for this flag to be removed and are currently waiting for a response from Google.

Report: "Delays in incoming email processing (gmail)"

Last update
resolved

Gmail has advised resolution of their issue. Groove is processing incoming emails from Gmail again normally.

monitoring

We continue to monitor the situation. Google has updated their status page but provided no ETA on resolution: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=74a7efa47b9de665d02699bbe9dd11fc

monitoring

We continue to monitor the situation. Google has updated their status page with more information on the outage: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid=74a7efa47b9de665d02699bbe9dd11fc Latest update from Google: "Our team is continuing to investigate this issue. We will provide an update by 02/06/2019 23:33 with more information about this problem. Thank you for your patience. We are experiencing high levels of network congestion in the eastern USA, affecting multiple service in Google Cloud, GSuite and YouTube. Users may see slow performance or intermittent errors. We believe we have identified the root cause of the congestion and expect to return to normal service shortly."

monitoring

We have narrowed the issue to the ongoing Google outage. We're monitoring the situation. As soon as the Google outage is resolved, the delayed emails will be loaded into Groove.

investigating

We are continuing to investigate the issue. Gmail mail processing is delayed, however forwarded inboxes continue to function regularly.

investigating

We're experiencing delays in processing incoming emails from Gmail. We're investigating a possible relation to current Google issues.

Report: "Customers Incorrectly presented with Demo Mailbox"

Last update
resolved

This incident has been resolved.

identified

Issue has been identified and a fix will be rolled out in a few moments.

identified

The issue has been identified and a fix is being implemented.

Report: "Customers Incorrectly presented with Demo Mailbox"

Last update
resolved

We've rolled out the fix and all customers should now be able to access Groove correctly.

identified

We're identified the issue and rolling out a fix right now.

investigating

We've noticed an issue where customers are erroneously presented with a demo mailbox when they have a fully working mailbox. We are currently investigating what's causing the issue.

Report: "Snooze intermittently not working for users"

Last update
resolved

The issue has been resolved and the snooze function is now working for all users.

identified

We've identified an issue with our database causing the snooze function to not work for some users. Our team is currently working on mitigating the issue.

Report: "Intermittent Connectivity Issues"

Last update
resolved

The incident has been resolved and all servers are functioning normally.

monitoring

We've determined the issue was caused by setting the “out of memory” killer too low, which ensures our servers don’t balloon too much in memory. We increased the setting, applying a fix to all servers, and now services are returning to normal operations. We are continuing to monitor the situation.

investigating

We're experiencing intermittent connectivity issues and our team is currently investigating the cause.