Historical record of incidents for Groove
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results - backlog is currently clearing over the next 30-60 minutes. Expected delays for incoming mails, outgoing mails being sent, rule processing, realtime, and other features during this time.
The issue has been identified and a fix is being implemented.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"
Last updateThis incident has been resolved.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results - backlog is currently clearing over the next 30-60 minutes.Expected delays for incoming mails, outgoing mails being sent, rule processing, realtime, and other features during this time.
The issue has been identified and a fix is being implemented.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates, sending mail)"
Last updateWe've identified and corrected a rogue process putting elevated load on the search cluster.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are currently investigating this issue.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We have implemented a fix to address the issue. Our team is actively monitoring the results to ensure system stability and performance. We will continue to observe the situation closely and provide further updates as needed. Thank you for your patience.
Report: "Live Chat delayed messages"
Last updateWe have successfully deployed a fix for the issue causing delays in some live chats. We want to reassure you that all delayed chats have successfully reached Groove and no messages were lost. Thank you for your patience. If you experience any further issues, please contact our support team.
A fix for delayed messages from Live Chat was rolled out - we're seeing intermittent issues since deploying the fix. The teams are working on resolving the issues still being seen with delays for Live Chat messages. Note: no other areas of Groove are affected.
Report: "Live chat delays"
Last updateWe have successfully deployed a fix for the issue causing delays in some live chats. We want to reassure you that all delayed chats have successfully reached Groove and no messages were lost. Thank you for your patience. If you experience any further issues, please contact our support team.
We have identified an issue where some live chats are delayed. We are in the process of deploying a fix. This has no impact on any other channels or features within Groove. We will update this status page when a fix has been deployed.
Report: "Service outage"
Last updateAn issue with one of Groove's database hosting providers caused a portion of Groove users to be unable to access Groove from certain devices for approximately 17 minutes. No data was lost during this time, and service has been completely restored for all Groove accounts.
Report: "Attachment issue - between 12:10 UTC and 4:10 UTC"
Last updateOne of our upstream providers experienced an issue that corrupted some attachments before the mails reached Groove. This issue has been resolved. Some attachments can still be retrieved from your mail provider.
Report: "New slack connections restricted"
Last updateThis incident has been resolved. Slack has re-approved our integration.
Our slack app is currently being reviewed for new scopes according to Slack's app policy. During this time, creating new Slack connections may be restricted.
Report: "Meta Outage - affecting Instagram and Messenger connectivity"
Last updateMeta (Facebook Messenger and Instagram) currently have an outage affecting API endpoints from connecting. Users may have disconnections from their Social Chats while the outage is happening. https://metastatus.com/
Report: "Degraded service on Groove inbox (delay on incoming tickets, logging in, live chat, ticket updates)"
Last updateFull service has resumed.
Users may see degraded performance in some areas in Groove.
Report: "Meta Messenger API Outage"
Last updateMeta has resolved the incident on their end. We are continuing to monitor any impact to affected Instagram Messenger users in Groove.
We are continuing to investigate this issue.
Meta is currently experiencing a major outage of the Instagram Messenger API. Unfortunately, no additional information has been provided by Meta regarding the cause or expected resolution time. While not all Instagram channels on Groove are affected, some are experiencing disruptions. Our team is actively monitoring the situation.
Report: "[Outage] Groove Web App Unavailable"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "[Outage] Delays with processing outbound emails"
Last updateThis incident has been resolved. Our email delivery partner inadvertently flagged our account for review this morning. Their action placed all outbound emails from Groove mailboxes that do not have custom email server settings and are not connected via Gmail into a temporary hold. All delayed emails have since been delivered. **To be clear, no emails were lost during this outage, they were simply delayed in their delivery.** We have been in contact with our email delivery partner regarding this incident and are awaiting addition information from them so that we can prevent this issue from occurring again.
The issue has been identified and a fix is being implemented.
Report: "Degraded service on Groove inbox (Delayed mail fetching, collision detection, realtime updates)"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Report: "Knowledge base display / slow loading issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Knowledge bases hosted by Groove may be experiencing slow loading or display issues. Our team is actively investigating the issue.
Report: "Connectivity issues"
Last updateWe have completed the fix and the incident is resolved.
We have identified the source of the issue and implemented a fix - Groove is back online and we're actively monitoring the situation.
We are currently experiencing major connectivity issues related to an Amazon Web Services issue. We are investigating ways of coming back online as soon as possible.
Report: "Service Outage"
Last updateThis incident has been resolved.
A solution has been implemented and we are monitoring the results.
A fix has been implemented and we are monitoring the results.
We are working with our upstream provider to resolve the issue as quickly as possible.
We are continuing to work on a fix for this issue.
We are in working with our upstream provider to resolve the issue as quickly as possible.
A solution has been implemented and we are monitoring the results.
We are in working with our upstream provider to resolve the issue as quickly as possible.
We're continuing to monitor the elevated load and seeing more slow requests
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are seeing degraded performance with an upstream provider, which impacts our ability regarding server requests - we're currently investigating
Report: "Service slowdown"
Last updateA subsystem for ensuring the accurate delivery of emails caused a gradual backup in job scheduling calls, which adversely affected the performance of our web workers.
This incident has been resolved.
A slowdown in service is currently being investigated. Groove may load slowly for customers at this time.
Report: "Groove slow to load in app and mobile."
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue with speed to load with the groove app and mobile.
Report: "Groove connectivity issues"
Last updateThis incident has been resolved.
We are currently investigating an issue where mailboxes aren't loading.
Report: "Groove connectivity issues"
Last updateThis incident has been resolved.
We've identified the issue and deployed a fix
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "DNS outage causing Groove outage"
Last updateThis incident has been resolved.
Here is the status page to monitor the issue with the DNS provider: https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj You can subscribe for updates as well.
We are continuing to investigate this issue.
We are currently investigating an issue with our DNS provider outage.
Report: "Email receive/send issues related to Google Outage"
Last updateThis incident has been resolved.
Our monitoring reveals that the situation on Google's side is resolved, we are seeing mails coming in via Gmail API correctly. We will continue to monitor.
We are still seeing errors getting returned from the Gmail API, we continue to investigate for a possible solution
We are continuing to investigate this issue.
We are investigating an issue with the Google Suite of products, which is limiting our ability to retrieve and send email messages using the Gmail API. More information can be found at - https://www.google.com/appsstatus#hl=en-GB&v=status
Report: "Gmail connectivity issue"
Last updateThis incident has been resolved.
We are deploying a temporary workaround that will allow gmail syncing to continue while Google addresses the core problem on their servers.
Google is currently experiencing a temporary api outage that affects Groove's ability to retrieve emails from gmail accounts. Google is addressing the issue internally, and a fix should be issued shortly. No messages will be lost, but you may experience delays in receiving messages in your Groove inbox during this time.
Report: "Connectivity issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Groove application is not functional because of connectivity issues - we are investigating.
Report: "Email processing delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are backed up in processing incoming emails. No emails are lost. They will be processed once we are back online.
Report: "Server overload"
Last updateThis incident has been resolved.
We have resolved the issue and are monitoring the situation
We are continuing to investigate this issue.
We are investigating a case of server overload. More info to follow.
Report: "Outage with our upstream provider"
Last updateThis incident has been resolved.
We are experiencing a full outage due to issues with our upstream provider. Our team of engineers is working closely with them to bring our services up as soon as possible.
Report: "Partially degraded performance"
Last updateThis incident has been resolved.
A fix has been deployed, and we are currently monitoring memory usage.
A recent deploy has triggered an excessive use of memory on our servers. Some users may experience slowness when using the application. All emails are still processed and stored properly, and no customer communications will be lost. A fix for this issue is currently being tested.
Report: "Intermittent outages on Groove Inbox & Knowledge base products"
Last updateThis incident has been resolved.
We have received multiple reports of degraded performance and outages. We are currently investigating the issue.
Report: "Inbox Authorization Issue"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Blog down"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating our non-responsive Blog site.
Report: "Slow app performance"
Last updateThis incident has been resolved.
Traffic has decreased and operations are returning back to normal. We are still in the process of adding application servers to handle any future spikes and we will continue to monitor the situation.
Our ops team has identified the issue as a sharp increase in traffic. We are in the process of adding more application servers to handle the increased load.
Our ops team is currently investigating slowdowns across the Groove app.
Report: "Technical incident"
Last updateThis incident has been resolved.
App access and performance have been restored.
Currently investigating a service interruption which may be causing issues loading the app for some accounts.
Report: "Degraded Performance"
Last updateOur background processing queue is caught up and our search cluster has properly rebalanced. This incident has been resolved.
A fix has been implemented and our search cluster and background processing jobs are recovering. We will continue to monitor the situation as the recovery progresses.
At 12:30 UTC our background processing queue began backing up jobs. At 14:00 UTC we received a notification of degraded performance in the web app, caused by the growing backlog of unprocessed jobs. We have identified an issue with our search cluster's rebalancing behavior as the source of the backlog and are currently implementing a solution.
Report: "Intermittent API request failures"
Last updateThis incident has been resolved.
We're experiencing intermittent access problems to Groove Application and Knowledge base pages. The issue being investigated and we will report back when it's resolved.
Report: "Errors in reports loading"
Last updateA fix has been applied and Errors rates have dropped to normal.
We are continuing to work on a fix for this issue.
We have received reports of errors in Reporting and are working on a fix. The problem is intermittent and refreshing the page after a short wait will normally delivery the numbers you are after.
Report: "Delays in new ticket notifications.."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Some customers might see a delay in seeing new ticket notifications appear in Groove. We're investigating a larger than usual load on our realtime cluster.
Report: "Delays in reports processing"
Last updateThis incident has been resolved.
Some customers might see a delay in data arriving in reports, we've applied fixes and are monitoring the situation.
Report: "File attachment downloads being flagged by Google"
Last updateThis incident has been resolved and files can be downloaded.
Google is currently showing a warning when downloading file attachments from Groove. We have submitted a request for this flag to be removed and are currently waiting for a response from Google.
Report: "Delays in incoming email processing (gmail)"
Last updateGmail has advised resolution of their issue. Groove is processing incoming emails from Gmail again normally.
We continue to monitor the situation. Google has updated their status page but provided no ETA on resolution: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=74a7efa47b9de665d02699bbe9dd11fc
We continue to monitor the situation. Google has updated their status page with more information on the outage: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid=74a7efa47b9de665d02699bbe9dd11fc Latest update from Google: "Our team is continuing to investigate this issue. We will provide an update by 02/06/2019 23:33 with more information about this problem. Thank you for your patience. We are experiencing high levels of network congestion in the eastern USA, affecting multiple service in Google Cloud, GSuite and YouTube. Users may see slow performance or intermittent errors. We believe we have identified the root cause of the congestion and expect to return to normal service shortly."
We have narrowed the issue to the ongoing Google outage. We're monitoring the situation. As soon as the Google outage is resolved, the delayed emails will be loaded into Groove.
We are continuing to investigate the issue. Gmail mail processing is delayed, however forwarded inboxes continue to function regularly.
We're experiencing delays in processing incoming emails from Gmail. We're investigating a possible relation to current Google issues.
Report: "Customers Incorrectly presented with Demo Mailbox"
Last updateThis incident has been resolved.
Issue has been identified and a fix will be rolled out in a few moments.
The issue has been identified and a fix is being implemented.
Report: "Customers Incorrectly presented with Demo Mailbox"
Last updateWe've rolled out the fix and all customers should now be able to access Groove correctly.
We're identified the issue and rolling out a fix right now.
We've noticed an issue where customers are erroneously presented with a demo mailbox when they have a fully working mailbox. We are currently investigating what's causing the issue.
Report: "Snooze intermittently not working for users"
Last updateThe issue has been resolved and the snooze function is now working for all users.
We've identified an issue with our database causing the snooze function to not work for some users. Our team is currently working on mitigating the issue.
Report: "Intermittent Connectivity Issues"
Last updateThe incident has been resolved and all servers are functioning normally.
We've determined the issue was caused by setting the “out of memory” killer too low, which ensures our servers don’t balloon too much in memory. We increased the setting, applying a fix to all servers, and now services are returning to normal operations. We are continuing to monitor the situation.
We're experiencing intermittent connectivity issues and our team is currently investigating the cause.