Historical record of incidents for GrowthZone
Report: "Some users are experiencing system errors on GrowthZone"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Email deliverability issue"
Last updateThe issue causing a delay in sending bulk emails and email previews has been resolved. Thank you for your patience as we worked through this issue.
We are continuing to to work on a fix for this issue. Another update will be provided as soon as we have more information.
We are continuing to to work on a fix for this issue. Another update will be provided as soon as we have more information.
We are continuing to to work on a fix for this issue. Another update will be provided as soon as we have more information.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being worked on. We expect that any emails have been processed or queued for delivery will be delivered when the issue has been resolved. We apologize for this delay in your email delivery.
We are currently investigating an issue regarding email deliverability.
Report: "GrowthZone Platform - Some functions unavailable or displaying errors"
Last updateThis issue with Azure cloud service has been cleared and operations have returned to normal. We appreciate your patience as we worked to mitigate the affects of this outage.
We have mitigated the issues to restore service at this time. We'll continue to monitor until the Azure incident is resolved. Azure updates can be monitored here: https://azure.status.microsoft/en-us/status
We are continuing to work on a fix for this issue.
Our cloud provider is currently investigating a service disruption issue affecting a subset of customers, including GrowthZone. We will post more information as soon as we have it available.
Report: "We are currently experiencing an outage on our GrowthZone platform"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "GrowthZone app is down"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Some users are experiencing system latency in completing some jobs"
Last updateThis incident has been resolved.
Due to high usage some customers are experiencing latency in some functionality such as billing. We are aware of the situation and monitoring closely. Our queues are processing and your jobs will be completed. We apologize for this inconvenience and appreciate your patience.
Report: "Intermittent availability for some GrowthZone websites"
Last updateThis incident has been resolved.
A fix has been implemented for GZCMS websites and we are monitoring the results.
We are currently investigating this issue.
Report: "Customers with websites created by GrowthZone - Websites are currently down."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating the issue and will provide an update as soon as we have more information.
Report: "Website outage for some GrowthZone customers"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Some GrowthZone customers are experiencing an issue with their website being down - we are currently investigating this issue and will provide an update as soon as we have more information
Report: "Delayed export/report times experienced by some user"
Last updateThis incident has been resolved.
Some users are experiencing delayed response times when downloading/exporting files - we have identified the issue and expect that the system will be caught up within the hour.
Report: "GZCMS Websites"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate issues with the Homepage, and dashboard issues with Pages, Posts Member Only screens.
We are investigating issues with the Homepage, and dashboard issues with Pages, Posts Member Only screens.
Report: "Vacation Page error on Membership Application"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue where users are receiving a Vacation Page when clicking on Membership Application
Report: "Email deliveries may be delayed"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Sendgrid has mitigated the issue and is monitoring email traffic for success.
Our email vendor is currently investigating an issue that is causing delays in email deliverability; You may track their status page using this link https://status.sendgrid.com/
Report: "System slowness on main GrowthZone app"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Degraded performance in GrowthZone app"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating degraded performance in the GrowthZone application.
Report: "Intermittent System slowness"
Last updateThis incident has been resolved.
We have identified the source of the issue and a resolution, We will continue to monitor and update as needed.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue where some GrowthZone users are experiencing intermittent slowness
Report: "GrowthZone users receiving Vacation Page when logging in"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
The service has returned to operational however we will continue to monitor
We are currently investigating an issue where GrowthZone users are receiving a vacation page error when loggin into the system or loading new pages.
Report: "Delayed Background Processing"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
We have identified a slowdown in background job processing. This causes delays in new Activity records, global search updates, email delivery statistics, and membership directory updates as well as other background jobs. These will be caught up as we resolve this issue.
Report: "Delayed Background Processing"
Last updateThis has been resolved. All delayed jobs have been processed.
The delayed background jobs are being processed at this time. Another update will be posted once the delayed jobs have been processed.
We have identified a slowdown in background job processing. This causes delays in new Activity records, global search updates, and membership directory updates as well as other background jobs. These will be caught up as we resolve this issue.
Report: "Degraded Performance"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
System performance has returned to normal. We are continuing to closely monitor.
GrowthZone is aware of and working to resolve the slowness issues that some customers are experiencing. We will keep the Status Page located here current with updates. We apologize for the inconvenience and appreciate your patience. Thank you.
Report: "M1 Sync Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified on the M1 side and the fix is in progress.
We are currently investigating delays in the data sync between GrowthZone and M1
Report: "Unintended event description is being populated across the platform"
Last updateThis incident has been resolved.
We are experiencing an issue where an event description is being populated across the platform and we're investigating. You may notice it is a GrowthZone event and while we'd love to have you all, we are working to remove it from the display. It is not public facing - only visible from the backend.
Report: "Scheduled Bill Dates, Payments, and Invoices are displaying as on day early"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue where schedule bill dates, payments and invoices are displaying as one day early - this is a display issue only, the dates within the database are correct. Updates will be provided within an hour or as soon as available.
Report: "Slow responsiveness of application"
Last updateThis issue has been resolved. We have identified the root cause and are making further changes to address this so as to prevent future issues.
Mitigation efforts are showing responsiveness is back to normal. We are continuing to monitor to ensure the issue is resolved.
We have identified the cause of the degradation and are performing mitigation actions at this time.
Monitoring identified system pressure causing degraded performance of GrowthZone Application.
Report: "Event Not Invoicing Additional Attendees"
Last updateThis incident has been resolved. The fix has been applied and data clean-up has been completed.
We are aware of an event registration issue impacting a group of roughly 300 customers. The issue occurred on the invoices of registrations where more than one attendee was registered. A fix has been applied to production this afternoon and our teams are working through the data clean-up activity related to the incident. The clean-up is expected to occur by the end of day Friday.
Report: "NRDS Integration delays in processing some records"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating delays in successful publishing from GrowthZone to NRDS.
Report: "NRDS Integration delays in processing some records"
Last updateRecords are processing normally again. Delayed records have been completed.
NRDS has reported the issue has been resolved - we're monitoring to ensure all systems are fully caught up.
We are currently investigating errors received from NRDS when sending data to NRDS. This will cause some changes and additions to be delayed from arriving within NRDS.
Report: "Contacts List not Populating"
Last updateThis incident has been resolved.
We have identified an issue causing the contacts list to not load results when the sort order by name is being used. A fix is in progress and will be deployed this morning. You can still use the global search at the top next to the Quick Actions button to access specific contacts.
Report: "Delays in processing Real Estate Integration Updates"
Last updateProcessing is back to normal frequency.
Older items have been processed but new jobs are continuing to be delayed in processing. We're continuing to work to mitigate this issue.
We are continuing to work to process all the delayed updates.
A fix has been implemented and we are monitoring the results.
We are currently seeing delays in processing integration updates for Real Estate Integrations.
Report: "Email Designer Impacted by AWS Outage"
Last updateThis incident has been resolved.
The email designer platform utilizes Amazon's AWS services. AWS US EAST-1 is down for many companies all over the world. We are closely monitoring it, and we apologize for this inconvenience. As per AWS, they have already identified the root cause and are actively working towards recovery.
Report: "Real Estate Sync Slow"
Last updateDelayed Jobs have been processed and we are continuing to ensure that all jobs are processing promptly as normal.
We have identified an issue causing a delay in the processing of information to integrated systems and are working to resolve this. All delayed records will be promptly processed as our team corrects this issue.
Report: "Requests to GrowthZone slow or unresponsive"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "GrowthZone Services Unavailable"
Last updateWe've identified the source of the traffic that caused and are implementing changes to prevent this in the future.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Error when Logging Calls or Notes"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are investigating an issue causing an error when trying to record a note.
Report: "GrowthZone is currently showing Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Incoming Email Processing Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue with incoming emails (replies and archives) being delayed in processing and our team is working to address this. These emails will be processed, recorded, and delivered once this is resolved.
Report: "Incoming Email Processing Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our internal monitoring has identified an issue with incoming emails not being processed. This started occurring at approximately 8 PM Central Time on 8/12/2021 Impact: - Replies to emails sent from GrowthZone have not been processed so you won't see them in GZ and the sender won't have received the reply. - Archive@ emails have not been recorded - Emails sent to a group address such as the group-#####@growthzoneapp.com have not been processed or sent. Once resolved these emails will be processed, recorded in GZ, and sent normally.
Report: "Errors Accessing GrowthZone Services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Services Unavailable."
Last updateOur team, after working with the service provider, confirmed that the cause of the outage was a Distributed Denial of Service (DDOS) attack. These types of attacks simply use massive amounts of requests to overwhelm the network causing legitimate traffic to have no path. This lasted an hour and services have returned to normal operations.
A fix has been implemented and we are monitoring the results.
The issue has been narrowed down to a connectivity issue at the ISP level. We are continuing to work with the provider and will provide further details as they are available.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "GrowthZone Services Degraded"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Issues Downloading/Printing Invoices and Payments"
Last updateThis incident has been resolved.
We have identified an issue preventing some users from being able to download or print invoices and payments. We are currently working to get this resolved promptly.
Report: "Error Creating an Invoice"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
A fix for this is currently being verified and prepared for implementation.
We've identified an error that is occurring when creating an invoice. Our team is working to resolve this promptly.
Report: "Services Unavailable."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Errors accessing Reports"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've identified an issue with accessing reports and are currently working to restore this service.
Report: "GrowthZone Services Degraded"
Last updateThis incident has been resolved.
Services are operational again. We have implemented changes to prevent service interruption of this kind.
Services are online again and we are continuing to implement mitigation to prevent further service disruption.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Intermittent Unavailability on Azure Powered Services"
Last updateAll services have been restored to normal operations.
We are continuing to monitor the Azure status updates and will post further updates when the services impacting GrowthZone are showing as restored. Update from Microsoft ( https://status.azure.com ) Due to downstream impact to a number of Azure services, recovery times may vary by service.
Update from Microsoft: Microsoft rerouted traffic to our resilient DNS capabilities and are seeing improvement in service availability. We are continuing to investigate the cause of the DNS issue.
Azure is currently reporting DNS issues with its infrastructure. This is impacting the functionality of areas within GrowthZone. These areas we are seeing impacted include Cloud files, Info Hub Resources, and Communication Details. You can follow updates from Azure directly at https://status2.azure.com/ We will post further updates as we track their progress on resolution.
Report: "Service Provider Outage"
Last updateThere was a power outage that impacted areas of the Minneapolis metro area, including our data center location. This caused a temporary outage that impacted the backoffice and websites. Service has been fully restored.
Report: "GrowthZone Service Degredation"
Last updateImpacted services have now been restored.
Services are available again and we are continuing to implement mitigation for the impacted services.
The issue has been identified and a fix is being implemented.
We're currently investigating issues impacting GrowthZone Application Servers.
Report: "GrowthZone Is currently Unavailable"
Last updateThis incident has been resolved. Our engineers are working to prevent this issue from happening again in the future.
Services have been restored and our team is still investigating the root cause.
Users are currently getting a vacation error page when accessing GrowthZone Pages. Our engineers are investigating.
Report: "NRDS Sync errors"
Last updateThis incident has been resolved.
Data has been syncing successfully for the past 5 minutes. Waiting on confirmation of resolution from NRDS
NRDS acknowledged that the issue is on their side and they are working to resolve it.
The NRDS Sync has been failing this morning due to an issue at NRDS. We have contacted their team and are waiting for an update. Once resolved these syncs that failed will all catch up. The error message you will see is "MQRC_NO_MSG_AVAILABLE"