Is GoFormz Down Right Now? Check if there is a current outage ongoing.

At approximately 7:20 am PT on Monday, March 3rd, it was identified that some GoFormz users were experiencing errors when logging into the platform. The GoFormz Engineering team quickly investigated and found that Microsoft Azure released an update that impacted our CDN layer. At approximately 11:10 am PT on Monday, March 3rd, the GoFormz Engineering team implemented a fix and restored all users’ ability to access the GoFormz platform as expected. This issue is now resolved, as we have continued to see stable performance throughout the day. If you have any further questions, please contact your account manager or email [support@goformz.com](mailto:support@goformz.com).

This issue is now resolved, as we have continued to see stable performance throughout the day. If you have any further questions, please contact your account manager or email support@goformz.com.

The root cause of this issue has been identified, and a fix has been implemented. All services are operating as expected. We are continuing to monitor performance.

Some accounts may be experiencing intermittent issues when attempting to log in to the GoFormz platform. We are investigating the cause of these errors and will share an update shortly.

This incident has been resolved.

As of 9:22 PM PST, all functionality on the GoFormz platform has been restored. Our engineers are continuing to investigate any potential impacts on our system. An ongoing incident at the Azure Datacenter persists, and we will continue to monitor our system's performance closely.

From Azure's Engineers: Starting at 18:44 UTC on 26 Dec 2024, there was a power incident in a portion of South Central US AZ03 which impacted one of the three physical AZ, affecting multiple services. We are currently investigating and attempting to restore the service More Info available at Azure's status page - https://azure.status.microsoft/en-us/status

We are experiencing interrupted performance throughout several areas of the platform, due to a power outage impacting several Azure Datacenters. We will continue to monitor performance and availability. More Info available at Azure's status page - https://azure.status.microsoft/en-us/status

On Oct 4, 2024, an issue was identified affecting users’ ability to log in when using an iOS 18\+ device. This issue did not impact users already logged into the GoFormz app or using an iOS version before iOS 18. The root cause of this issue was a workaround implemented yesterday \(10/3/24\) in the GoFormz mobile app version 4.0.777 to accommodate compatibility with iOS 18 beta versions. However, Apple also recently updated iOS 18\+ devices to address the workaround’s original need, causing the fix applied for the beta versions to exhibit unexpected behavior and prevent users from logging in. To resolve this issue, GoFormz released a new iOS mobile app version \(v.4.0.789\) that removes this code and operates as expected. The GoFormz team is focusing on implementing adjustments to our internal mobile release testing processes to prevent incidents like this from occurring again.

This issue has been resolved, and a new iOS GoFormz app version is available to download from the App Store. If you are using a device running iOS 18 or higher, please update your GoFormz app to version 4.0.789 as soon as possible. A postmortem of this incident will be shared shortly.

We have identified the source of the issue and are working to release a fixed iOS mobile app version.

GoFormz iOS app version 4.0.777 is experiencing a partial outage when opened using an iOS 18-enabled device. We are working to release a new iOS app version as soon as possible, removing the root cause of this occurrence.

This incident has been resolved.

A fix has been implemented and the Web App Support Chat Bubble is operating as expected. We are continuing to monitor.

We are experiencing interrupted performance of the web app support chat bubble. As we work to resolve this issue, please send requests for support to support@goformz.com.

This incident has been resolved. A postmortem update detailing the resolution will be issued next week.

A fix has been implemented and we are monitoring the results.

GoFormz is currently experiencing two issues impacting our workflow and login services for some customers: Issue 1: Certain workflows are modifying forms unintentionally when executed. Issue 2: Some users attempting to access the GoFormz Web App are experiencing login issues preventing them from moving forward into the app We have identified the source of both issues and are taking steps to resolve them as quickly as possible.

This incident has been resolved.

This interruption has been resolved and all email delivery is operational. We are continuing to monitor the service closely.

We are investigating an interruption to the email delivery service leveraged by GoFormz. This includes system emails such as reset password requests and user invites, as well as, automated emails triggered through Workflow and/or Template Events.

This incident has been resolved.

This interruption has been resolved and all email delivery is operational. We are continuing to monitor the service closely.

We are currently investigating an interruption to the email delivery service leveraged by GoFormz. This includes system emails such as reset password requests and user invites, as well as, automated emails triggered through Workflow and/or Template Events.

**Overview** GoFormz’s SMS/Text delivery option for Public Form links experienced instability due to misuse by fraudulent accounts. As a result, the text service supporting this function was temporarily suspended. **Timeframe of Impact** Intermittent outages between 5/30/23 8pm PT to 6/2/23 9am PT **Who was this impacted?** 1. Users who created a Public Form link based on an existing Form and sent it using the “Text” option manually from the Public Forms Modal during the timeframe above 2. Workflows that created a Public Form link based on an existing form and sent it using the “Text” option in the Workflow step during the timeframe above **Resolution** The GoFormz team took and is continuing to take several mitigating actions to return to operational service and ensure this does not occur again including: 1. Locked all activity from the offending accounts 2. Adjusted the rate limit set for how many public forms can be sent in a given time period 3. Added URL detection to the body of public form messages that finds and prevents any links from being sent \(with the exception of the Public Form Link to the GoFormz Form\) **Current Status** The Workflow Create Public Form Step has been returned to full operational status. Sending a Public Form as a text message will remain unavailable when using the Create Public Form modal in the web application as we continue to take additional measures that prevent similar abuse. We expect this functionality to be returned in the near future.

This incident has been resolved. GoFormz is continuing to monitor this function closely.

A fix has been implemented for our Workflow solution and we are continuing to monitor the create public form on the Web App pathway closely before resolving.

GoFormz is investigating an issue that is preventing Public Form links sent using the Text option to not be delivered as expected. Note: this impacts Public Forms that are sent using the Text option in the Create Public Form modal and the Create Public Forms Workflow step. As an alternative, please send Public Forms using the Email option until resolved. Updates will be provided here shortly.

Sending a Public Form via text message has been restored to Operational status for both the Workflow and Web App components of this feature. We are continuing to monitor this function closely and will share a postmortem shortly.

GoFormz is investigating an issue that is preventing Public Form links sent using the Text option to not be delivered as expected. Note: this impacts Public Forms that are sent using the Text option in the Create Public Form modal and the Create Public Forms Workflow step. As an alternative, please send Public Forms using the Email option until resolved. Updates will be provided here shortly.

This incident has been resolved and we are continuing to monitor closely.

GoFormz has identified an issue that is preventing Public Form links sent using the Text option to not be delivered as expected. We have identified the root cause and are working to resolve this interruption. Note: this impacts Public Forms that are sent using the Text option in the Create Public Form modal and the Create Public Forms Workflow step. As an alternative, please send Public Forms using the Email option until resolved. Updates will be provided here shortly.

This incident has been resolved.

Beginning on Thursday (April 6, 2023), the GoFormz Workflow-Salesforce app began to experience latency and timeouts due to a Salesforce outage impacting our integration. This, combined with related client issues, created a backlog of Salesforce workflow jobs in the GoFormz workflow queue. Once resolved, workflows were successfully running, but we were still experiencing slowness from items in the queue. At approximately 11:00am PST today (April 11, 2023), all queue items have processed and operating times have returned to normal.

This incident has been resolved.

All GoFormz Web App services are now operational. GoFormz Engineering is continuing to monitor all services and the Azure Status page (https://status.azure.com/en-us/status) for updates.

Microsoft Azure is currently experiencing an outage impacting the GoFormz Web App. GoFormz Engineering is investigating and monitoring the Azure Status Page closely (https://status.azure.com/en-us/status).

Our team is investigation a partial service interruption. We’ll provide an update as soon as we learn more.

After further investigation, GoFormz Engineering identified the following preliminary root cause of the platform’s intermittent login issues. Please see the following from Microsoft, our cloud provider. —————————————————————————————— Update from Microsoft Azure: —————————————————————————————— Summary of Impact: Between approximately 16:27 UTC on 23 Jan 2023 and 19:38 UTC on 23 Jan 2023, a subset of customers in the South Central US region may have experienced increased latency and/or intermittent connectivity issues for some services in the region. Downstream impact to other impacted services in the region also occurred. Preliminary Root Cause: We identified an unhealthy network device in the regional network gateway in the South Central US region. This device would have impacted traffic between Availability Zones and data centers in South Central US and traffic into and out of the South Central US region. Mitigation: The unhealthy network device was removed from service so that traffic was served through other healthy paths. (https://status.azure.com/en-us/status/history/) —————————————————————————————— End update from Microsoft Azure: —————————————————————————————— If and when further details are available we will update this incident.

After applying mitigation steps to impacted micro services, GoFormz login errors have abated and become stable. GoFormz Engineering is investigating the root cause and will provide details as they become available. We are continuing to monitor the micro services that were impacted.

GoFormz is currently experiencing intermittent login issues. The engineering team is investigating the cause. We’ll provide updates as we learn more, thank you for your patience.

This incident has been resolved.

All GoFormz Web App services are operational. GoFormz Engineering is continuing to monitor all services and the Azure Status page (https://status.azure.com/en-us/status) for updates.

Microsoft Azure is currently experiencing an outage impacting the GoFormz Web App. GoFormz Engineering is investigating and monitoring the Azure Status Page closely (https://status.azure.com/en-us/status).

Please review the postmortem below shared by the Microsoft Azure Team: [https://status.azure.com/en-us/status/history/](https://status.azure.com/en-us/status/history/) ### Azure Firewall, Azure Data Explorer, Azure Synapse Analytics, Azure Back up, Azure Search - Multi Region \(Tracking ID YVTL-RS0\) Starting at 05:26 UTC on 28 Jun 2022, a subset of customers may experience intermittent network failures and packet loss while trying to connect to the Azure Firewall hosted in affected regions. Additionally, those attempting to perform Backup operations in the affected regions may see intermittent failures. We have determined some downstream services impacted have now been mitigated, this includes Azure Data Explorer and Azure Stream Analytics, with the cause identified as an underlying issue in the Azure Software Load Balancer service. **Workstreams**: * Complete: Our first workstream was to update and reset the state in our load balancing platform for impacted endpoints.  * In Progress: Our second workstream is a hotfix deployment is in progress to mitigate and is estimated to be completed by 04:00 UTC on July 1. This hotfix is specifically for our software load balancer platform. A subset of customers using service may experience a recurrence of impact during the rollout of this hotfix. **Current Status**: Until our second workstream is completed, an automated mitigation workflow is running to prevent customer impact. We will continue to provide updates via Service Health in the Azure Portal.

This incident has been resolved. We will be posting a Postmortem shortly.

GoFormz are continuing to monitor, however, all GoFormz services are operational. Please see the latest update below from Microsoft Azure: Azure Status Page: https://status.azure.com/en-us/status Azure Data Explorer and Firewall- Global Starting at 05:26 UTC on 28 Jun 2022, a subset of customers have reported intermittent network failures and packet loss while trying to connect to the service hosted in this region when using Azure Firewall. Customers also reported issues with ingesting data to Azure Data Explorer instances as well as timeout errors when trying to query data. Some other downstream services may be impacted by this incident and their status will be updated via service health. Current Status: We currently have two workstreams to mitigate the issue: our first workstream is updating and resetting storage nodes. The recent deployment caused traffic to be diverted to compute nodes that were marked unhealthy. When the requests would land of these unhealthy nodes they would fail. We are currently working on addressing these nodes and are 73% completed. Our second workstream is that we are building out a hotfix which was deployed around 23:00 UTC. This hotfix is for our software load balancer and was completed at 13:20 UTC and we have been validating its ability to mitigate across the platform. This hotfix will not reintroduce any impact to customers that have been manually mitigated so far. We expect full mitigation to be completed within 24 hours. The next update will be provided in 2 hours, or as events warrant. This message was last updated at 23:03 UTC on 29 June 2022

All GoFormz services have returned to operational. We are continuing to monitor closely.

The Microsoft Azure Networking team is currently investigating network issues impacting various services. For more information, please visit the Microsoft Azure Status page (https://status.azure.com/en-us/status). We are continuing to track this situation closely and will provide further updates as more information is learned.

Users are experiencing trouble logging and and performing actions GoFormz Web App. Engineers are investigating the issue and will report updated status as soon as possible.

At approximately 11 am PDT on Monday 6/20, a version [3.3.337] of the GoFormz Windows app was containing a bug preventing users from logging in to their GoFormz account. A fixed version was released and is now available in the Windows App Store at approximately 1pm PDT Tuesday 6/21. Please navigate to the Windows App Store to update to the newest version [3.3.334].

This incident has been resolved.

An update has been implemented and all Workflow functionality is operational. We are continuing to monitor this component.

GoFormz Engineering is working to mitigate this issue by updating the underlying infrastructure. We are still investigating the root cause of this issue.

We are currently investigating an issue causing the degraded performance of Webhook triggers and experience editing a Workflow in the Workflow Tool. Note: This does NOT affect Workflow jobs that are not related to Webhooks.

**Timeline**: Sunday 12/19/21 6:28 am - 8:32am PST **Root Cause Analysis**: After a detailed investigation, the GoFormz Engineering team determined that one of the database pools had insufficient storage space, due to unreleased data during our normal database maintenance. This resulted in a lack of space available for a subset of customers. **Mitigation**: Immediate action was taken to increase the available space on the database pool. Additionally, alerts were updated to include total available storage space.

This incident has been resolved.

GoFormz Engineering has identified the cause of this issue and all web app services are now operational. We are continuing to monitor all services.

We are continuing to investigate this issue.

GoFormz Engineering is currently investigating a partial disruption to GoFormz Services for some accounts. Actions may be unavailable or exhibit degraded performance.

**Summary**: On 11/16, code containing a defect was released. This code was not in use until 12/02, when customers attempting to generate a pdf saw service errors or latency. **Root Cause Analysis**: After a detailed investigation, the GoFormz Engineering team determined that there was an underlying code defect in a 3rd-party pdf tool that caused a higher-than-normal amount of CPU. During high traffic, this defect created a deadlock and resulted in several servers stalling when attempting to generate a pdf.   Consequently, our API and Web services experienced errors and timeouts, and our backend queues experienced an increase in service latency. **Mitigation**: Immediate action was taken to scale out the pdf services, allowing for the increase in CPU to be handled. Complete mitigation was accomplished when the engineering team was able to correct the code defect and deploy a hotfix to all pdf services. **Next steps**: While this issue is resolved, GoFormz Engineering is continuing to closely monitor each service for errors or instability.

This incident has been resolved.

We are currently monitoring degraded performance in our PDF generation services. GoFormz processes involving PDF generation may be slow or delayed.

GoFormz Engineering is investigating the degraded performance of our PDF generation services. GoFormz processes involving PDF generation may be slow or delayed.

GoFormz Engineering implemented improvements to the PDF export service and all operations have returned to normal. We are continuing to monitor this incident.

We are currently monitoring degraded performance in our PDF generation services. GoFormz processes involving PDF generation may be slow or delayed.

During Tuesday’s (9/21 at 1:15pm PDT) production release, a configuration issue was introduced across various services that resulted in a degraded performance of some customer Workflows under heavy load. GoFormz Engineering identified the root cause of this issue and released a fix Wednesday morning (9/22 at 9:30am PDT). Recommended Action: Workflows that failed during this time may need to be manually rerun in the GoFormz Web App. Additionally, Connected Apps may need the re-authenticated from the Connected Apps page of an ‘Unauthorized Error’ shown or if the status for a particular connection is red.

GoFormz engineering has identified an error with a recent production release. The faulty code has been rolled back, and services are beginning to return to normal operations.

This issue has been resolved. If you are still experiencing this, please follow the instructions on the below document to clear your browsing history and cache and cookies for your respective browser. https://help.goformz.com/en/articles/5410107-how-to-clear-browsing-history-and-cache

We are continuing to monitor this issue. If you are still experiencing this, please follow the instructions on the below document to clear your browsing history and cache and cookies for your respective browser. https://help.goformz.com/en/articles/5410107-how-to-clear-browsing-history-and-cache

We have identified an issue causing Date fields to display incorrectly on the Web App Form Editor. To rectify this, please follow the below steps to clear your cache and cookies in your browser. Chrome: https://support.google.com/accounts/answer/32050?co=GENIE.Platform%3DDesktop Safari: https://support.apple.com/guide/safari/clear-your-browsing-history-sfri47acf5d6/mac Edge & Internet Explorer: https://support.microsoft.com/en-us/microsoft-edge/view-and-delete-browser-history-in-microsoft-edge-00cf7943-a9e1-975a-a33d-ac10ce454ca4

This incident has been resolved.

We are working to resolve an issue affecting account access and most service functions.

As GoFormz services continue to be operational and Microsoft Azure has confirmed the resolution of their DNS outage, we are closing this incident as resolved. A more detailed summary from Microsoft Azure follows (https://status.azure.com/en-us/status/history/): Summary of Impact: Between approximately 21:30 UTC and 22:30 UTC on 01 Apr 2021, customers may have experienced intermittent issues accessing Microsoft services, including Azure, Dynamics, and Xbox Live. Preliminary Root Cause: We are continuing to investigate the underlying cause for the DNS outage but we have observed that Microsoft DNS servers saw a spike in DNS traffic. Mitigation: Microsoft engaged resilient DNS capabilities to absorb the spike in DNS traffic.

As of 3:05 PST - GoFormz services are back up and operational, however Microsoft Azure has not updated their DNS issue as resolved. Thus, GoFormz engineering will continue to monitor the situation and will provide an update as more information is available. The latest message from the Microsoft team: "Microsoft rerouted traffic to our resilient DNS capabilities and are seeing improvement in service availability. We are continuing to investigate the cause of the DNS issue."

GoFormz services are currently impacted by a Domain Name Service disruption with our cloud platform host Microsoft Azure. Most recent update from the Azure team: Azure DNS - Investigating We are currently investigating reports of an issue affecting Azure DNS. More information will be provided as it is known. This message was last updated at 21:59 UTC on 01 April 2021

GoFormz engineering is currently investigating isolated service disruptions that appear to involve Domain Name Services across the Microsoft Azure cloud platform. We investigating the issue and will update as we learn more.

Between approximately 7:30 am PST and 9 am PST, a spike in activity caused degraded performance of the GoFormz PDF Generation service. The affected services have now returned to normal operations and are executing as expected.

We are currently investigating degraded performance in our PDF generation services. GoFormz processes involving PDF generation may be slow or delayed.

A spike in activity caused degraded performance of GoFormz PDF services this morning. Automated service scaling did not initially improve performance. GoFormz engineering reviewed service instances and determined a manual restart of the service was required. After this process completed, PDF services returned to normal operations.

We are currently investigating degraded performance in our PDF generation services. GoFormz processes involving PDF generation may be slow or delayed.

This issue has been resolved as of Dec 15, 2020 17:24 PST. Please see the below statement from SendGrid (https://status.sendgrid.com/incidents/q8z2lk11r24g) "We have received confirmation from our third party that this issue has been resolved. Users can now reattempt mail sends to domains with Google mx records. Please note some users may continue to see impacted mails listed as suppressed. Our engineers will be continuing to focus efforts on correcting this as soon as possible. We sincerely apologize for the inconvenience and thank you for your patience"

The GoFormz email delivery provider, SendGrid, is currently experiencing an issue when delivering emails to some Google domains. Please see the below update from the SendGrid Status page. SendGrid Status Page - Dec 15, 13:44 PST - https://status.sendgrid.com/ "Due to a third party incident starting around 12PM MT, users sending mail to domains with Google mx records may experience an increased number of bounces due to the response code that is being returned. Our engineers have identified the issue and are working towards mitigating the impact. We will provide updates as they become available. Thank you for your patience.

Starting at approximately 9 pm PST on Wednesday, December 2nd, a corrupt Azure instance within the GoFormz Gateway service began causing isolated degraded performance in the GoFormz Web App and Mobile Sync performance. After standard recovery operations did not immediately resolve the issue, GoFormz Engineering implemented startup configuration changes that returned the service to normal operations. As mitigation steps, GoFormz has verified service startup configurations, are adding additional monitoring, and investigating Azure instance instability with Microsoft Azure support.

GoFormz Engineering has identified the cause of this issue and implemented a fix. We are continuing to monitor all services.

We are continuing to investigate this issue.

GoFormz Engineering is investigating a partial outage of the GoFormz Web App affecting areas including some Template editing actions, access to the Workflow page, and Users & Groups actions.

This incident has been resolved.

At approximately 10:30 am PST, GoFormz Engineering identified the cause of this issue and executed measures to return Automated Workflow performance to operational. We are continuing to monitor this service.

GoFormz Engineering is currently investigating an issue causing degraded performance in automated workflow execution.

After investigating this issue, GoFormz Engineering adjusted autoscaling configurations to mitigate the impact of a similar event on our platform. This issue is now resolved, but we are continuing to monitor each service moving forward. Please reach out to GoFormz Support at support@goformz.com if you have any further questions.

Between approximately 7:51 AM and 8:10 AM PST, GoFormz experienced intermittent issues when logging in to and syncing data on mobile devices. GoFormz Engineering has resolved this issue and we are continuing to monitor this incident as we investigate the root cause. All services are operational.

This incident has been resolved.

Starting at approximately 11:10 AM PDT, Microsoft Azure reported degraded performance with services including Azure Active Directory and Network Infrastructure. This issue has been resolved by the Microsoft team. GoFormz Engineering is continuing to monitor impacted services and performance. For more information on this incident, please see the Azure Status History Page: https://status.azure.com/en-us/status/history/

This incident has been resolved.

At approximately 5:57 pm PDT, Microsoft announced they have applied mitigation steps for the issue impacting Azure AD Authentication. The GoFormz Workflow services have returned to operational, however, we are continuing to monitor this event. For more information, please visit the Azure Status page at http://status.azure.com.

Microsoft is currently investigating an issue impacting Azure AD Authentication. GoFormz Workflows requiring authentication from Microsoft Azure (Office 365, OneDrive, and KeyVault) may be affected. More information and updates can be found on the Azure Status page at https://status.azure.com/en-us/status

This incident has been resolved.

We are continuing to monitor for any further issues.

At approximately 5:00 am PST, GoFormz experienced a temporary outage that prevented access to the GoFormz website and sync services within the GoFormz application. This outage was due to a delay in the renewal process of an authentication certificate, which has now been successfully resolved. The service disruption lasted approximately four hours. We sincerely apologize for any inconvenience.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Essential components of the GoFormz platform became unresponsive on Friday 12:25pm Pacific Time due to a corrupt configuration change in our Authentication service. After identifying the underlying issue, GoFormz engineering rolled back the change and restarted impacted services. Upon service recovery GoFormz’s primary authentication database required upscaling to recover completely. All GoFormz services have returned to normal levels as of 1:20pm Pacific Time. GoFormz engineering will be reviewing and adjusting these configuration change procedures to mitigate future occurrences.

This incident has been resolved.

We have identified the issue and implemented a fix. As of approximately 1:20 PM PDT, GoFormz services have returned to normal functionality. We are continuing the closely monitor this issue.

The issue has been identified and a fix is being implemented.

GoFormz is currently experiencing degraded performance. This primarily affects accounts using the GoFormz Workflow or API, and users not already logged into a GoFormz account through the mobile or web app.

This incident has been resolved.

We are continuing to monitor for any further issues.

GoFormz's Public API services have returned to normal operation. We will continue to monitor our API services throughout evening (PST) and update this incident as status changes.

GoFormz's Public API is experiencing degraded performance with multiple end points. After up scaling and restarting API services, errors continue to persist. GoFormz engineering is performing troubleshooting procedures to mitigate the issues. We will provide an update as soon as possible.

This incident has been resolved.

GoFormz's Public API services have returned to normal operation. We will continue to monitor our API services throughout evening (PST) and update this incident as status changes.

GoFormz's Public API is experiencing degraded performance with multiple end points. After up scaling and restarting API services, errors continue to persist. GoFormz engineering is performing troubleshooting procedures to mitigate the issues. We will provide an update as soon as possible.

This incident has been resolved.

We are continuing to work on a fix for this issue.

Due to planned system maintenance with GoFormz Workflow, customer workflows are currently queued while work is being performed. We will update this message as work is completed and Workflow services return to normal.

After review and consultation with Microsoft Azure support, it was determined that an Azure platform event, occurring shortly before GoFormz experienced degraded performance, triggered an application recycle leading to instability with our Forms, Template, and Auth services. Azure support has provided GoFormz engineering and ops teams with specific mitigation steps to more quickly resolve any impact to our services if a future similar platform event occurs.

This incident has been resolved.

After troubleshooting with Azure Support, Forms and Template services have returned to normal operation. We will continue to work with Azure Support to determine the root cause and update this issue upon such findings.

We are currently investigating an issue that is impacting Forms and Template pages. Our engineers are working with Azure support to determine the root cause.

This incident has been resolved. Please see below message from the Microsoft Azure support team: Multiple Services - Mitigated Summary of Impact: Between 00:57 and 03:40 UTC on 20 Nov 2019, customers and services utilizing Azure Front Door (AFD) services were impacted by an infrastructure service failure. This resulted in a loss of connectivity to multiple services reliant on AFD. Azure Front Door services provide Edge caching and network entry point services to the Microsoft global network. This issue impacted a large percentage of Microsoft Services, though not all services were impacted. Many impacted services were able to initiate fail over from the AFD platform, providing immediate mitigation to their customers. Preliminary Root Cause: During a recent periodic deployment, initial safety checks did not detect the issue and prevent the roll out. Monitoring detected once service failure was experienced and alerted engineers. Mitigation: Engineers immediately initiated deployment rollback procedures to correct the underlying Azure Front Door issue. This was completed at 02:40 UTC on 20 Nov 2019, at which point Impacted services began recovering. A detailed root cause analysis will be published within 72 hours.

At 5pm Pacific Time GoFormz was impacted by Azure CDN outages causing problems with login for approximately 45 minutes. The CDN outages have been traced to a current issue with Microsoft Azure, our cloud services provider. Azure engineers have identified outages of their Network Infrastructure across all regions. Even though GoFormz services remain stable, we will continue to monitor all services and will update if our status changes. Current Message from Azure Status as of 7:50pm Pacific Time: Multiple Services - Applying Mitigation Summary of Impact: Starting at 00:57 UTC on 20 Nov 2019, a subset of customers may experience failures when attempting to access Azure Resources globally. Engineers have identified a root cause and mitigation is in progress; customers may see signs of recovery as this progresses.

After successful code rollback, all services have remained stable.

Account services and related services have returned to normal after a rollback from a failed code release. We will continue monitoring services throughout the day.

We are currently experiencing an issue with our Account services that can impact login (mobile and web) and some website functions. We will update this message as the issue progresses.

This incident has been resolved. GoFormz engineering has determined that this incident was caused by intermittent connection issues with MongoDB. To limit further exposure to this type issue impacting customers, we have modified configurations and strengthened monitoring of these connections.

Connection errors have ceased and workflow performance has stabilized.

GoFormz engineering has identified intermittent a database connection errors impacting our Workflow services that may cause some workflow to fail. Work is being done to stabilize the database connections and bring these services back to normal operating behavior. We will update this status message and this incident progresses.

This incident has been resolved.

Service monitoring has revealed degraded performance of the Template Events functionality which may impact the on-time execution of customers' Template Events. GoFormz engineering is currently troubleshooting this issue and will update this status as more information is available.

This incident has been resolved.

GoFormz engineering has implemented a configuration change and our PDF services have stabilized. Although PDF creation and email delivery has returned to normal operations, we continue to monitor these systems.

GoFormz engineering is working to mitigate platform issues involving PDF creation and delivery via email. We will update this status as services stabilize.

This incident has been resolved.

At approximately 7:24 AM PDT, GoFormz's authentication and permissions services began experiencing degraded performance causing outages with some GoFormz features including login, API, form loading, and workflows. The GoFormz engineering team performed immediate restart actions and all services are back to full functionality as of approximately 7:50 AM PDT. We are continuing to monitor this issue.

Starting at approximately 1:47pm, GoFormz's authentication and permissions services experienced degraded performance causing outages with some GoFormz features including login, API, form loading, and workflows. The GoFormz engineering team reviewed configurations and conducted system scaling operations to bring all services back to normal at 2:05pm.

We have received a post mortem from Microsoft's Azure Engineering team which details that the root cause of the partial database outage was a misconfiguration in their system. Per Microsoft: "We identified a misconfiguration in our system that have caused control plane to have intermittent hiccups when communicating to instances on tenant rings. The short term mitigation was performed by reestablishing fresh connections, and long term fix will be done by adjusting the configuration. We have finished the short term mitigation and working on the long term mitigation. We apologize for the inconvenience this may have caused to you."

At approximately 11:57PM PDT on Saturday July 27th, Microsoft Azure reported the event causing the partial outage as resolved. At this time, all functions are operational. We are continuing to monitor this issue.

We are continuing to work on a fix for this issue.

We are continuing to monitor and communicate with Azure Support as the partial database outage affecting some GoFormz accounts is investigated and rectified by the Microsoft engineering team. We will continue to update this status page as more information is shared.

We’ve identified that the cause of the issue is coming from Microsoft Azure. According to the most recent update from Azure Support, Microsoft engineers are working to resolve a critical situation affecting database performance which may be impacting some GoFormz accounts.

We have identified the intermittent disruptions experienced by a limited set of GoFormz users as an Microsoft Azure issue. We are continuing to work with Azure Support to find the root cause and solution.

We are currently investigating intermittent disruptions affecting a limited set of GoFormz customers.

This incident has been resolved.

On June 18th 2019, an outside security company (Barracuda Networks) made an update that resulted in the IP address (198.37.153.141) being registered as Blocked. GoFormz has requested that this IP address be removed from this list and returned to a successful status. As of June 19th 2019 at 9AM PDT this removal has been executed, however, it may take 24-48 hours to take full effect. Please contact support for further assistance.

At approximately 10:24 AM today, GoFormz began to experience issues triggering events on a mobile device. We have rolled back the update causing this issue and are continuing to monitor affected accounts. At this time, events are firing as expected, however, all forms completed between 10:24 AM and 12:20 PM PDT did not trigger events or workflows as expected. To re-fire these events, please reopen and complete the form. This will only need to be done for forms expected to trigger an event and completed during this timeframe from a mobile device.

After service recovery, the Microsoft Azure team is providing more detail regarding the service disruption that occurred on May 2, 2019. Here is that detail directly from Microsoft: > Network Connectivity - DNS Resolution > > **Summary of impact:** Between 19:43 and 22:35 UTC on 02 May 2019, customers may have experienced intermittent connectivity issues with Azure and other Microsoft services \(including M365, Dynamics, DevOps, etc\). Most services were recovered by 21:30 UTC with the remaining recovered by 22:35 UTC. > > **Preliminary root cause:** Engineers identified the underlying root cause as a nameserver delegation change affecting DNS resolution and resulting in downstream impact to Compute, Storage, App Service, AAD, and SQL Database services. During the migration of a legacy DNS system to Azure DNS, some domains for Microsoft services were incorrectly updated. No customer DNS records were impacted during this incident, and the availability of Azure DNS remained at 100% throughout the incident. The problem impacted only records for Microsoft services. > > **Mitigation:** To mitigate, engineers corrected the nameserver delegation issue. Applications and services that accessed the incorrectly configured domains may have cached the incorrect information, leading to a longer restoration time until their cached information expired. > > **Next steps:** Engineers will continue to investigate to establish the full root cause and prevent future occurrences. A detailed RCA will be provided within approximately 72 hours.

As of 5:00 PM today (May 2nd), Microsoft Azure has updated all services to Operational. GoFormz has been stable since approximately 3:53 PM PDT and we are confident all users should be fully returned to their normal GoFormz services. If you are still experiencing any errors or issues please reach out to us at support@goformz.com.

GoFormz services are performing as operational. We are continuing to monitor this issue. Please see below for the latest update from Microsoft Azure. "Starting at 19:43 UTC on 02 May 2019, customers may experience intermittent connectivity issues with Azure and other Microsoft services (including M365, Dynamics, DevOps, etc). Engineers have identified the underlying root cause as a name server delegation issue with DNS resolution, affecting network connectivity and downstream impact to Compute, Storage, App Service, AAD, and SQL Database services. Mitigation has been applied, and engineering teams are clearing resolver cache to fully mitigate the issue. Most services are showing recovery. This message was last updated at 22:10 UTC on 02 May 2019"

GoFormz is beginning to experience an intermittent return of services. We are continuing to monitor this situation as Microsoft Azure's performance improves. Please reference the Microsoft Azure status page for more information: https://azure.microsoft.com/en-us/status/

Here is the latest from the Microsoft Azure Status Page: "Customers may experience intermittent connectivity issues with Azure and Microsoft services. Engineers are investigating DNS resolution issues affecting network connectivity. Connectivity issues may affect the availability of Compute, Storage, and Database services, and some customers may be unable to file support requests. More information will be provided as it becomes available. This message was last updated at 21:10 UTC on 02 May 2019" Please refer to the Microsoft Azure Status page for more information: https://azure.microsoft.com/en-us/status/

Here is the latest update from Microsoft Azure Support: "Engineers are aware of intermittent connectivity issues with Azure Services. More information will be provided as events warrant." Please refer to the Microsoft Azure Status page for more information: https://azure.microsoft.com/en-us/status/

Starting at approximately 12:57 PM PDT, GoFormz started experiencing an outage in services provided by Microsoft Azure for our mobile and web platforms. GoFormz sync and other network services are impacted by this outage. Here is the latest update we have from Microsoft Azure Support: “Engineers are investigation connectivity issues with Azure Services. More information will be provided as it becomes available.” - Twitter (https://twitter.com/AzureSupport/status/1124046510411460610)

From approximately 8:38am PDT to 8:50am PDT on Monday, April 29th, GoFormz Workflow and API services experienced degraded performance. GoFormz engineering performed standard troubleshooting steps and returned the services to normal operations. GoFormz will continue to monitor these services.

This incident has been resolved.

As of approximately 4:45pm PDT, all functions are operational and performing normally. While we are continuing to monitor this service, please contact the GoFormz support team if you experience issues surrounding the functions mentioned below.

We are continuing to work on a fix for this issue.

We are currently experiencing an outage with our communication service on our mobile and web platforms. This primarily impacts form completion events, form transfer events, workflows, PDF generation, and outgoing webhooks.