Historical record of incidents for OpenGov
Report: "Google outage affecting 811"
Last updateSubject: Intermittent Gmail Outage Affecting 811 Ticket Delivery We are currently experiencing an intermittent Gmail outage impacting 811 integrations within Enterprise Asset Management. This is part of a broader, global issue with Gmail. Google is actively working to resolve the problem. You can view the latest status updates here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW Important Action Required: Please manually check your 811 provider for any SEV-1 (high severity) issues, as email-based ticket delivery may be delayed or disrupted. At this time, we cannot guarantee successful delivery of 811 tickets via email. We will keep you updated as we learn more. Thank you for your patience and understanding. (edited)
Google Workspaces has recovered according to google. But based on this link there is still down services we think might affect proper gmail usage through their API https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
We are currently investigating this issue.
Report: "Permitting & Licensing : Intermittent Slowness"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Permitting & Licensing : Intermittent Slowness"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Permitting & Licensing : Site Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Permitting & Licensing : Site Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Permitting & Licensing: Potential display issues"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Reporting & Transparency: Settings are Inaccessible"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Permitting & Licensing: Potential display issues"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Reporting & Transparency: Settings are Inaccessible"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Open Data: Open Data Portals are Inaccessible"
Last updateThis incident has been resolved.
Server 503. All portals unavailable
Report: "Open Data: Open Data Portals are Inaccessible"
Last updateThis incident has been resolved.
Server 503. All portals unavailable
Report: "Permitting & Licensing: Site slowness"
Last updateThis incident has been resolved.
Resolved
We are continuing to investigate this issue.
We are currently experiencing site slowness, we are investigating.
Report: "Dashboards are not appearing for Customer"
Last updateThis incident has been resolved.
Dashboards are currently not appearing for customer. Our Engineering team is working to put in release to fix this.
Report: "Community feedback has degraded in production"
Last updateThis incident has been resolved.
Users will see a page poorly formatted due to missing css styles and possibly not be able to interact with it. This impacts all customers
Report: "Permitting & Licensing: Site Inaccessible"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Permitting & Licensing: Intermittent Slowness and Performance Degradation Issue"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Cartegraph Asset Management - ArcGis is Experiencing an Issue Impacting Login"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
ESRI ArcGIS has implemented a fix and Cartegraph Asset Management has now returned to service.
The issue has been identified and a fix is being implemented.
Cartegraph Asset Management customers that are configured to use ESRI ArcGIS authentication will not be able to login. We have opened a case with ESRI ArcGIS and will continue to provide updates.
Report: "Cartegraph Asset Management -- Reporting Service is Unavailable"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Cartegragh Asset Management reporting has returned to service.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Cartegraph Asset Management -- Reporting Service is Unavailable"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Reporting has returned to service. There was an issue with AWS US-East-1.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Reporting & Transparency: Integration Data Ingestion Failure"
Last updateThis incident is resolved and all data integrations are successful.
We are targeting to have all integrations updated by 12 PM PT.
We have been able to successfully re-run and import data for a subset of customers. We will continue to provide updates on our progress throughout the day.
We are continuing to manually re-run the integrations of data.
We have identified an intermittent issue with the data ingestion APi. To mitigate the issue, we are manually re-running the integration of the data.
OpenGov has detected that some our integrated datasets failed to update starting the morning of 9/3/24. This will impact data in Budget & Planning and Reporting & Transparency. The Engineering team is currently investigating the issue.
Report: "Cartegraph Asset Management: Reporting Issue"
Last updateThis incident has been resolved.
Reporting has returned to service.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently experiencing issues with reports, we are actively investigating.
Report: "Access Control Issue"
Last updateWe are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
All systems have returned to service.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "All Open Data Portals are Unusable-Internal Server Error"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Uploading and viewing attachments"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue with uploading and viewing attachments, not all clients will experience this issue.
Report: "Budgeting & Planning: Performance Degradation"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and the performance of the Budgeting & Planning application is operating optimally since 7:15 AM PT. We will continue to monitor the system.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Open Data: Open Data Portals are Inaccessible"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
This incident is now upgraded to a major outage as all portals are down.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Cartegraph Asset Managment: Portal Login Users are unable to access maps"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
The fix has been applied and we have Returned to Service as of 5 PM PT.
A fix is being targeting for release in the next 2 hours to resolve the incident.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Permitting & Licensing: Intermittent Slowness and Performance Degradation Issue"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Budgeting & Planning: Minor Rounding Discrepancy"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue. Knowledge Article - https://opengov.my.site.com/support/s/article/1ccf421d-3fbe-43e2-bd0c-f915e523eab3
We have identified and are currently investigating a small subset of customers that have been affected by a rounding issue in itemization totals when the copy budget process was used to create a budget.
Report: "Cartegraph Asset Management: Reporting Issue"
Last updateThis incident has been resolved.
The ability to export reports to PDF is now operational as of 11:30 AM PT. Note: Users may be required to clear browser cache for the PDF export to function.
We have restored service for running reports and sending reports via automations/emails. The ability to export reports is currently unavailable. You can print and select PDF as your output device. This issue is with Engineering and we are working to resolve the issue.
We are continuing to investigate the issue. We will provide another update tomorrow morning.
We are experiencing an issue with reporting. Customers will experience a "System Error: Cannot process command..." error message. Engineering is investigating and we will continue to work around the clock until the issue is resolved. We will provide updates every 4 hours during business hours on this Status Page.
Report: "Permitting and Licensing: FTP Site Inaccessible"
Last updateThis incident has been resolved.
All export integrations have been manually re-run. This issue is now resolved.
We are continuing to monitor for any further issues.
We have identified that a subset of customers Integrations did not run during this time. MAT, GIS Flag integrations will update data on the next scheduled run Export integrations are being manually re-ran, customers may see a delay in receiving export files.
We are continuing to monitor. Knowledge article: https://opengov.my.site.com/support/s/article/2cb16519-3d4b-4d7b-b048-48e731bec947
A fix has been implemented and we are monitoring the results.
Engineering will continue to monitor. Access to FTP site has been restored.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue for accessibility to integration.viewpointcloud.com
Report: "NextGen Reports: Intermittent 'Failure to Load' Error When Accessing Reports"
Last updateThis incident has been resolved.
A fix has been released and we are currently monitoring to ensure resolution
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "OpenGov Permitting & Licensing Performance Update"
Last updateWe have completed numerous enhancements in the past month following performance issues in early November. As a result, both users and internal monitoring systems are showing notably improved query and workflow performance. Although our recent fixes have been very effective, we will continue to work to improve even further. We have additional enhancements planned through the end of the calendar year and into 2024. These will increase resiliency for our users and provide OpenGov with advanced visibility to identify future issues before they arise. Based on these improvements and over 2 weeks of consistent performance, we are closing the related cases today. If your issue is still unresolved, or if you have other ongoing performance concerns, please open a new support ticket. This will help document any ongoing customer issues for the OpenGov support team. As mentioned in an earlier update, OpenGov has shifted our Permitting & Licensing roadmap priorities to a focus on performance, scale, and quality. We have recently stepped up our efforts to resolve customer-reported defects, and we are focused on small, carefully qualified improvements to our suite that will be meaningful for your community. We are extending current timelines for the release of larger upcoming features. We acknowledge that this will impact technology planning for some communities, but we believe that the focus of our stability efforts is the right thing for our customers. For the latest information on our roadmap, we encourage you to join our quarterly product highlights webinar or regional user groups. We remain committed to partnering with you to bring the cloud to our nation's governments. Our intention is to continue rebuilding your trust, delivering value, and advancing OpenGov’s mission to power more effective and accountable government. Thank you for your continued partnership.
This is an update on our work to improve your communities’ overall experience with OpenGov’s Permitting & Licensing suite. We have made over a dozen performance enhancements in the last 14 days. As a result, both users and our internal monitoring systems are showing improved response times. However, we still see opportunities for improvement. We are planning to implement several additional enhancements by December 1st that are expected to restore performance for our Permitting & Licensing users. In addition, OpenGov is shifting roadmap priorities with a primary focus on performance, scale, and quality. As a result, we are extending current timelines for the release of upcoming features, including pausing the rollout of the enhanced Inspection Scheduling and Flexible Workflow capabilities. We acknowledge that this will impact technology planning for some communities, but we believe that focusing our efforts on the stability of the platform will be the most beneficial for our customers. We are committed to listening to our customers and partnering with all of you to bring the cloud to our nation's governments. We are very grateful for your partnership and remain focused on using these recent events to rebuild trust, deliver more value, and to continue to power more effective and accountable government.
Performance monitoring continues to show positive results for a large amount of customers from Friday afternoon through Thursday. We have received numerous validations from customers that performance has improved from last week. Some customers have stated that performance has improved, but still could be better. We remain committed to improving the performance of the system.
Performance monitoring continues to show positive results for a large amount of customers from Friday afternoon through Wednesday. We have received numerous validations from customers that performance has improved from last week. Some customers have stated that performance has improved, but still could be better. We remain committed to improving the performance of the system. We will continue to monitor the system performance and provide another update tomorrow.
Performance monitoring continues to show positive results for a large amount of customers from Friday afternoon through Tuesday. We have received numerous validations from customers that performance has improved from last week. Some customers have stated that performance has improved, but still could be better. We remain committed to improving the performance of the system. A configuration change was completed this morning and that is yielding more performance gain to the end user experience. We will continue to monitor the system performance and provide another update at 1 PM PT (4 PM ET).
Performance monitoring continues to yield positive results from Friday afternoon through Tuesday afternoon. We have received numerous validations from customers that performance has improved from last week. We will continue to monitor the system performance throughout the day and provide another update at 1 PM PT (4 PM ET).
Performance monitoring continues to yield positive results from Friday afternoon through Tuesday morning. We have received numerous validations from customers that performance has improved from last week. We will continue to monitor the system performance throughout the day and provide another update at 10 AM PT (1 PM ET).
Performance monitoring continues to yield positive results throughout all day Monday. We have received numerous validations from customers that performance has improved from last week. We will continue to monitor system performance tonight and provide another update tomorrow at 6 AM PT (9 AM ET).
Performance monitoring continues to yield positive results throughout Monday morning and now the afternoon. We have received numerous validations from customers that performance has improved from last week. There are some long running requests that may experience slower than normal response time. If you encounter this experience, it is recommend to click the browser refresh button. We will continue to monitor the system performance throughout the day and provide another update at 1 PM PT (4 PM ET).
Performance monitoring continues to show positive results from Friday afternoon through Monday morning. There are some long running requests that may experience slower than normal response time. If you encounter this experience, it is recommend to click the browser refresh button. We will continue to monitor the system performance throughout the day and provide another update at 10 AM PT (1 PM ET).
We continue to see lower rates of failures since the configuration change last night. Users should start to see improvement in the performance of the system. Note: it is also recommended for users to clear their browser's cache, cookies, and browsing history. How to clear cached files and cookies in your Internet Browser: https://opengov.my.site.com/support/s/article/33618cd5-cbdb-40aa-b8cf-78ac65a7e1d8 We will continue to monitor the system performance throughout the weekend and provide an update on Monday at 6 AM PT (9 AM ET).
We continue to see lower rates of failures since the configuration change last night. Users should start to see improvement in the performance of the system. Note: it is also recommended for users to clear their browser's cache, cookies, and browsing history. We will continue to monitor the system performance throughout the day and provide another update at 1 PM PT (4 PM ET).
We have been monitoring web requests success/failure and are seeing much lower rates of failures since the configuration change last night. Users should start to see improvement in the performance of the system. We will continue to monitor the system performance throughout the day and provide another update at 10 AM PT (1 PM ET).
We have completed the configuration change to increase the worker services. We will continue to monitor system performance tonight and provide another update at 6 AM PT (9 AM ET).
We are testing a change that includes increasing our worker services. We have seen positive results in our quality assurance testing. Once it has fully passed regression testing it will be deployed with no downtime.
Overnight on November 7th, we completed a hardware upgrade and ran simulation and diagnostics which helped improve performance. We also added additional monitoring to ensure the measures we’re taking result in fully restored performance. We take this very seriously with an all-hands-on-deck approach, including our cloud provider. We will continue to work diligently until the issue is fully resolved.
Permitting & Licensing is experiencing periodic slowness.
Report: "Opengov 3rd Party Vendor (Salesforce) | Experiencing Degradation Impacting our Customer Support tooling for Chat Messaging, Case Generation, Voice services"
Last updateSalesforce has reported all systems are fully operational and have resolved the incident.
A fix has been implemented and Salesforce is monitoring the results.
Salesforce is continuing to work on these incidents.
Issue: On December 4, 2023, at 14:49 UTC the Salesforce Technology team noted a feature degradation issue affecting Messaging, Case Submission, and Voice services on multiple North American instances. Their technical teams have identified a potential trigger for this incident and are actively working to restore services as quickly as possible. Impact: Multiple instances in North American data centers may experience Messaging Channels sessions failing, leading to the inability to send and receive messages or create new sessions. Customers may also encounter intermittent failures within Case Submission and Voice services.
Report: "Cartegraph Asset Management: Investigating a Login Issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Permitting & Licensing: Workflow Steps not Activating"
Last updateThis incident has been resolved.
We have validated the issue is resolved for customers.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Permitting & Licensing: Intermittent Slowness and Performance Degradation Issue"
Last updateWe are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We have identified the issue and the system is back to normal operations.
Knowledge Article: https://opengov.my.site.com/support/s/article/1dcef9d3-48d7-419a-a4de-cdfbb119a62b
We are investigating performance degradation in the Permitting & Licensing application.
Report: "Permitting & Licensing: Intermittent Public Site Inaccessible"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are investigating where some users get a blank page when accessing Public Site.
Report: "Permitting & Licensing: Intermittent Slowness"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Permitting & Licensing: Error Loading Record Details"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Data Integration Incident | Integrations Failed due to Rest API Service Endpoint Error"
Last updateThis incident has been resolved.
Engineering teams have implemented a fix on 9.30.2023 and are continuing to monitor.
We are continuing to investigate this issue.
OpenGov has detected that some our integrated datasets failed to update starting the morning of 9/22/23. The Engineering team is currently investigating the issue.
Report: "Procurement - Platform User Not Found Error"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Engineering team has identified the incident regarding Platform User Not Found and will be performing a hotfix at 7pm ET 9/29/2023. Please note: there will be a brief interruption of service during this time. Thank you for your patience.
We are continuing to investigate this incident and will provide an update. Thank you
We are continuing to investigate this issue.
Engineering team is currently investigating issue.
Report: "Permitting & Licensing: Site Inaccessible"
Last updateAll systems have been operational since yesterday.
The Permitting and Licensing application has Returned to Service as of 12:15 PM PT (3:15 PM ET). We continue to monitor for the remainder of the weekend.
Systems are in stages of coming back online.
Microsoft Azure continues to work to repair services.
Microsoft Azure is engaged and working to repair services.
Microsoft Azure is experiencing an issue which is impact the Permitting & Licensing application.
We are currently investigating this issue.
Report: "Permitting & Licensing: Intermittent Slowness"
Last updateOur cloud vendor (Microsoft Azure) was experiencing an issue that caused the intermittent slowness in the Permitting & Licensing application. As of 1 PM PT (4 PM ET) yesterday the issue was resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and the system performance is improving.
We are currently investigating this issue.
Report: "Control Panel - Visitor Analytics Page is Displaying Blank Page"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Workaround: Users can bypass this issue to access Budget & Planning and Reporting & Transparency by using the saved direct URL link.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Permitting & Licensing: Intermittent Page Loading Issue"
Last updateCustomers have confirmed all systems are operational.
Microsoft identified an issue that has been resolved as of 1:30 PM PT (4:30 PM ET).
We have escalated to Microsoft Azure and working with their Engineers.
We are currently investigating this issue.
Report: "Budget & Planning: Budget Builder Intermittent Loading Issue"
Last updateThe Budget & Planning application has operated within normal limits for 4 hours and this issue is now closed.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Financials: On Prem Email Service not Functioning"
Last updateACH for Accounts Payable emails are now functioning and the issue is resolved.
We are in the process of applying a hotfix for Utility Billing (bills, bill reminders, and cutover notices) and Payroll (check stubs and direct deposit notices) emails. ACH for Accounts Payable emails are still not functioning and are anticipated to be fixed on Friday.
The issue has been identified and a fix is being implemented.
We are currently investigating the issue that customers can not send emails for On Prem Financials.
Report: "Permitting & Licensing - Some Users Unable to Login"
Last updateThis incident has been resolved.
A fix has been applied to impacted users.
We are continuing to work on a fix and once code is complete we will begin testing before we rollout a resolution.
The issue has been identified and a fix is being implemented.
We are currently troubleshooting the issue. Knowledge Article https://support.opengov.com/hc/en-us/articles/16005024422811-5-31-2023-Incident-Some-Users-Unable-to-Login
Report: "Budget & Planning: Stories content not appearing when accessing a Story"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Stories has Returned to Service as of 3:40 PM PT.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Permitting & Licensing - Global Search Returning No Results"
Last updateThis incident has been resolved.
The Permitting & Licensing search service has operated within normal limits for 17 hours and this issue is now resolved.
Search performance is operating within normal operating limits after the maintenance. We will continue to monitor throughout the day.
We will be performing maintenance on the Permitting & Licensing suite from 5 PM to 5 AM PT (8 PM to 8 AM ET). During this maintenance period there will be degraded performance and brief interruptions. Maintenance Page: https://www.opengovstatus.com/incidents/whlf9pkbvzvg
Global search is now returning results with slight performance delays.
Microsoft Azure engineers are investigating the issue.
We are currently investigating this issue.
Report: "Permitting & Licensing: Intermittent Search Latency"
Last updateThe Permitting & Licensing search service has operated within normal limits for 36 hours and this issue is resolved.
The Search service is operating within normal limits and we will continue to monitor for the remainder of today.
A fix was implemented and search performance has improved. The Search service is operating within normal limits and we will continue to monitor for the remainder of today.
Microsoft Azure continues to work on a fix to improve Search Latency.
Microsoft Azure continues to work on a fix to improve Search Latency.
Microsoft Azure is working on a fix to improve Search Latency.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Permitting & Licensing: Delayed Login and Intermittent Accessibility"
Last updateThis incident is closed and customers have validated resolution.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
OpenGov and Microsoft Azure are investigating the issue.
Report: "Permitting & Licensing: Intermittent Slowness and Performance Degradation Issue"
Last updateMicrosoft Azure and OpenGov have confirmed that the system performance issue is resolved as of 3 PM ET.
We have a Severity A incident open and both PLC Engineering and Microsoft continue to troubleshoot the performance issue.
We are investigating performance degradation with Microsoft Azure. Knowledge Article: https://support.opengov.com/hc/en-us/articles/13253495170587-2023-02-27-Permitting-Licensing-Slow-Performance-Issue
Report: "Permitting & Licensing: Intermittent Slowness and Performance Degradation Issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Microsoft Azure is investigating the issue.
We are continuing to investigate this issue.
Knowledge Article: https://support.opengov.com/hc/en-us/articles/13247485265051-2023-02-27-Permitting-Licensing-Performance-Degradation-Issue
We are currently investigating this issue.
Report: "Proposals and Stories are not Displaying in Budgets"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.