Historical record of incidents for HappyCo
Report: "Potential service disruption - Reporting"
Last updateAn upstream outage in our cloud services provider is causing an interruption in reporting and report generation. We are monitoring.
Report: "Intermittent issues loading inspections and tasks on the HappyCo mobile app"
Last updateWe are currently investigating this issue.
Report: "Intermittent Failures"
Last updateThis incident has been resolved.
All services are operational. We are investigating potential DNS issues.
We are currently investigating reports of intermittent errors and failed page loading.
Report: "Due Diligence Service Requests are Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Standard requests should be sent to dd@happy.co and include the DD Order name during this delay. Once the connection is restored, all delayed Due Diligence Service Requests will be re-synced with our ticketing system.
Due Diligence Service Requests are not moving into HappyCo’s ticket queue. This may delay our response to customer Due Diligence requests. Once the connection is restored, all delayed Due Diligence Service Requests will be re-synced with our ticketing system.
Report: "Yardi Integration Service Outage"
Last updateWe have accelerated import and export jobs to catch up to a successful state across all queues, with a high rate of success. With the license in place, confirmed bidirectional data sync, and jobs caught up, we are marking this issue resolved.
We are continuing to monitor for any further issues.
We have deployed the updated Yardi license and verified our two-way Yardi integration is online and syncing as expected. We're now working through accelerating the data sync process to ensure data is back to immediate push and 15-minute pull.
We have received an updated license from Yardi and preparing a deploy with the new value. After we have released the new license, we will initiate a sync of all data.
The Yardi license we use to perform data syncs (data import and export) has expired, resulting in no data transfer. No other PMS integrations are impacted. We have escalated to Yardi's customer support and are currently working with their support team to resolve the license issue. Once we have resolved the license expiration, we will initiate a re-sync of all data.
Report: "Issues with Work Assignment Board"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue is now resolved
We have found the cause for the issue and are working on deploying a solution.
We are currently investigating this issue.
Report: "Investigating intermittent degraded performance"
Last updateSystems are restored and healthy. We are continuing to monitor.
We are continuing to monitor for any further issues.
Applications and services are restored and being monitored. Task-related emails are suspended. We are working to quickly restore service.
We are working on restoring interrupted services.
Services are recovering and we are continuing to monitor system health.
We are continuing to investigate this issue.
We are investigating possible performance degradation in some services.
Report: "Production Issues with Property"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Services are restoring and we are monitoring.
We are currently investigating an issue impacting Property related features (e.g. Property Selector).
Report: "Property provisioning unavailable"
Last updateThis incident has been resolved.
We have identified the cause and applied a fix. We are monitoring the results.
Property provisioning is currently unavailable. We are investigating and will provide an update shortly.
Report: "Task Creation Category Dropdown Misplaced"
Last update**Summary** A software change was deployed that introduced a bug impacting task creation. **Impacted Systems** Task creation via the Web UI was unusable due to a positioning bug that pushed category selection outside of the browser viewport. **Resolution** A patch was deployed effectively reversing the offending change, restoring functionality. **Timeline of Events** 1600 UTC: Issue reported 1841 UTC: Fix developed and tested 1916 UTC: Fix deployed and confirmed in staging 2010 UTC: Hotfix deployed to production resolving issue
This incident has been resolved.
We've released a fix that addresses the mis-positioned category dropdown within Tasks.
We've identified the changes in question, related to an upgrade of two external dependencies, which caused the rendering placement bug. We are in the process of releasing a hot-fix rolling back the version, which we've confirmed addresses the issue, with a resolution ETA of 3:15pm ET / 12:15pm PT.
We're reviewing recent code changes to identify the root cause of the issue.
We have received reports of the category select dropdown displaying in the lower left-hand side of the screen, resulting in UX issues creating new tasks.
Report: "Call Management After Hours and On Call Schedule unavailable"
Last updateThe service disruption on the HappyCo platform has been resolved.
Report: "Planned maintenance and downtime on Optigo Platform"
Last updateThis incident has been resolved.
Maintenance has been completed and we are monitoring the results.
We have commenced maintenance on the Optigo platform. The HappyCo Maintenance platform and all other systems are unaffected.
Report: "Scheduled Maintenance"
Last updateThis incident has been resolved.
Maintenance has been completed, and we are monitoring the results.
We are conducting maintenance. Some services may be affected, including: * Happy Force ticket triaging (After Hours calls not affected) * Asset Management * Property Provisioning
Report: "Scheduled downtime"
Last updateThis incident has been resolved.
Maintenance has been completed and we are monitoring performance
We are progressing with scheduled downtime for server maintenance. HappyCo services are temporarily unavailable.
Report: "Manage experiencing degraded performance and extended load times"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Unexpected Database Downtime"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing high disk usage within the database serving Manage.
Report: "Call Complete - Unauthorized Responses"
Last updateWe have confirmed the fix on production successfully resolved the 401 Unauthorized error. All interactions within Call Complete in Manage should be working as expected. Again, this was a staff-only facing issue. All IVR services went unaffected.
We've validated the hotfix is working in a staging environment and are releasing shortly.
We've identified a solution and are working to release a hotfix to address this issue.
Call Complete is currently experiencing a partial outage for property staff attempting to access via Manage. Call Complete standalone customers are unaffected. Property operations, IVR interactions, and tasks creation is unaffected.
Report: "Yardi Integration Service Outage"
Last updateWe have successfully re-run outstanding failed jobs.
We have completed working through asset import/export as well as task export failures, and now moving onto task import failed jobs.
We are continuing to work through the backlog of previously failed API requests and making steady progress.
We've implemented a fix and are currently working through the backlog of failed API calls to bring our data up-to-date across all Yardi instances.
This outage currently only affects Yardi; no other PMS integrations are impacted.
We are currently investigating this issue.
Report: "Call Complete Web Interface Unavailable"
Last updateThis incident has been resolved. Thank you for your patience.
A fix has been implemented, and we are monitoring the results.
We have identified the issue and are investigating.
Customers are unable to access features of Call Complete via the web. We are actively investigating this issue.
Report: "Degraded system performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing degraded performance across our API's and services. Customers will experience intermittent failures to requests to HappyCo systems.
Report: "Uncheduled maintenance"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Insights is experiencing degraded performance"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Insights is experiencing degraded performance"
Last updateThis incident has been resolved.
We are currently investigating this issue. Insights is currently out of sync and some dashboards may not be presenting live data.
Report: "Increased error rates"
Last updateThis incident has been resolved.
The upgrade has been completed and we are monitoring the results.
Upgrade is in progress - services will be unavailable briefly during this time.
We have identified a cause, and have scheduled a priority upgrade at 2:00am EST. During the upgrade, the system will be offline for approximately 15 minutes.
We are receiving reports of increased error rates across our services. We are currently investigating.
Report: "Report downloading issues"
Last updateThis incident has been resolved.
We are aware of an issue preventing users from downloading newly created reports. We have determined the cause of the issue and are working on a resolution. Please note that pre-existing reports are unaffected.
Report: "Insights is experiencing degraded performance"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "HappyCo Platform Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our teams are aware of the issue and are working to restore service as quickly as possible
Report: "Scheduled outage - planned maintenance"
Last updateMaintenance has been completed. All services are restored
We are continuing to investigate this issue.
HappyCo services are currently down to allow for planned maintenance to be completed. Services will be restored shortly.
Report: "RealPage integrations are down."
Last updateThis issue has been resolved.
Report: "Yardi integrations are down."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Unable to share properties when using business inspection templates"
Last updateSome customers were experiencing issues with templates and are unable to assign business templates to properties. The issue was limited to a small subset of customers with Business Templates enabled. This incident has been resolved. We’ve implemented the solution, and our customers are no longer experiencing this issue. We’re really sorry to have held up customers today!
Some customers are experiencing issues with global templates and are unable to assign new templates to properties. We're aware of the issue and are working on it urgently. This does not affect any templates that have already been assigned and does not affect creating and completing inspections. We're really sorry to be holding you up today! Please know our teams are working hard to get everything up and running, and we will update you in 1 hour with the latest information.
Report: "Mobile App"
Last updateThis incident has been resolved.
Companies that have been migrated to Global templates are unable to start scheduled inspections on their devices
Report: "Potential service disruption"
Last updateThe service disruption on the HappyCo platform has been resolved.
We are continuing to investigate this issue.
Dashboards in Happy Manage may not be working
We've seen a potential disruption in the HappyCo platform. We're investigating it!
Report: "Problem opening inspections in mobile apps"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
A few customers may be unable to open scheduled inspections on their mobile device, receiving an error message: "infrastructure. network error 0"
Report: "Potential service disruption"
Last updateThe service disruption on the HappyCo platform has been resolved.
We've seen a potential disruption in the HappyCo platform. We're investigating it!
Report: "Delayed delivery of some email"
Last updateThis issue is fully resolved.
The fix to the root cause has been deployed, and this will be monitored over the next week to ensure that no further issues arise.
The root cause has been identified and a fix will be rolled out soon. In the meantime the backup solution is being monitored to ensure email operations contribute as normal.
Emails are now being processed and sent out. We are investigating the root cause of issue.
We are investigating the issue.
Report: "Temporary outage"
Last update20 minutes down - error connecting to primary database
Report: "Temporary Outage"
Last updateWe had a temporary outage this morning due to a database issue. It has been resolved.
Report: "Intermittent issues with integrations"
Last updateIssue has now been resolved
Engineering have have applied a fix and we are now monitoring all integrations
We are currently investigating reports of issues with some of our integrations. We will update as more information available.
Report: "Gray screen appearing on login to management portal"
Last updateRoughly a 5 minute outage due to an incorrect deployment
Report: "HappyCo website outage"
Last updateOur primary website happy.co experienced a 12-hour outage overnight. We've made some changes and added additional monitoring that should help prevent this occurring again. This outage did not affect any of our mobile or web applications (manage.happyco.com).
Report: "Degraded Performance"
Last updateEngineering confirm issue now resolved
Performance has been restored and engineering are monitoring the situation
Engineering team are investigating an incident with parts of our service.
Report: "Reduced Performance on Happy Manage"
Last updateThis incident has been resolved.
An update to our backend has been released. Engineering are continuing to monitor performance of all systems.
Engineering have identified performance issues with some parts of Happy Manage and are working on a fix. Customers may experience delays on page loads and potentially some timeouts. If affected, continue trying, or return later, if possible. Please monitor this page for updates.
Report: "Intermittent Loading Issues"
Last updateWe've rolled back our latest release which caused some intermittent loading issues.
Report: "Temporary Outage"
Last update1 minute outage due to connection issue
Report: "Dashboards unavailable"
Last updateIssue with dashboard vendor was resolved
We are currently investigating this issue.
Report: "Temporary Outage"
Last updateThis incident has been resolved.
A connection issue caused a temporary outage of around 5 minutes. We're monitoring now.
Report: "Dashboards are temporarily unavailable"
Last updateA fix has been deployed. We'll be monitoring for 24 hours to ensure all dashboards are working as expected.
A fix is being prepared for deploy.
The issue has been identified and a fix is being implemented.
Report: "Temporary Outage"
Last updateA slow down of request processing this morning required us to make upgrades to our infrastructure resulting in around 3-4 minutes down time.
Report: "Routing Error"
Last update100% message delivery failure to our upstream resulted in around 15 minutes of downtime.
Report: "Photo uploads will not complete"
Last updateThe AWS S3 issues have been resolved and all our services appear to be running smoothly again.
Existing photos and PDFs are available again. Photo uploads and PDF creation is still unavailable.
PDF and photos still unavailable due to upstream outage.
Our upstream photo host is currently having issues. Service should resolve normally once they've resolved their issues. https://status.aws.amazon.com
Report: "Temporary Outage"
Last updateThis incident has been resolved.
We experienced a temporary outage lasting approximately 5 minutes during a routine upgrade.