Historical record of incidents for Codetree
Report: "Web service outage"
Last updateAWS Fargate changed the way containers share volumes within a task. This change in behavior caused a deployment failure along with failures on rollback. The solution required redesigning the container infrastructure to not require shared volumes
The issue has been identified and currently being fixed
Web service became unstable and no longer accessible
Report: "Search / Indexing degraded"
Last updateresolved
A fix has been promoted and looking to see if performance improves
Search and Indexing services are degraded and we are working to fix this issue. When data is saved / updated, you may receive an error, but in most cases, all data is persisted (but not indexed).
Report: "Slow Performance and Rendering failutes"
Last updateThis incident has been resolved.
We are currently investigating slow performance with Codetree site.
Report: "Burndown Activities unavailable"
Last updatethis incident is resolved and ongoing monitoring is in place
The issue with burndown activities data has been identified and is being resolved
Report: "Degraded Experience"
Last updateThis incident has been resolved; We'll continue to monitor to ensure all clients are back to normal
We are currently investigating a degradation of performance and partial outage today.
Report: "Github authentication is degraded"
Last updateGithub resolved their outage as of 5:01 UTC
continuing to monitor Github for a resolution to their performance issues: https://www.githubstatus.com/incidents/4sh7f76nh5gq
Github authentication and API services are degraded due to Github itself currently experiencing issues
Report: "Rendering some issues is failing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring results
We have identified the problem with a recent release and are working on a fix.. We hope to have something within the next hour or so
Some community members are having difficulty opening some issues due to a rendering error
Report: "Intermittent Performance Issues"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
We are currently experiencing issues with our caching and client side notifications, overall degrading the experience. We have identified a couple of the issues and working to get fixes in place as soon as possible
Report: "Rebuilding Search Indices"
Last updateThe index issues have been resolved
The issue has been identified and the index is being rebuilt
We are currently experiencing some issues with search; The search indices are being rebuilt
Report: "GitHub Service is down"
Last updateGitHub seems to be running smoothly again
GitHub API and other services are currently degraded and unreliable for usage
Report: "503 error affecting a fraction of accounts accessing the issue list view"
Last updatefix has been pushed and issue has been resolved
We are continuing to work on a fix for this issue.
There seems to be an issue with a recent change to the issues view for a handful of accounts. If you run into this issue, you should still be able to access your project view the board view. https://codetree.com/projects/:project_id/board
Report: "Caching server failure"
Last updateThis incident has been resolved.
Caching services restored and full connectivity is available again
A caching server failed leading a small portion of users to not be able to access the site. We are restoring service now
Report: "Search is not returning desired results"
Last updateSearch Indexes have all been refreshed and operations are back to normal
Search Indexing server failed due to an AWS us-east event today. New Search Indexing instance is online and indexes are currently being rebuilt
We are currently investigating this issue
Report: "flapping instance"
Last updateThis incident has been resolved.
We are currently investigating intermittent 500 gateway issues
Report: "Website Outagage"
Last updateconnectivity has been restored; access will continued to be monitored for the next couple of hours
resolved connectivity failure
We are continuing to investigate this issue.
We are currently experiencing intermittent access to website. investigation ongoing
Report: "Delays syncing from GitHub"
Last updateWe're back up and issues are syncing from GitHub normally again. You may see a slight delay as we process the backlog.
We are experiencing some delays syncing issues from GitHub. We are investigating the cause of the issue.
Report: "GitHub API outage is impacting Codetree"
Last updateGitHub has "addressed the source of the exceptions and are continuing to monitor as the site recovers." At Codetree we're doing the same.
GitHub is experiencing an API outage that is causing occasional errors with Codetree. Until GitHub resolves their outage Codetree service won't be 100%.
Report: "GitHub API Slowness"
Last updateGitHub API slowdown has been resolved.
GitHub is experiencing API slowness which is affecting Codetree performance.
Report: "Outage"
Last updateThe outage has been resolved and we're back up.
The issue has been caused by an outage with the company hosting our Redis instance. We've contacted support and will update when we know more.
We are experiencing an outage right now. We're investigating and will update as soon as we know more.
Report: "Outage"
Last updateThe incident has been resolved and all systems are back online.
We've identified the cause and are working towards a resolution.
Report: "Slowdown"
Last updateWe've resolved the slowdown and performance should be back to normal.
There is a slowdown processing issues created on GitHub and with general web performance. We have identified the issue and are investigating a resolution.
Report: "GitHub API Outage"
Last updateThere was a GitHub API outage from 12:08pm PT to 12:54pm PT. This may have prevented data entered into Codetree from being saved to GitHub. According to GitHub's status page, the outage has been resolved and everything is operating normally: https://status.github.com/messages/2016-12-06
Report: "Issues with DNS resolution"
Last updateBased on the latest update from from DynDNS this issue has been resolved.
The DNS resolution mitigations Heroku has made seem to have corrected the issue. We'll continue monitoring.
We have confirmed an outage with our DNS provider, DynDNS. The issue is intermittently affecting DNS resolution for codetree.com. The issue is being worked on by both DynDNS and Heroku. You can track status here https://status.heroku.com/incidents/965 and here https://www.dynstatus.com/incidents/nlr4yrr162t8.
Report: "Database maintenance"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "GitHub outage"
Last updateThis incident has been resolved.
GitHub is experiencing a major network outage https://status.github.com/
Report: "GitHub API issues"
Last updateGitHub is continuing to monitor. We are no longer seeing API timeouts on our end.
The GitHub API is experiencing problems. Updates here: https://status.github.com/
Report: "Requests timing out"
Last updateThis incident has been resolved.
Performance is stabilizing, but we are continuing to monitor.
Timeouts seem to be easing, but performance is still degraded.
Our database provider appears to undergoing a service disruption.