Historical record of incidents for Jobylon
Report: "Updates taking long to be reflected on search"
Last updateThe issue has been resolved and the changes should be reflected in the search with little delay again.
We have identified the issue, where the queue managing the updates is being throttled. We have scaled up the infrastructure to clear out the queue.
We are currently experiencing a delay between an update and it being reflected on the search inside the app.
Report: "Searchindex out of sync"
Last updateSearchindex out of sync since the reindex queue was spiking.
Report: "Slow application"
Last updateSome inefficient queries were identified that was blocking the database. The queries were cancelled and we are looking into optimising the queries that were causing the issue.
We are currently investigating this issue.
Report: "Changes are not being reflected in the app"
Last updateThis incident has been resolved.
Updating changes to the search database is taking longer and backlog of changes are building up. No changes have been lost
We are currently investigating an issue where changes are not reflected in the application.
Report: "Database degradation"
Last updateThe incident has been resolved and all system are back to operational.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating issues with database performance.
Report: "Degraded search"
Last updateFixes rolled out and search is back to normal again
After a version update, re-index of some candidates in the search failed. There are also performance issues with the re-index, which we plan to fix with new DB indexes.
Report: "Applications and jobs not found in search"
Last updateAll search data is up to date again
The search database is up an running again, we will reindex the applications and jobs that were updated during the outage and all should be back to normal in a few minutes.
We are experiencing issues with our search database and are working to fix it. At this moment no applications or jobs can be found, but candidates are not stopped from applying to job.
Report: "Application and job search down in the app"
Last updateWe were experiencing issues with our search database, which had an impact on functionality inside the Jobylon app. No applications or jobs could be loaded in the app. It did not have any impact on job lists and candidates could still apply for jobs. The search database was down for roughly 40 minutes.
Report: "The jobylon application is unreachable"
Last updateIssue identified and resolved with the provider
Our platform provider (Heroku) is experiencing a major outage in the EU, which is causing all of the jobylon application to be unreachable. More information about the incident can be found here https://status.heroku.com/incidents/2359
Report: "Candidates stopped from applying when phone number was required"
Last updateTo prevent this from happening again, we will update our synthetic testing to include this scenario.
On the 6/1 candidates applying to jobs that required a phone number, was being stopped from applying. This was due to a JavaScript file only partially uploaded during the release process. When this was discovered, the faulty JavaScript file was re-uploaded and the issue was resolved.
Report: "Overloaded queue causing error messages to applicants"
Last updateTo prevent this from happening again, we have now updated our alarms that notify the engineering team when the queue is under a heavy load.
On 23/12 a faulty path was applied, which started spawning messages on our async queue. This kept on running through the 24-25/12 when it finally took down our queue. During the night between the 25/12 and 26/12 the queue was brought back up by our engineering team and a path with a fix was applied. Candidates who was applying during the period had sporadic issues and when the queue was brought down, all submitted applications was met with a error message. The applications was not lost and they could later be recovered.
Report: "Site Down"
Last updateThe whole application was down for 30 min due to a database issue. The database was running out of storage, which was quickly solved by scaling the storage and recovering the database.
Report: "The application does not work using Danish language"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Error when logging into the application"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Faulty Cut-e configurations"
Last updateA deploy was made with faulty Cut-e configurations (3:rd party assessment provider). This only affected a small number of clients using the integration.
Report: "Search not returning applications"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "CDN and DNS issues"
Last updateOur CDN and DNS was experiencing issues causing our site being not accessible. Read more about the Cloudflare outage here: https://blog.cloudflare.com/cloudflare-outage/
Our CDN and DNS experienced issues causing Jobylon not being accessible.
Report: "Redis instance moved"
Last update[07:54:00 UTC - 07:55:00 UTC] Redis instance moved causing a 1 minute downtime for logged in users.