Historical record of incidents for Simployer
Report: "Can't access the 'Contracts' module"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some customers are reporting that they can't access the 'Contracts' module. We're working on resolving the issue.
Report: "Due to an ongoing incident, some customers may experience increased latency. We are currently investigating the incident."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Issues in the main portal"
Last updateThis incident has been resolved.
Some users are currently experiencing issues on the main portal in Simployer Classic. We are investigating the situation at the moment.
Report: "Delayed results calculation in Employee Survey (&Frankly)"
Last updateThis incident has been resolved.
We are experiencing some issues with the result emails today, some delay might occur.
Report: "Issues with connections between some of our services"
Last updateThis incident has been resolved.
Our third-party provider is experiencing issues with the VPN tunnel to Azure. As a result, certain functions/modules are currently not responding. We are fully focused on finding a solution.
Report: "Handbooks experience performance issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Classic Tid login not working"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Trouble with serviceprovider Microsoft Azure"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
There is ongoing issue with Microsoft Azure region Norway East. This makes it impossible to log on to Capitech Tid (Classic).
Report: "Currently ongoing issue with Fortnox integration in Alexis HR"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results
We are investigating the issue.
Report: "Database connection issues in Employee survey (&Frankly)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating problems with connection to user snapshot database.
Report: "Connectivity issues with batch integrations"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are investigating a connectivity issue with the Simployer middleware service. The middleware service is used for batch integrations between Simployer HRM and 3rd parties. We are currently working on resolving the issue.
Report: "Slowness in the application"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We experienced some slowness in the application for a few minutes due to the release of a new version. This is now being rolled back, and we assume that it will resolve the issue.
Report: "All handbooks are temporarily unavailable. We are working to resolve the issue."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Issues with logging in"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing some issues with logging into our system. The teams are investigating this atm.
Report: "Handbooks are currently unavailable"
Last updateThis incident has been resolved.
The Handbooks are currently unavailable. We are working to resolve the issue.
Report: "Absence modules unavailable"
Last updateThe incident has been resolved! We are sorry for any inconvenience for those of you that were affected.
The issue has been identified. It is related to a combination of roles on some users in the leave and sick leave modules. A hotfix for the leave module has been released and we are working hard to fix the sick leave module.
Some of our customers are experiencing unstabilities with our moduels for leave and sick leave. We are working hard to identify and fix the issue.
Report: "Håndbøker er ustabile/utilgjengelige"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Når man prøver å åpne start siden til vår håndboks portal, så ender det i en «time out». Vi jobber med å få produktet stabilt igjen.
Report: "Webserver unavailable"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We restarted the server - seems to work. Investigating root cause now.
1 webserver is unresponsive. We are investigating.
Report: "Individual results loading issue in Engagement"
Last updateThis incident has been resolved.
We have investigated and are in process of fixing an issue with loading individual results in Engagement following an application update. Until resolved, customers may use the prior version of Individual results as a fallback.
Report: "My Capitech - unstable authentication"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified a problem, released a patch to some customers with this problem, and we will be monitoring this issue further on.
Some employees of some customers gets message "Autentiseringsfeil" or similar. We are currently investigating this issue.
Report: "Major system outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our systems are experiencing a major outage. We apologize for the inconveniance, and we are working to restore the systems as fast as possible. Status updates will be posted to this page, and it is possible to subscribe to updates and get notifications in your channel of choice.
Report: "My Support is currently unavailable"
Last updateThe incident has been resolved, and My Support is now available again.
My Support, the support system for case registration at the Support Center, encompassing both technical assistance for HRM and our Expert Services, is currently unavailable. We are actively working to address and resolve the issues.
Report: "Problems executing tests in LMS for some customers"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
A fix has been applied.
Report: "Issues with login to Talent during ~10min"
Last updateUsers where prompted to login even if they where already authenticated. This was due to a configuration-change in the application gateway. The incident was active for about 10min before the change was reverted.
Report: "Talent Public API Outage"
Last updateThe Talent Public API was failing due to the incorrect SSL related configurations in the Application Gateway. This caused the all requests to fail during this duration. When Identified, the incorrect configurations were applied which resolved the issue. **Steps taken to prevent this kind of issue in future:** * The processes and routines related to Application Gateway has been review and updated. * Alerts have been updated to address these types of issues as well.
Talent Public API was experiencing issue and all the requests to API was failing. The API was unavailable from 2024-02-23 14:00 to 2024-02-26 08:40
Report: "Few customers with ADFS SSO Login are unable to login Talent Portal"
Last updateThis incident has been resolved.
The certificates of the effected customers has been updated. This has resolved the issue for the effected customers.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Issue with results computation, results emails and insights"
Last updateWe have now been able to run the result and insights generation, and the result emails missing this morning are being sent now.
Answer service lost connection to mongoDB cluster. We have now restarted the services and will be re-running our results weekly calculations.
Due to an issue with results generation, results emails haven't been sent and insights are also impacted. We are investigating the issue.
Report: "Users with SimployerAccess Login are unable to login to Talent Portal"
Last updateThis incident has been resolved.
The issue has been Identified and a fix has been implemented.
Currently, the user with SimployerAccess Login are unable to login to Talent Portal. we are currently investigating the issu
Report: "Files do not load in the Deviation module"
Last updateThis incident has been resolved.
We have rolled back to the previous version, and files works again. We will monitor the situation
Files do not load and can't be added in the Deviation module. The team is investigating this.
Report: "Talent Portal & API: Outbound IP Address change"
Last updateThis incident has been resolved.
The Outbound IP Address for the Simployer Talent Portal (Web) has changed.
The Outbound IP Address for the Simployer Talent Portal (Web) will change Thursday, 2023-12-07 between 21:00 - 22:00. Due to this change, there is a possibility that end-user sessions might end unexpectedly. Consequently, users may need to log in again. So, it is recommend that users log out of their Talent Portal Web sessions before the scheduled change.
The Outbound IP Address for the Talent Public API will change 2023-11-23 at 12:00
Due to some technical issues, the Outbound IP address (for Talent portal & API) will not change this weekend. we will come back with a New Date. If your organization uses a custom domain for the Talent Web Portal, then you don't need to change your DNS record until further information. More information about the Outbound IP address change: https://support.simployer.com/se/simployer-hrm/release-notes/outbound-ip-address-change-for-talent-portal/
Report: "Talant Public API was Unavailable"
Last updateThe issue was due a configuration issue. _A Tenant Url was incorrectly configured_ which caused the following issues: * Due to this application was unable to start. * The errors related to application start up was not being logged correctly as well. **Steps taken to resolve the issue:** * Fix the incorrectly configured Tenant Url. * Update the code to for better error logging at the application startup.
Talant Public Api application was unable to start and was unavailable from 22:30 to 01:00 last night.
Report: "&frankly - Left side menu"
Last updateThis incident has been resolved.
We have now fixed the issue with the elements in the left side menu, and continue working on the performance issues.
It has been reported that some items in the menu that should be visible are not (old Individual Questions) and others that shouldn't be visible are (Pulses Old). There are also some minor performance issues being investigated (navigating from an older page to a new one - like IQ beta to Users and Groups). We will be addressing those issues during the day.
Report: "Incident in search - Handbooks"
Last updateThis incident has been resolved.
The search in Handbooks is not giving any results. We are working on a solution. You can reach all your content through table of contents.
Report: "Web-server Time & Plan with unplanned restart"
Last updateUnplanned restart of 1 webserver in Time & Plan. The restart took a few minutes (max 10). Server is now up and running.
Report: "portal.simployer.com is experiencing an outage"
Last updateThis incident has been resolved.
Our systems are experiencing an outage. We apologise for the inconvenience, and we are working to restore the systems as fast as possible. Status updates will be posted to this page, and it is possible to subscribe to updates and get notifications in your channel of choice.
Report: "Performance issues in Talent Portal"
Last updateThe issue has been resolved for now. A long term solution is also being implemented to address these performance issues. More information about this can be found here: https://www.simployer.se/support/simployer-hrm/cancellation-of-talent-portal-migration/ https://support.simployer.com/se/simployer-hrm/release-notes/outbound-ip-address-change-for-talent-portal/
A fix has been implemented and we are monitoring the results.
We are sporadically experiencing issues with performance in Talent portals.
Report: "Login issues"
Last updateToday has unfortunately been a hectic day in Simployer! ## Summary: Today many of our customers experienced a critical incident where some users were unable to log into our services, including the Simployer Portal. This incident severely impacted our users' ability to access our solution. Despite our monitoring efforts, we were initially unable to determine the root cause, leading to downtime and user frustration. ## Timeline: * **8:40**: First alert received from monitoring system. * **9:00**: We can verify that login is not working like it should and we are initiating the investigation. * **11:52**: Initiation of infrastructure changes to increase the number of machines. * **12:29**: Infrastructure changes were blocked due to issues with a 3rd party repo system. * **13:05**: 3rd party system is online again and login services are moved to new infrastructure and deployed. * **13:15**: Login API began to stabilize, showing improvement in responsiveness on new nodes. * **13:20**: The issue was resolved, and services became fully responsive on all nodes. * **13:37**: Incident status was updated to reflect the resolution. ## Root Cause Analysis: The incident appeared to be related to throughput issues due to high traffic on Monday morning. High traffic on Monday morning is not unnormal, but this Monday reached a request limit on a machine, which again overloaded other machines. We were not able to deploy new infrastructure as fast as we wanted due to an issue with a 3rd party vendor. ## Actions Taken: * Various troubleshooting steps were performed to restore service stability, including redeployments, app pool resets, and downscaling to working nodes. * Infrastructure changes were done to increase machine capacity. * The incident was closely monitored, and updates were provided to keep customers informed through our statuspage. ## Lessons Learned: * **Improved Monitoring**: Enhancements to monitoring systems are needed to proactively detect such incidents, potentially identifying the root cause more swiftly. * **Robust Redundancy**: Implement better redundancy mechanisms to ensure that critical services remain accessible even during unexpected downtimes or traffic spikes. * **Infrastructure Resilience**: Strengthen infrastructure to accommodate increased loads and ensure timely scalability during peak usage periods. We will spend the time ahead to mitigate on our lessons learned. We deeply apologize for any inconvenience this outage may have caused.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are experiencing login issues for some of our modules at the moment. Our team is investigating the issue.
Report: "Absence integration between Simployer HRM and Capitech Time & Plan is unstable"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Middleware integrations are unavailable"
Last updateThis incident has been resolved.
We have identified the issue with authentication towards SFTP and a fix has been deployed. We are monitoring the system for normal operation. We apologize for any inconvenience.
We are continuing to investigate this issue.
We are currently investigating an issue with middleware integrations depending on SFTP file uploads. Currently it is not possible to authenticate against the SFTP server. Customers without middleware integrations are not affected.
Report: "Simployer Login unstable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are receiving reports that logging into Simployer is currently unstable. The team is investigating the incident. We apologise for the inconvenience.
Report: "Performance issues in Talent Portal"
Last updateThis incident has been resolved.
The issue has been identified and a fix has been deployed.
We are sporadically experiencing issues with performance in Talent portals.
Report: "Simployer portal is down"
Last updateThis incident has been resolved.
We have identified the issue and provided a fix. All systems should be operational again, but we continue to monitor the situation. A sincere apology from us to those that were affected.
Simployer Portal is currently experiencing an outage. We are investigating the issue.
Report: "Performance issues in Talent Portal"
Last updateThis incident has been resolved.
We are experiencing instability in the environment for Talent Portal. We are currently investigating it.
Report: "Login issues"
Last update# Problem Description Between the 11th and 13th of April, we had transient incidents related to login in capabilities. This resulted in issues with accessing modules like e.g. Travel, Expense, Contracts, Vacation, and Handbooks for end users. * The first symptoms of the incident occurred on Tuesday, April 11th around 14:30. It lasted for 21 minutes according to our logs. * A second time the issue occurred was on Wednesday, April 12th at 8.03 ending 14:32. * The issue occurred last on Thursday, April 13th during two timespans - 7:59 - 9.02 and 10:29 - 10:37. Our team was responding and working continuously on the issue during the whole timeframe. # Root Cause and impact The root cause of the problem was changes introduced on April 4th in our authentication services that in the end caused our services to hit physical limitations in our Azure storage environments. As a result of this, some end users were not able to log in to their applications during these timeframes, or login was very slow. There were no external attacks on Simployer, and no data has been exposed or compromised during this incident. # Actions Taken The team responded immediately when the first issues was reported and started to track error logs. Through our monitoring solution we experienced an unnormal increased traffic towards the login services. We scaled up the hosting environments to mitigate for increased traffic and kept searching for root cause. The team deployed several code changes in the login services that gradually improved the situation and kept monitoring the login services to assure healthy operation. # We are sorry! The changes introduced on April 4th were supposed to improve performance and stability in our authentication services. Unfortunately the changes backfired and we are truly sorry for any inconvenience this has caused our customers.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are experiencing login issues for some of our modules at the moment. Our team is investigating the issue.
Report: "Log on issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are experiencing login issues for some of our modules at the moment. Our team is investigating the issue.
Report: "Login issues"
Last updateThe problem was found and fixed. It was related to a error in the configuration that lead to connection issues.
We are experiencing login issues for some of our modules at the moment. Our team is investigating the issue.
Report: "Time and Plan - all systems down for some customers."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
Our vendor is having big issues with our hostingplatform. This also includes Globalmenu for Capitech (Time and Plan). We are working to solve this issue.
Report: "Performance issues in Talent Portal"
Last updateThis incident has been resolved.
We are sporadically experiencing issues with performance in Talent portals.
Report: "Talent Portal is down"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are looking into this and will update info as soon as we know more.
Report: "Login issues"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are aware of the issues while logging in and investigating the root cause