Historical record of incidents for ModernLoop
Report: "Reports of slowness"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Reports of slowness"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Zoom is down"
Last updatehttps://status.zoom.us/incidents/pw9r9vnq5rvk
Zoom Official Status Pages (may not load) https://status.zoom.us/ https://uptime.zoom.us/ ~50K people reporting Zoom is down for them in the last hour https://downdetector.com/status/zoom/
Report: "GMail email sends sporadically failing"
Last updateDue to a version upgrade for our email service, some emails sent through gmail failed to deliver between 3/3 10pm PST - 3/4 7am PST
Report: "Slack is down"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
Slack is down: https://slack-status.com/2025-02/1b757d1d0f444c34
Report: "Lever Hire Platform is returning 500 error"
Last updateSee resolution here: https://status.lever.co/incidents/r94mjyqwbxtv
ModernLoop team is following along here: https://status.lever.co/incidents/r94mjyqwbxtv This does not affect customers on other Applicant Tracking Systems.
Report: "ML Slowness"
Last updateThis incident has been resolved.
This incident has been resolved. And we are continuing to monitor.
We are currently investigating this issue.
Report: "Site Not Loading"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "ModernLoop down"
Last updateThe issue was due to a combination of hitting our IP address limit for our Kubernetes deployment and a bad version number for our Datadog Java Agent. Datadog Java Agent is 1.46.0 on Maven but that version doesn't not exist on Google Cloud Console Artifact Registry. The incorrect version prevented new pods from starting up and the ip address allocation prevented proper visibility into this issue. Followups include: * Adding a new subnet to increase the number of IPs * Add a CI check to make sure the JAR artifact exists for the Datadog Agent Library we need
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Reports of slowness"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Experienced High Load from Microsoft Outlook Webhooks"
Last updateThis incident has been resolved.
Load is back down. Site is performant.
ModernLoop just received a high level of calendar sync events. Causing a high load.
Microsoft calendar sync experiencing high load. Investigating immediately.
Report: "Empty Lists on Critical ModernLoop Tabs"
Last updateWe have reindexed customer's candidates to be searchable bringing full functionality to Candidates Tab and Command+K.
Currently all tabs should be populated except Candidates Tab. We are currently re-indexing our Candidate search. Candidates may not be searchable for a brief time from “Command+K” or the Candidates tab. To access candidates, please use the Tasks tab or the chrome extension until this is resolved. This process should be complete by 12pm PT. Candidate-facing flows are not impacted by this incident. Candidate-facing flows include: Candidate Portal, Candidate Self-Schedule, Candidate Submit Availability
We have reindexed all objects besides Candidate search
ModernLoop loads with empty lists on Tasks, Candidates, Interview Modules and Employees tabs.
Report: "Reports of slowness"
Last updateThis incident has been resolved.
A root cause analysis will be be conducted and posted as soon as possible.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "ModernLoop Service Outage"
Last updateRoot Cause Analysis: Workday Integration Performance Issue Overview: On Jan 23 2023, customers experienced slower response times and intermittent service issues. We take these disruptions very seriously, and we want to share a transparent account of the issue, its root cause, and the steps we are taking to address it both in the short and long term. Root Cause: The issue was traced back to our Workday integration. Specifically: - The integration was fetching and processing an unexpectedly large volume of data. - This data processing occurred on our web servers, which caused high memory usage and led to out-of-memory errors and service outage. Impact: - Customers experienced slow application response times or outages. Fixes: Short-Term: We have optimized the processing of Workday data to reduce its memory usage and overall load on our web servers. This optimization has already been implemented and has mitigated the immediate issue. Medium-Term: To ensure long-term stability and scalability, we are in the process of moving this data processing workload off our web servers. Instead, it will be handled by dedicated worker nodes that are better equipped to process large datasets without impacting web server performance. This change is currently in progress and will be deployed as soon as possible. Next Steps: We will continue to monitor system performance closely to ensure the short-term optimizations are effective. Once the medium-term fix is deployed, we will communicate this update to all customers. We are also reviewing all other data integrations to proactively identify and mitigate similar risks. We sincerely apologize for the inconvenience caused by this issue and appreciate your patience as we work to improve our systems. Ensuring a reliable and high-performing service for our customers remains our top priority. If you have any further questions or concerns, please do not hesitate to reach out to our support team.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Report: "ModernLoop Errors (502)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Site not loading"
Last updateThis incident has been resolved.
The site has recovered, but we are continuing to investigate the root cause.
Site has recovered but issue is still being investigated.
We are continuing to investigate this issue.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Site unresponsive after new feature launch"
Last updateAfter the launch of One-click and Automated Scheduling, the site was unavailable from 8:51 am PST to 9:01
Report: "Reports of ModernLoop application unresponsive"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Reports of errors on schedule creation"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Reports of errors on schedule creation"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Customer reporting slowness through out the web application"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Some SAML users downgraded to minimum permission roles on login"
Last updateThis incident has been resolved. Please reach out to ModernLoop if you need assistance.
Fix is in production and we are monitoring.
The issue has been identified and fix is on the way.
Report: "Google API Latency Spike - Customers experiencing slowness"
Last updateThank you for your patience. The team has implemented a workaround to the current Google API issues. We’ll be working with Google to help them address the root cause. With our workaround we are actually seeing the latency reduce to below pre-incident numbers. (Lower is better) We truly appreciate your patience and understanding while we worked through this. If you continue to experience any slowness within the schedules flow, please let us know.
We are currently investigating this issue.
Report: ""Network Error" shown to customers"
Last updateThis incident has been resolved. We will post a post-mortem after internal review.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Reports of candidate self-schedule failing to complete"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: ""Network Error" message shown to customers"
Last updateCustomers experienced error messages saying there was a "Network Error". This event occurred from 14:31 to 14:38.
Report: "Candidates may experience difficulties with self-scheduling."
Last updateThe incident has been resolved! Thank you for patience.
Our team is diligently addressing the issue. Thank you for your patience!
Report: "Google Calendar Outage"
Last updateThe problem with Google Calendar has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
We're experiencing issues with scheduling and calendar syncs due to a Google Workspace outage
Report: "Slack channels not created"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Slack is currently experiencing technical difficulties that are preventing the creation of channels in ModernLoop. Please refrain from adding a Slack channel until the issue is resolved. More updates here: https://status.slack.com/
Report: "Can not complete Schedule Flow"
Last updateWe received reports of not being able to complete the recruiting-side schedule flow. We found this was due to a third party failure that resolved after a couple of minutes.
Report: "Network Error"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Some Components Not Loading"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Auth0 is experiencing high latency"
Last updateThis incident has https://status.auth0.com/incidents/qbc7xp8gj6j1
We are continuing to work on a fix for this issue.
Hi all, we are unfortunately experiencing an issue with our login provider. You may have noticed that you are unable to log in to ModernLoop or are experiencing issues while in the ModernLoop application. We apologize for the inconvenience this has caused and will update everyone as soon as we have updates. Our login provider is Auth0 and this is their status page: https://status.auth0.com/incidents/qbc7xp8gj6j1