Historical record of incidents for MadKudu
Report: "Copilot Partial Outage"
Last updateThis incident has been resolved.
The LaunchDarkly outage triggered an overload on our production database by one of our systems, which in turn prevented the Copilot service from handling some requests. After restarting the database and with LaunchDarkly gradually recovering, the load has decreased and Copilot is now functioning normally. We are closely monitoring the situation and are developing a plan to improve our resilience to similar incidents, which will be implemented in the coming weeks.
One of our provider, LaunchDarkly, is experiencing a major outage: https://status.launchdarkly.com/ We use Launchdarkly for our feature flag infrastructure (i.e. control how features are released). We are looking for a way to fix the situation without having to wait for Launchdarkly to fix their service.
Report: "Copilot Partial Outage"
Last updateThis incident has been resolved.
The LaunchDarkly outage triggered an overload on our production database by one of our systems, which in turn prevented the Copilot service from handling some requests. After restarting the database and with LaunchDarkly gradually recovering, the load has decreased and Copilot is now functioning normally. We are closely monitoring the situation and are developing a plan to improve our resilience to similar incidents, which will be implemented in the coming weeks.
One of our provider, LaunchDarkly, is experiencing a major outage: https://status.launchdarkly.com/We use Launchdarkly for our feature flag infrastructure (i.e. control how features are released). We are looking for a way to fix the situation without having to wait for Launchdarkly to fix their service.
Report: "Scoring Push to CRM Outage"
Last updateDuring April 19th 2025 07:00 AM - April 22nd 2025 our Push ECS agent experienced intermittent disconnections, causing push delivery to be unstable. This was due to the agent becoming unresponsive, likely from machine lock or CPU overload. Actions taken: • Identified and rebooted the affected instance • Doubled CPU and memory reservations for the push tasks • Push service is now stable and under close monitoring Next steps to prevent recurrence: • Continue monitoring agent status and container performance
Report: "Explore View Degradation"
Last updateThis incident has been resolved.
The explore view is experiencing outage and the message "It seems that there are some problems loading your data. Please update the rules or contact support if the issue persists." is displayed on some request. We are currently working on solving it and the system should be back very soon.
Report: "Explore views outage"
Last updateThis incident has been resolved.
The service is back and we upscaled the machines to make sure you don't experience degraded performance. We will keep monitoring the situation for 24 hours.
The service in charge of executing explore queries is experiencing an outage due to an overload. We are restarting the service and scaling up the machines to bring it back online.
Report: "Intermittent 504 Errors"
Last updateThis incident has been resolved.
The issue has been identified and the fix has been released. We are actively monitoring the platform.
We identified the issue and are actively working on fixing it. We expect the performances to go back to normal before 6:00 UTC
Our AWS infrastructure is suffering partial outage that cause Intermittent 504 Errors on the public API and can prevent new attributes mapping from being published. We are actively investigating the issue but due to its complexity, we don't expect it to be completely fixed before 24h
Report: "Public API is returning 504"
Last updateThis incident has been resolved.
The fix has been deployed and the error rate dropped significantly. We are monitoring the systems to ensure stability.
We are continuing to work on a fix for this issue.
The AWS region where the public API is hosted is having issues. We are redeploying the instances.
Report: "Explore views are empty"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
A major database migration created disturbance in our explore views. As a result, these views are empty. The problem has been identified and we are working to fix the problem.
Report: "Downtime & Slowness on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Delay with score push"
Last updateThis incident has been resolved.
Issue AWS on Saturday 27 degraded our ability to push scores into destination systems (e.g. SFDC). We identified the issues and push resumed. We expect every score to be up-to-date in a few hours.
Report: "Delays with batch scoring (fit, behavioral), refresh of MSI and publication of mappings"
Last updateThis incident has been resolved.
Our infrastructure is experiencing delays. This is affecting fit and behavioral scoring, the refresh of the MSI as well as the publication of new mappings. Our team has been notified and is looking into the issue
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded Performance with Data Endpoint"
Last updateThis incident has been resolved.
Our data.madkudu.com endpoint is experiencing degraded performance. Data from Segment may be delayed. We are investigating the issue.
Report: "Degraded Performance with Data Endpoint"
Last updateThis incident has been resolved.
Our data.madkudu.com endpoint is experiencing degraded performance. Data from Segment may be delayed. We are investigating the issue.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "Delays with demographic scoring of new records"
Last updateIncident caused by the network outage in AWS - Oregon. [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status)
This incident has been resolved.
Our infrastructure is experiencing delays in realtime demographic scoring of new records. Our team has been notified and is looking into the issue
Report: "app.madkudu.com may be inaccessible"
Last updateIncident caused by the network outage in AWS - Oregon. [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status)
This incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "Degraded performance on api.madkudu.com"
Last updateIncident caused by the network outage in AWS - Oregon. [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status)
This incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "app.madkudu.com may be inaccessible"
Last updateIncident caused by the network outage in AWS - Oregon. [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status)
This incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateIncident caused by the network outage in AWS - Oregon. [https://health.aws.amazon.com/health/status](https://health.aws.amazon.com/health/status)
This incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "app.madkudu.com may be inaccessible"
Last updateThis incident has been resolved.
Our app is experiencing errors and may be inacessible. We are investigating the issue.
Report: "app.madkudu.com is inaccessible"
Last updateThe MadKudu app is now accessible again. We have resolved the issue and will continue to closely monitor the situation. We sincerely appreciate your patience and ongoing collaboration with us.
Our upstream provider, Netlify, is currently experiencing an issue that is affecting the accessibility of the MadKudu app. We are actively monitoring the situation and working to resolve the incident as quickly as possible.
Our app is experiencing errors and is inaccessible. We are investigating the issue.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.
Report: "Degraded performance on api.madkudu.com"
Last updateThis incident has been resolved.
Our API endpoints are experiencing higher latencies or error rates. We are investigating.