Historical record of incidents for Stonly
Report: "Service Issue"
Last updateDear Customer, We are currently investigating an issue impacting: Stonly customers KB's - location C.Our technical team is working to identify the cause and restore service as quickly as possible. We apologize for this disruption and appreciate your patience.As soon as we have more information, we will post updates on our status page. Sincerely, The Stonly Team
Report: "Service Issue"
Last updateThis incident has been resolved.
Dear Customer, In response to the issue with, we have taken corrective actions. The system is currently under monitoring to ensure its stability.We appreciate your patience and understanding. Sincerely, The Stonly Team
Dear Customer, We are currently investigating an issue impacting: Stonly customers KB's - location C.Our technical team is working to identify the cause and restore service as quickly as possible. We apologize for this disruption and appreciate your patience.As soon as we have more information, we will post updates on our status page. Sincerely, The Stonly Team
Report: "Service Issue"
Last updateThis incident has been resolved.
Dear Customer,In response to the issue with, we have taken corrective actions. The system is currently under monitoring to ensure its stability.We appreciate your patience and understanding.Sincerely,The Stonly Team
Dear Customer,We are currently investigating an issue impacting: Stonly customers KB's - location C.Our technical team is working to identify the cause and restore service as quickly as possible.We apologize for this disruption and appreciate your patience.As soon as we have more information, we will post updates on our status page.Sincerely,The Stonly Team
Report: "Service Issue"
Last updateThis incident has been resolved.
Dear Customer, In response to the issue with, we have taken corrective actions. The system is currently under monitoring to ensure its stability.We appreciate your patience and understanding. Sincerely, The Stonly Team
Dear Customer, We are currently investigating an issue impacting: Stonly customers KB's - location C.Our technical team is working to identify the cause and restore service as quickly as possible. We apologize for this disruption and appreciate your patience.As soon as we have more information, we will post updates on our status page. Sincerely, The Stonly Team
Report: "Service Issue"
Last updateThis incident has been resolved.
Dear Customer, In response to the issue with, we have taken corrective actions. The system is currently under monitoring to ensure its stability.We appreciate your patience and understanding. Sincerely, The Stonly Team
Dear Customer, We are currently investigating an issue impacting: Stonly Website, Stonly Editor, Stonly customers KB's - location N, Stonly customers KB's - location C.Our technical team is working to identify the cause and restore service as quickly as possible. We apologize for this disruption and appreciate your patience.As soon as we have more information, we will post updates on our status page. Sincerely, The Stonly Team
Report: "Issue with email deliverability"
Last updateOur email provider informed us that the issue should be fully resolved.
Our email provider informed us that the issue should be resolved. We will now enter the monitoring phase and will update once we have confirmation that it's completely resolved.
We are experiencing a delay in email delivery due to an outage with our third-party email provider. The provider is aware of the situation and is actively working on a resolution. We will update our status page once the issue is confirmed to be resolved.
Report: "General network issue"
Last updateThis incident has been resolved.
Our individual fix improved situation, but we've just pushed another change that should further improve stability and response times.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Performance degradation"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Performance degradation on Stonly Guides"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Stonly guides and KBs unavailable"
Last updateThis incident has been resolved.
We've rolled back the changes, things seem to be stable now, but we keep monitoring. If your guide or KB is failing to load, please refresh the page.
We've rolled back the changes, things seem to be stable now, but we keep monitoring. If your guide or KB is failing to load, please refresh the page.
We're seeing an issue with our Javascript resources failing to load, making guides and KBs unavailable
Report: "Stability issues"
Last updateThis incident has been resolved.
Things have stabilized. We keep monitoring our platform.
We're seeing bigger than usual load on our servers that is causing slow responses and occasional failures when loading stonly KB and guides
Report: "Targeting delays"
Last updateThe issue is now resolved, the queue is fully processed and all targeting events and properties should be captured close to real-time now.
We've identified the issue and fixed it. Our queue is slowly decreasing, and we keep monitoring the situation.
We're seeing delay in consuming new targeting events. Our team is investigating the issue.
Report: "Stonly guides not loading"
Last updateThis incident has been resolved.
We've identified the underlying issue, fixed the problem and monitor the systems.
Stonly guides are not loading
Report: "Degraded performance"
Last updateOur response times are back to normal levels. We're marking this incident as resolved.
We are continuing to monitor for any further issues.
We've identified the issue and scaled up our systems to match the increased traffic demand. Things are looking stable and we're monitoring the situation.
We are continuing to investigate this issue.
We're noticing slower than usual response times from our API, which affects loading times for widgets, knowledgebases and guide saving in the editor.
Report: "Stonly customers KB's partially unavaliable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Knowledgebase search issues"
Last updateIssue is fully resolved.
We've verified the fix is deployed and working as expected. We continue to monitor for outliers.
The fix is being released to our production systems. We expect the issue to be fully resolved in 15-20 minutes.
We've prepared a fix and are now testing it in pre-production environment.
We were able to reproduce the issue and we're working on a fix.
We've investigating an elevated number of errors when searching content within Knowledgebase for some of our customers.
Report: "Slow loading times and intermittent service unavailability for widgets and knowledgebases"
Last updateThe issue is now fully resolved. We've determined that due to an unusual spike of traffic, our servers became overloaded at 8:59 UTC. Our monitoring caught the issue and scaling mechanisms kicked in, but due to large volume of traffic it took until 9:18 UTC to fully recover from the situation. Between those times, Knowledgebases and guides could occasionally fail to load, or load slower than normally. As next steps, we'll be reviewing our server capacity, reviewing our autoscaling mechanisms and thresholds, to be able to serve similar traffic spike's without interruptions.
The platform is stable now under traffic, we keep monitoring the situation
We are investigating an increased load on our systems causing slow load times and occasional failure to load knowledgebases and widgets
Report: "Content unavailable with "Access restricted" error"
Last updateWe've confirmed all systems are stable now. This incident is now resolved.
We've disabled the problematic API node and are monitoring the situation.
We're identified an incident with one of our API servers affecting ability to display or edit guides in some cases. Since identifying the issue, the problematic server was disabled to restore full functionality.
Report: "Contact Forms delivery delayed"
Last updateAll messages have been processed and sent. Issue is now resolved. Between Oct 10th, 21:13 and Oct 11th, 06:21 Contact Forms were not delivered to the recipients. Due to very small amount of contact form submissions within this timeframe, our monitoring didn't alert us until Oct 11th, 05:58, when amount of undelivered messages crossed the monitoring threshold. After learning about the issue our team quickly identified the underlying cause and restored contact form delivery.
Contact Form delivery is resumed, queued messages are being sent
Contact Forms messages are not delivered to email and third party systems. No data is lost.
Report: "AI Chatbot not returning responses"
Last updateA fix has been implemented and the issue is now resolved.
We've identified the problem and are working on a resolution
Our AI Chatbot will not return answers in cases where guided answer should be used as a response.
Report: "track.stonly.com was unreachable"
Last updateOn September 4th, 2024, between 13:14 and 15:48 UTC, due to a configuration error, our targeting API endpoint at https://track.stonly.com was unreachable, returning 404 errors. After conducting a post-mortem, we've changed our change-management process for this part of the system and added additional alerts in case of it's unavailability.
Report: "Degraded performance"
Last updateAn update to our codebase introducing new performance logs caused slight performance degradation, which caused server overload and restarts during peak hours, which lead to requests failing and site being unavailable for short periods of time. Between 12:07 and 14:47 UTC, Aug 27th, guides and Knowledge Bases were randomly failing to load with a 500 response code. Reloading the page should have fixed the issue. After analyzing the issue, we're implementing a set of improvements to our code, infrastructure, and processes: We’ll be using an alternative approach to performance monitoring We’ll improve our performance testing to better mimic production traffic patterns We’ll be gradually rolling out similar changes which may impact performance to minimize impact to a single server at a time rather than the whole platform We’ll be re-doing our capacity planning to consider adding more CPU power to better accommodate increases in traffic and load
Report: "System impaired during deployment"
Last updateThe system was back to normal after 25 minutes.
Report: "System impaired after deployment"
Last updateThe system was back to normal after 40 minutes.
Report: "Contact Form delivery delayed"
Last updateSince 09:45 UTC the Contact Form message processing was delayed. Our team fixed the issue at 11:39
Report: "Email delivery issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We've identified an issue with emails not being delivered. Our team is working with our email provider to resolve it.
Report: "Stonly partially outaged"
Last updateWe've experienced a 1-minute downtime due to networking issue at our hosting provider. All systems performed automatic recovery and are up and running now
Report: "Stonly Widget partially outaged"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Stonly Widget partially outaged"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "System unavailable due to internal issue with SSL certificate"
Last updateThe SSL certificate for stonly.com domain expired on our nginx servers. As a result, all stonly domains were unavailable. We successfully renewed the certificate and restored access to all our services at 16:20. We also implemented additional monitoring to prevent the same situation from happening in the future.
Report: "PDF Export system is impaired"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Several customers KB's were unavailable"
Last updateThe system was back to normal at 8:15AM (UTC)
Report: "Contact form messages are not being delivered"
Last updateThis incident has been resolved.
Contact form messages are not being delivered
Report: "Stonly unstable due to configuration issue"
Last updateThe system was back to normal in several minutes
Report: "System partially unavailable due to internal network issue"
Last updateDefault Stonly subdomains were unavailable. System back to normal at 8:35
Report: "System unstable during deployment"
Last updateThe system was back to normal in a few minutes
Report: "Invalid SSL certificates in several customers KB"
Last updateAll certificates have been fixed.
Report: "Zendesk forms partially unavailable"
Last updateThe system was partially unavailable to customers using Stonly integration with Zendesk contact forms. The identified issue was between 5:00 am and 5:45 pm on 5/5/2023 UTC. Functionality along with all queued tickets was restored successfully at 3:45 pm.
Report: "System unavailable due to AWS internal network issues"
Last updateSystem back to normal
Report: "Stonly tracking script and Stonly widget partially unavailable"
Last updateSystem partially unavailable to customers using Stonly tracking script and Stonly widget System back to normal at 2022-06-09 12:40
Report: "System unavailable due to internal network issues"
Last updateCritical network routing issue regarding the server’s connection to the outside world. System back to normal
Report: "System partially unavailable during deployment procedure"
Last updateSystem back to normal
Report: "System partially unavailable due to internal network issue"
Last updateSystem partially unavailable to customers using Stonly widget System back to normal at 2023-02-01 8:00
Report: "System unstable due to a lot of malicious requests"
Last updateNew security policies and mechanisms have been introduced
Report: "Degraded experience"
Last updateIssue with caching System back to normal at 11:40
Report: "System partially unavailable due to DDOS attack"
Last updateSystem back to normal
Report: "Stonly Editor for personal space partially unavailable due to configuration issue"
Last updateSystem back to normal at 3/2/2022 8:00AM
Report: "System partially unavailable due to an internal cache issue in Redis"
Last updateProblem solved
Report: "System unavailable due to invalid SSL certificate"
Last updateSystem back to normal in a few minutes
Report: "System unavailable due to internal tool issues"
Last updateSystem back to normal