Spotlightr Incident History

Incident reports have been submitted to relevant parent orgs of top IP clusters.

Starting at 22:12 UTC on the evening of June 30th, Spotlightr became a target of a major Distributed Denial of Service attack. The attack continued until July 8th and was repelled effectively throughout. Here’s what happened. Before getting into details, it’s important to stress that the privacy of data, breach of access, security of credentials and other proprietary IPs has never been compromised. The nature of the attack was such that it only flooded the gates, preventing others from accessing them, but never breaching in. At this time an unknown actor has deployed a large scale bot-network of compromised and dedicated machines directly against our load balancers (gate keepers of the network). The scale of attack was such that thousands and sometimes tens of thousands of different servers have continually looped and persisted requested access in terms of occupying as many active / free connection slots as possible. The attack was successful and it managed to bring down our network several times that night for several periods varying 7-28 minutes as the network and volume of attacks went back at each other at scaling infrastructure. On the morning of July 1st, our team responded by instigating the “under attack” operational responses and activated firewall policies which banned over 30,000 IP address which were the top set of what we believed were attack related connections. Unfortunately this ban has caught a significant portion of our real power users (especially high rate API consumers). Over the next several days, up until July 7th, we’ve gone through several iterations of adjusting the filters for the firewall, sometimes mistakenly letting the attackers back in, sometimes mistakenly banning real users. We believe that this balancing act has yielded a good set of rules which will remain standing as a first line of defense moving forward. The attack remains ongoing, but everything is effectively repelled before becoming a risk. We still don’t know who did this and what their motivation was, but the full attack was logged, access records remain, IPs have been attributed to parent data centers and organizations running them and we are in the process of filing abuse reports with all the various stakeholders. Once we have more information on this we will share it.

Starting at 22:12 UTC on the evening of June 30th, Spotlightr became a target of a major Distributed Denial of Service attack. The attack continued until July 8th and was repelled effectively throughout. Here’s what happened.Before getting into details, it’s important to stress that the privacy of data, breach of access, security of credentials and other proprietary IPs has never been compromised. The nature of the attack was such that it only flooded the gates, preventing others from accessing them, but never breaching in.At this time an unknown actor has deployed a large scale bot-network of compromised and dedicated machines directly against our load balancers (gate keepers of the network). The scale of attack was such that thousands and sometimes tens of thousands of different servers have continually looped and persisted requested access in terms of occupying as many active / free connection slots as possible. The attack was successful and it managed to bring down our network several times that night for several periods varying 7-28 minutes as the network and volume of attacks went back at each other at scaling infrastructure.On the morning of July 1st, our team responded by instigating the “under attack” operational responses and activated firewall policies which banned over 30,000 IP address which were the top set of what we believed were attack related connections. Unfortunately this ban has caught a significant portion of our real power users (especially high rate API consumers). Over the next several days, up until July 7th, we’ve gone through several iterations of adjusting the filters for the firewall, sometimes mistakenly letting the attackers back in, sometimes mistakenly banning real users. We believe that this balancing act has yielded a good set of rules which will remain standing as a first line of defense moving forward. The attack remains ongoing, but everything is effectively repelled before becoming a risk.We still don’t know who did this and what their motivation was, but the full attack was logged, access records remain, IPs have been attributed to parent data centers and organizations running them and we are in the process of filing abuse reports with all the various stakeholders. Once we have more information on this we will share it.