Server Slowness

[anthony]

For those that have realised the site has been slow, and well, all over the place for the past week, it has been due to someone deciding to take great interest in disturbing the sites ability to operate by trying to overwhelm it. The server scales quickly, and even with that scaling, the traffic simply intensified.

It has been a game of whack a mole, and I am trying to isolate the bad actors, but as I isolate one set, it pops up from another location. VPN's aren't the issue, as I have a system in place to block them automatically now, and its doing its job. That is probably the issue... since that system came about, someone got grumpy who was abusing the site within reason, anonymously, and are now just abusing the site, via telco sources.

Much easier to track now, much easier to mitigate, but this is something bigger than your normal amateur. Systems are implemented to tackle these problems, but it can have an affect at times. The issue seems primarily from Asia continent, but there are 48 countries in that, and well... like said, they're moving about between server farms. I have blocked the majority of major server farms previously, maybe another reason someone is upset, don't know. Simply, people don't visit the site from a server farm, they visit via an ISP.

I am chasing though, so that always makes me slower, unfortunately.

 

[Ecdysis]

Gosh...

Thanks for the work you do, without us even realising it, and thanks for the update

 

[anthony]

I've also done a site upgrade to ensure security and stability.

 

[anthony]

Ok, I've put some more blocks in place, removed a bunch too. See what happens overnight.

 

[LuckiLee]

Seems to have improved. Thanks for all you do!!

 

[PamelaS]

For those that have realised the site has been slow, and well, all over the place for the past week, it has been due to someone deciding to take great interest in disturbing the sites ability to operate by trying to overwhelm it. The server scales quickly, and even with that scaling, the traffic simply intensified.

It has been a game of whack a mole, and I am trying to isolate the bad actors, but as I isolate one set, it pops up from another location. VPN's aren't the issue, as I have a system in place to block them automatically now, and its doing its job. That is probably the issue... since that system came about, someone got grumpy who was abusing the site within reason, anonymously, and are now just abusing the site, via telco sources.

Much easier to track now, much easier to mitigate, but this is something bigger than your normal amateur. Systems are implemented to tackle these problems, but it can have an affect at times. The issue seems primarily from Asia continent, but there are 48 countries in that, and well... like said, they're moving about between server farms. I have blocked the majority of major server farms previously, maybe another reason someone is upset, don't know. Simply, people don't visit the site from a server farm, they visit via an ISP.

I am chasing though, so that always makes me slower, unfortunately.

I have great respect for you in mitigating all of this. Thanks for all you do for us. Keep up the good work. You are appreciated.

 

[FFnerd]

Geez that's insane, thanks for the update.

 

[anthony]

To put the issue into perspective, for the past week we've had over 1.5M uniques. Most of that is bad traffic. Our normal weekly uniques should be 500k which would be all good traffic, real people, real search engines. Take into account, I am already blocking every type of AI bot and all known bad bots by Cloudflare. I am blocking a lot of existing bad server farms, companies, etc. So after doing all that, this is a completely new assault that has been jumping around the world. The likely cause is someone who farms data, their systems automatically jump to a new network they have established the moment it reads a network is blocked. Eventually something isn't blocked and the assault begins, overwhelming my server by scraping so much data at once, everything slows down. We already have DDOS coverage in place, so its not that. Cloudflare is the worlds largest DNS company, they have the resources to handle defence department level attacks. This is people / companies trying to scrape data, but they don't limit themselves, they want it all at once and then ongoing, repeating. Servers can't take that type of use. Good bots, search engines, they scrape with limits to avoid this on any server. To take a guess, they have probably been doing it reasonably via VPN's or Proxies, as VPN's limit traffic to some degree, Proxies not so much, but as I block them all now, they've switched tactics.

To put simply, companies like this exist, which is the game being played: ProxyScrape | Unlock the power of the web with proxies

 

[anthony]

I've just provisioned more resources for caching, fingers crossed. This can help mitigate other factors happening when the entire database and search database are in cache.

 

[anthony]

Ok, after monitoring, tweaking a lot of stuff, you can see where we were all good, then got hit maliciously, implemented further tools, and have since then been A-OK. With additional server performance tuning, hopefully, fingers crossed, we can take a hit again without so much issue to users, where the site becomes sluggish. Everything done should help mitigate that now going forward.

 

[Rose White]

Thanks @anthony !

 

[PamelaS]

Ok, after monitoring, tweaking a lot of stuff, you can see where we were all good, then got hit maliciously, implemented further tools, and have since then been A-OK. With additional server performance tuning, hopefully, fingers crossed, we can take a hit again without so much issue to users, where the site becomes sluggish. Everything done should help mitigate that now going forward.

View attachment 66386

Thank you Andrew for your dedication and knowledge of how to mitigate these issues. Really appreciate everything you do for us! Have a pleasant day.