Was curious if any work was being done currently. (I'm assuming you can use the time stamp to know when I'm referring to.). Server is the slowest it's been for me in a long time. Not complaining, just providing info in case its useful.
I've done some further monitoring for this, the result is that I had to change more PHP settings. I have temporarily added Google Analytics onto the site to measure the real-time traffic after Cloudflare (blocked / refused through to server). What is happening is that there are peaks and troughs happening constantly. In one five minute period I will have 20 active users, then it will go to 300 or 500 for 5 minutes. When you start doing math of page loading times, how long a page takes, blah blah blah, its a lot.
Basically, I just changed PHP settings again to scale workers further to handle peaks, but drop them off in troughs. It can sound counter productive to people when fully explained, but it works. By default the server has about 5 workers allocated, so what happens is each one becomes heavily loaded and starts fighting with itself, so one will consume 60% - 80% CPU (300% - 400%). 100% is 1 CPU core. It was actually worse than this. I scaled that out to 40, but it still wasn't enough. When you scale the workers you spread the load and remove the fighting within itself, basically each user online per second has their own worker (hopefully). The result is whilst you have more, you endup with something more like 5% per worker (200% total, or 2 cores). I have scaled it out to 80 now, which will soak up the peaks better, as a result, each worker uses less CPU again, like around 2% (160% or 1.6 CPU cores).
This is not a relative thing, you can't keep scaling out like that, because there is a sweet spot, then you have the opposite issue again. So I've manually overridden the settings I implemented a few days ago and scaled out further to try and limit CPU use during peaks.
Hopefully what you experienced, fingers crossed, has ended.
All of this has consequences to other areas, ie. database load. When PHP becomes overloaded, it then screws up database calls and then that becomes loaded, and the consequences roll through multiple areas. It's a balance, and I am trying to find it right now.
Singapore is currently on my shit list... the abuse coming from that country is just astounding me right now. I have it managed, but its just mind boggling.
