Server Performance Tuning

[anthony]

Just FYI to all, I am currently doing some server tuning, tweaking, to improve performance. Why I'm telling you, is because to do this, I have to also slow some things down to force errors in order to know what needs tuning. It should all be done within 2-3 days, but I have to let things sit and load to find the issues. Some issues, might now technically be issues, but lets say there are thousands of queries all within spec, but its the thousands of small queries that need to be identified and tuned, not necessarily the time it takes each query.

 

[anthony]

This is coming along. I have upgrade elasticsearch, currently in the process, and improving its responsiveness for the size and traffic the site gets. So you may experience some search issues during the process.

 

[anthony]

Ok, done a lot of tweaking and fiddling over the last 24hrs. Let me know if its better, worse or no change for you.

For me, the server is running a lot better today than what it was yesterday, when I started this. Performance wise. A LOT better. Feels snappier using this site... but the real difference will be for those in other countries during peak hours.

 

[anthony]

Seems to be working, the performance tuning. Still more to go, but it seems to be working. Right now, time of writing this, this site is getting 60k+ per hour uniques to it, and its handling it really well. Everything is still snappy. That's about 16 unique visitors per second, and that was crashing the server just weeks ago. I will be able to fine tune the database further in 2 days, once the assessing tool has enough data to make recommendations.

 

[anthony]

Ok, so we put through nearly half a million uniques yesterday, the caching has improved dramatically with some fixes I implemented, and I personally didn't see any slow down in the site, everything was snappy. Did anyone experience any issues in their last 24hrs or so of the site? Caching was often around 40% and got it over 60%, which is a huge win.

 

[gorgonzola]

The site has been noticeably snappier for me in the last few days.

I always wonder about those uniques: they seem so high because the number of commenters in any 24 hour period is a few score at most. Seems an unusually high ratio of readers to writers. The 1% rule is doing a lot of heavy lifting!

 

[anthony]

Yes, we average around the 150 comments per day. That is somewhat usual for forums. Think about websites that don't have comments, yet they can receive lower, similar and higher readers per day. Everything online is subjective to the niche.

 

[anthony]

Tweaked our caching. Well, stripped it down and rebuilt it, to be exact. That means... caching is often the highest means to potential new issues. If you experience something weird all of a sudden, please let me know with specifics. I think I got it right, but think and is, two very different things.

 

[anthony]

Had some issues yesterday, so had to disable some caching. Fixed now, enabled, and should be snappier again.

 

[Muttly]

Was curious if any work was being done currently. (I'm assuming you can use the time stamp to know when I'm referring to.). Server is the slowest it's been for me in a long time. Not complaining, just providing info in case its useful.

 

[anthony]

No, sorry. Still measuring and tweaking though, but yes, it will get picked up for performance tuning and be recorded in the software tracking things for me. I will look at that later today. I can see a massive spike from servers attempting to flood the site though, so that was likely a DDOS attack, which Cloudflare automatically manages for me.

The easiest way for people to take a site out these days is not to compete with it directly, but to instead try and take it down with constant attacks. I have been fighting this for months now.

 

[anthony]

I've put a managed challenged onto the entire of Asia countries, as those are the most prevalent for presenting issues, with the least participation for the site. Next to nothing. Hopefully that rectified the majority of the issue.