khawk said:
GDNet sees ~100 requests/minute on average

I wonder how much of that is bots.

Brian Sandberg said:
That doesn't sound bad at all.

It is not just about requests, but also by total data transferred (just opening exactly current page on forums transfers (incl. resources) give or take 16MB … opening home page incl. resources is like 85MB). I'll be generous and state that those 16MB is average - with (processed) 100 requests/min, that gets to 1.6GB each minute (or about 30MB/s give or take which is → 240 Mbit/s required by server). Caches can help but overall it is a lot of resources.

But that still may not explain anything - because it can have bottleneck in completely different spot (missing index/tree on database (and GameDev.net database will be large as it grew over the years), uses some debug-heavy script somewhere in underlying module, or hosting just lying about their capabilities (sadly, I've seen that one), etc.).

Aressera said:
One problem I see is that avatar images…

You don't want to use JPEGs today, or rather - use those only as a backup. Using WEBP is preferred to those (in terms of both - size and quality … you will usually save about 50% of space compared to JPEG at the same quality).

Aressera said:
Wow that sounds terribly inefficient for a page where the only major non-text content are a few icons and avatars, all less than 32px in display size. It should be less than 1MB. This is the problem with modern web development based on gigantic frameworks: the bloat is unreal and unnecessary. It's refreshing these days when I see a plain HTML or PHP page, the speed is amazing in comparison to most of the “modern” web.

I remember setting up a site(s) in AngularJS and later also in React. The “nothing big” eventually grew into something huge and bloated - mainly due to dependency hell (especially as you have to update dependencies over time to ensure some security level), which still takes about multiple times more to maintain than other web services we run with lightweight custom front end (yes it is custom, but documented and lightweight). You can especially see how pages became extremely heavy when you try to run them on older hardware (take 10-15 years old laptop) … and they're far from “great user experience”.

This would end up in endless rant though - modern web is not necessarily bad, but the bloatness of the most (in)famous and well known apis has grown extremely over the years. It often ends up in unmaintainable mess. There are apis which are great and lightweight enough (like F.e. Vue).

Tom Sloper said:

khawk said:
GDNet sees ~100 requests/minute on average

I wonder how much of that is bots.

A lot. Good bots (search engines) and bad bots. Try to keep the filter up-to-date for the bad bots, but it's difficult to block the script kiddz and attempted hacks. Most common thing to bring things down is an occasional massive traffic spike, which seems to be an every few week occurrence.

After looking again I was wrong - it's ~350-400 requests/minute (oops 🙂).

If everyone is that interested.. the current setup is almost entirely on Google Cloud:

• gamedev.net IP goes to a load balancer + caching/CDN
• App VM is 4 CPU, 4GB RAM
  • nginx for proxy + URL 301's/redirects of 10-20 year old URLs that still get hits + bad bot and IP filters
  • uwsgi for the app server (python/django)
  • running background tasks for emails/notifications and other various services (blog imports, topic archiving, cleanup)
    (finding that some of these services may be a cause for slowdowns, not enough CPU to go around)
  • App is SSR, have explored using React in various ways but haven't enabled anything yet
  • JS usage limited to basic interactions + CKEditor for the editor, which is a decision that goes back to the community raising hell over switching away from WYSIWYG - maybe opinions are different now
• DB is MySQL 5.7, 2 CPU, 15GB (to hold DB in RAM), tables indexed but could use better maintenance
• Redis, elasticsearch for data caching (including db queries), searches, feeds caching (1 CPU each)
• gamedev.jobs and gamedev.social (mastodon) use the same DB/redis/elastic backends
• stream.io for notifications (sometimes the source of slow page loads… looking at alternate solutions)
• AWS SES for email

Site activity and traffic rank:

• Forums (35%)
• Projects (20%)
• Blogs (15%)
• Tutorials (15%, although submissions are barely a trickle)
• Careers (10%)
• Portfolios (5%)
• News and others (< 1%)

Most people land on the home page (70% of all traffic). Graphics & GPU Programming is the most popular forum. Not a surprise, always has been.

Average ~80-120 new spam accounts per day, automatically filtered using a combination of techniques. Some still get through occasionally and post. Some false positives - sorry if that happened to you. I am notified of every account that is flagged.

Anything else of interest? 🙂

Things like image management (i.e. using webp), smarter use of resources, moderation tools, improved user settings, GDNet+ subscriptions, etc are known deficiencies that I get to as time and interest allow. As my full time career has evolved further, it's become challenging to keep up with things, which is why moving to something like Discourse is in the conversation.