DDOS Attacks / AI botfarm overload

[johu]

Starting yesterday we saw massive bulks of requests that overloaded our server. I blocked the most active subnets but it is near impossible to catch all.

I'm not sure who is behind this and what their intention is.

A frequently queried item is the memberlist which I now bluntly disabled. This takes some load off the database. I also pruned all users that never posted anything and were last active before 2025.

I will keep this topic updated.

UPDATE: openinverter.org is now protected by anubis. If you have trouble logging in, go to https://openinverter.org:8444/forum and log in there once. Thereafter the normal site should work as well

[Proton]

That is weird. Hopefully they cannot VPN into another country and do it from there. Maybe some kids.

[johu]

Today it somewhat picked up again, this time from Brazil. I have removed the Vietnam ban and banned some hand-asorted subnets.
BTW you can see something is wrong when there are more than, say, 200 active users

[Proton]

If that server is at home you can install a PFsense firewall - Free software and then you can have lists added to known bad servers or IPs.

You just need a computer with 2 NIC cards

these are some of my lists.

you can also have VPN tunnels from you phones or PC to send all traffic back home through your firewall. All phones in my family send all traffic back home tunneled . that way you can connect to any wifi hotspts and nobody can see your traffic.

I installed the pFsense software on something like this:

https://www.aliexpress.us/item/32568068 ... 00237956_2

You just have to compare the processors to see what you need.

they all use about 6W but the n150 has more power.

[linda.ljungdahl]

Seems like most of the pictures getts http 500 errors on the openinverter wiki when you klick on them, is that related to this in any way?

/Linda

[linda.ljungdahl]

is there a way to download the openinverter wiki database to browse it offline with kiwix?

/Linda

[johu]

Seems like most of the pictures getts http 500 errors on the openinverter wiki when you klick on them, is that related to this in any way?

Thanks for pointing that out. That must be caused by one of the recent updates though. Am looking into it.

If that server is at home you can install a PFsense firewall - Free software and then you can have lists added to known bad servers or IPs.

The forum runs on a hired server somewhere in Germany, so can't play with the hardware. I assume the PFSense Firewall could be installed on it?

[johu]

I installed MediaWiki 1.43.1 now and disabled the newly installed SemanticBundle (viewtopic.php?p=82317#p82317). This was the last alteration 10 days ago and I'm afraid it may have broken things. The installation was very intrusive.

[Proton]

The forum runs on a hired server somewhere in Germany, so can't play with the hardware. I assume the PFSense Firewall could be installed on it?

I am sure Iit can Be installed on a VM but not sure whatbis involved.

[johu]

The requests are still going, now from China also. I will need to find a solution that automatically blocks an entire /16 subnet as soon as it detects too many requests from that same subnet. That is what I currently do manually.

[Proton]

The requests are still going, now from China also. I will need to find a solution that automatically blocks an entire /16 subnet as soon as it detects too many requests from that same subnet. That is what I currently do manually.

can you limit the max connection per second from an IP address on your web server? You would think that the webhosting provider would give you that optinon.

Pfsense has a way to do that but you would have to make the pFsense your default gateway. Pfsense would need to have your public IP and your
webserver to be behind Pfsense.


"
1. Configuring Firewall Rules for DDoS Mitigation

pfSense's firewall rules are your first line of defense against DDoS attacks. By setting up specific rules, you can filter out malicious traffic and protect your network.

Block Malicious IPs: Manually block known malicious IP addresses or use automated blocklists from sources like Emerging Threats. Navigate to Firewall > Aliases and add these IPs to a block list.
Restrict Traffic by Geographic Region: Use pfBlockerNG to block traffic from countries that are not relevant to your user base. This reduces the risk of attacks originating from certain regions.
- Limit Incoming Connections: Set up rules to limit the number of connections per second from a single IP address under Firewall > Rules > WAN. This helps mitigate floods from individual IPs.

2. Using pfBlockerNG for Enhanced Protection

pfBlockerNG is a powerful tool within pfSense that allows for advanced IP and domain blocking capabilities. It's essential for automated updates and enhanced DDoS protection.

Install pfBlockerNG: Go to System > Package Manager > Available Packages and install pfBlockerNG.
Enable GeoIP Blocking: Block traffic based on country using GeoIP filtering. This is especially useful for blocking traffic from regions where you don't expect legitimate users.
Automated Block Lists: Configure pfBlockerNG to download and apply multiple IP block lists. These lists can focus on known malicious IPs, botnets, and other harmful sources.

[johu]

Since removing the firewall rules the DDOS picked up again. Their nature is that no single IP causes a lot of requests but rather many 100 or 1000 IPs create one request per second or so. So it is hard to distinguish from legit activity.

Currently looking into things such as "JavaScript Computational Challenge" that requires the browser to execute some javascript to put more load onto the attackers side or even block it out that way if it doesn't attempt to solve the challenge.

[Proton]

SO you will have to find a way to rate limit the max connection per second from an IP address.

[johu]

SO you will have to find a way to rate limit the max connection per second from an IP address.

Like said, it cannot be told apart from legit access.

It got much worse today. I have to take drastic measures and completely disallowed guest access

[Bigpie]

Someone on discord suggested it may be AI scrapers gobbling up data. Have you checked the user-agent and other request data?

[johu]

Yes, some are but 99% of all requests are normal user agents such as "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_16_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36"

Edit: here is an example of some random /16 subnet

Code: Select all

170.231.140.241 - - [16/Jul/2025:13:47:30 +0200] "GET /forum/download/file.php?id=25675&mode=view&sid=f5f60d848d8760afa858553d19b70e52 HTTP/1.1" 403 3000 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36"
170.231.132.119 - - [16/Jul/2025:13:47:33 +0200] "GET /forum/viewtopic.php?sid=febc63cc02d433d02d876f224da41cd0&t=6365 HTTP/1.1" 200 3804 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36"
170.231.235.243 - - [16/Jul/2025:13:50:12 +0200] "GET /forum/viewtopic.php?sid=93f5102039081d522462debbbbfdcffc&start=25&t=6256 HTTP/1.1" 200 3747 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Brave Chrome/89.0.4389.90 Safari/537.36"
170.231.141.35 - - [16/Jul/2025:13:50:23 +0200] "GET /forum/viewtopic.php?sid=17c8ee2fc1e7ef0308dbed8dce1a8fb6&t=5244 HTTP/1.1" 200 3730 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.79 Safari/537.36"
170.231.121.33 - - [16/Jul/2025:13:50:39 +0200] "GET /forum/ucp.php?mode=privacy&sid=43d38fd160aa14d38afe8d2186079286 HTTP/1.1" 200 4185 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.69 Safari/537.36"
45.170.231.139 - - [16/Jul/2025:13:50:59 +0200] "GET /forum/viewtopic.php?sid=686153c0bfb423971d193c6e7d1f6180&t=4743 HTTP/1.1" 200 3798 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36"
170.231.133.78 - - [16/Jul/2025:13:51:02 +0200] "GET /forum/viewforum.php?f=5&sid=d1cb8be08a4d06784fe32f1dbc22eb77 HTTP/1.1" 200 3798 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36"
170.231.122.199 - - [16/Jul/2025:13:51:02 +0200] "GET /forum/viewtopic.php?sid=7064eec68510c6210693e07283afa83f&t=6494 HTTP/1.1" 200 3796 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
170.231.132.235 - - [16/Jul/2025:13:51:11 +0200] "GET /forum/search.php?sid=175be1ebcf71add40421c8016447bfbd HTTP/1.1" 200 2978 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
170.231.143.67 - - [16/Jul/2025:13:51:16 +0200] "GET /forum/viewtopic.php?p=60153&sid=ca237265be2cb00c3f4a02c6610ef57f HTTP/1.1" 200 3797 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"
170.231.28.110 - - [16/Jul/2025:13:51:59 +0200] "GET /forum/viewtopic.php?sid=85599218f4968cf7271dd71adfb3c412&start=25&t=2322 HTTP/1.1" 200 3746 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Brave Chrome/89.0.4389.114 Safari/537.36"

There a long delays between requests and in the 5 minute period the same address never shows up twice

[projectgus]

FWIW, AI scrapers are crippling small web servers worldwide at the moment. They're sophisticated enough to use different IPs and common browser user agents. If no one has yet tried to extort money from you to stop it then it's most likely scrapers and not an "intentional" DDoS.

A lot of open source sites have installed Anubis to limit access to real browsers. Take a look at this usage graph for an idea of the impact: https://mastodon.social/@dbaio/114820378778350969

(If you don't want to go open source then there's always Cloudflare, etc. on the corporate end.)

[johu]

That seems to hit the nail on the head
Thanks very much, will try it when I get the chance to

[johu]

Installed Anubis. Database server load decreased from 200% cpu usage to hardly anything

Lets see if there are adverse effects, i.e. TOR users no longer able to access the site or so

EDIT: tested. Still works over Tor

[Bigpie]

I'm no longer able to access with Brave Browser.

[johu]

I'm using Brave as well and had a recusrsion problem. Try deleting cache and cookies

[TassieDevil]

I'm no longer able to access with Brave Browser.

I had the same problem.
Johu, I was looking at Anubis for my own use. Do you have to use it with a reverse proxy? I don't use one, just Apache. Edit: looks like it is required.

[johu]

I followed this: https://anubis.techaro.lol/docs/admin/installation/
I did a native install but probably Docker is the preferred method.
I'm running nginx here. One thing that wasn't mentioned in any manual:
I the frontend server (the one receiving requests on ssl port 443) I had to specify

Code: Select all

proxy_redirect http://openinverter.org:8090 https://openinverter.org:443;

As otherwise http redirects would send you to the none-reachable internal server.

For a similar reason where php fpm is linked to nginx I had to specify

Code: Select all

fastcgi_param HTTP_HOST openinverter.org;

as again some php scripts use HTTP_HOST to assemble urls. On the forum for example the gallery images weren't visible because they linked to "localhost"

I'm sure you'll need to do something similar in apache to overcome these limitations.

[Jacobsmess]

Brave works fine for me

[Bigpie]

I'm using Brave as well and had a recusrsion problem. Try deleting cache and cookies

This worked. Now able to access again.