When I checked my Cacti installation just then and spotted the following graph.
WTF?!!! About 1.5 hours ago the out bound bandwidth utilization of this low end VPS went from ~50kbps to almost 5Mbps! I thought, hurray I have finally made it to the Slashdot. But then I looked through the logs and spotted something else instead. Someone was using Load Impact to “load testing” this very site. Who the @#$% did that?!
I have never used Load Impact before but I’ll be amazed that you can do free denial of service “load testing” on any site you wish on the Internet — 50 simultaneous users are enough to crash some badly configured shared hosting sites. Anyway, this VPS at QuickWeb seems to cope pretty well. Here is the loadavg graph of the same period.
That’s it for today. Guys — don’t kill it while I am away.
Update 10 Hours Later…
Yup. LowEndBox got DDoS’ed.
Here is an updated graph of what was going on.
The big spike of in-bound traffic there 2 hours after someone’s “load test” is caused by floods invalid HTTP requests. It did not last very long, but this VPS becomes irresponsive shortly afterwards. Nginx’s error log got flooded with garbages so it filled up all the available disk space on this VPS (which only has 5GB storage). Access log also got truncated so it is not possible to get the full picture. The site did not come back properly until I free up more space.
- DDoS starts at around 15:31 UTC.
- 64,000+ requests after my logs died at 15:48 UTC.
- Top offenders are from Germany, Uruguay, Vietnam, Russia and Brazil. Most are residential IP addresses.
Someone did asked me 2 days ago that “just wondering what you use for DDoS protection on such a little RAM VPS — or don’t you?”. Okay here is my answer Jack — no I don’t have DDoS protection on this small VPS. Not worthwhile — if I want to spend more effort on it I’ll get a bigger meatier server with more active monitoring.
Now someone can go and boast to his mates on IRC “hey my botnet brought down that site with just 80MB of memory, l33t huh?!”.
Update Another Day Later…
WTF. Can you guys stop?! Someone is dragging down 20Mbips continuously for 2 hours and pretty much disables the site due to too much files opening (I forgot to change the ulimit so it’s only at 1024 file descriptors). Here’s the graph from Cacti:
However the loadavg of the system was pretty constant at below 1. Didn’t run out of the disk space this time :) It’s someone doing a LoadImpact test again from their Stockholm server — but probably with much more severe setting this time.
Time to block them from the firewall.
Related Posts:
- 5 Reasons Why You Want a Low End Box - May 26, 2021
- Dead Pool January 2012 - February 2, 2012
- exit(0); - January 19, 2012
Oh! Don’t worry. We won’t let it be :) lol
I see Error 500 on LEB. Hmm.
There’s Error 500 on every page except for the main one.
Did he/she at least have the decency to email you the results!? ;-)
I am also getting 500 bad gateway pages
new benchmark protection? got a 500 for a while :D
@C and Asim Zeesham
http://www.lowendbox.com/blog/wordpress-cheap-vps-lowendscript/#comment-6505
Damn I know it is going to happen when I am away. Someone sent in heaps of bad HTTP requests 2 hours after the “load test” — might not be the same person. Invalid HTTP requests with big payload as it’s all returning 400. Nginx not communicating with PHP because of load.
I’ve got all the IP addresses and on the phone to the police and lawyers… Just kidding :) Probably a bit pointless as these are probably botnet addys.
Heh. I don’t really get why someone would want to DoS attack this place… What are they hoping to achieve?
http://loadimpact.com/result/freevps.tk-ff21cad86b43ff9f82405a4b98247718
Interestingly, my VPS handled 50 users better than 10 users.
Post updated with more info on the DDoS.
@Steve: maybe they are children =))
You can prevent your error logs from filling the disk by setting a disk quota for the nginx user account. Should stop this happening in the future :).
@hughesey — hey that’s a good idea. Implemented.
I had a **smurf** attack once on my dsl/router, request was amplified from an amazon EC2 server (204.236.224.255). I’ve been wise, I just emailed the Amazon EC2 service, guy got busted and they offered me free mp3 of surfer music I enjoyed listening with my fiancée, while drinking glasses of Merlot in the terrace of our house, comtemplating the seascape.
Nay… Amazon never answered back and guy just stopped being an asshole.
lol @ “hey my botnet brought down that site with just 80MB of memory, l33t huh?!”
@LEA:
Since you put the LoadImpact website link in the beginning of this thread, any visitors will also curious, and they will try to use that service to Load Testing.
And thanks for the link, I have used it to test load my website which is using the VPS with QuickWeb.
It didn’t show something necessary under the Geek XEN VPS package with 50 users. Maybe I will try to test it under 2000 visitors.
My few suggestions to make LowEndBox.com more “DDoS resistant” :)
– add LoadImpact IP’s to DROP list on your firewall (http://loadimpact.com/forum/viewtopic.php?id=133)
– turn “limit_rate” feature in Nginx (about 4kb per connection should be OK).
– install eAccelerator in your PHP instance (this will speedup php execution)
– install “WP SuperCache+” plugin on your WordPress
– think about using Squid as ReverseProxy (it uses about 10MB RAM per 1GB disc cache) before NginX
– compress your jquery lib by gzip, and serve it as compressed (it’s 70kb now! :o) maybe use Google hosted libs?
– add cache headers to all you static files (now is “max-age=0” <- this is BAD)
– read & implement: http://vpslife.blogspot.com/2010/02/anti-ddos-nginx-server-configuration.html
@Unknown — those are good suggestions although some of them will be limited by the memory I have on this VPS.
– add LoadImpact’s IP to firewall — good idea :)
– Nginx rate limiting — already done.
– eAccelerator/Xcache — needs more memory.
– WP-Cache/SuperCache — as seen in the loadavg graph the actual load is not an issue when LoadImpact hits.
– Squad for reverse proxy — need more memory.
– jQuery — yup will use Google’s
– cache headers — will also need to implement versioning in query string just in case these files are changed.
And setting “worker_processes 10” in nginx.conf is definitely unnecessary for an event based HTTP server (+2MB RSS each per worker process).
Thanks for the suggestions. Those are good for popular sites and I’ve done a lot of them already for my other sites with a lot more page views.
Not sure about ddos, but that test didn’t do anything then leech some traffic and used 300mhz ram over a few minutes, ram didn’t even move.
@Tom — yeah the first load test wasn’t really a DDoS and the VPS handles the traffic without much issue. The actual DDoS came 2 hours later that killed the site for a few hours until I can get around to fix it. I did not show the loadavg on that one because it’s a bit “messy” :)
Hi. Might not even be a botnet. Did you scan the IPs to see if they were open proxies? I would be interested in the IP list, mail to CnZTFQmJZ648o3WdO0nf Áṭ hushmail.com if you would be so kind. ;)
They could take you down much easier by mailing DMCA takedown requests, so I doubt it’s a pissed off vps provider. Probably just some random kid.
@A — There are about 100 IP addresses. Could be all spoofed, and most are residential broadband/dialup addresses. Anyway, I took them out from the access log and they are here for all of you to enjoy:
http://www.lowendbox.com/media/incident-access.log.gz
Hello,
You mention that I asked you what you use for DDoS protection. I was genuinely asking you as I couldn’t get DDoS Deflate to work on my Debian system and thought you may have some better suggestion.
I didn’t DDoS your website – I don’t know how to run a DoS/DDoS.
Just though’t I’d clear that up.
Jack :-)
Ha. No problem :)
Would be great if one day you could provide a post on how to install Cacti. It looks like a great tool.
@Jackk I actually tried cacti but end up removing it because I didn’t knew how to run this up. Especially because I am using nginx or lighttpd as the web-server.
@Asim Zeeshan – I tried installing cacti myself but had some problems because I am using nginx. An article taking me through step by step would be really useful and much appreciated.
subscribing to this topic :).
Hoping to get more info about anti ddos technique
@Bekanosky – If you have Iptables or APF installed on your machine then (D)DoS Deflate is a perfect little tool. Not sure if it runs on Debian machines though – I tried it and the cron wouldn’t run.
http://deflate.medialayer.com/
Jack xD
@Jackk it needs some tweaks to run on debian. it helped me lot of times!!
@IWTFI – Do you have any links to a tutorial for this?
don’t remember what changes i made.
open ddos.sh and search and change it to work on debian
like this: service cron restart do it /etc/init.d/cron restart
don’t really remember
LoadImpact pulled ~20Mbps for two hours hammering the front page. Damn you! Can I please also have the report please? :)
Wow. What a good DOS… er.. load testing….. tool.
hm…
it have been raising.
I would like to learn how to do that to my website.
Just curious, how my site configuration can handle that :(
@Erawan – You can do it at http://www.loadimpact.com
@Jack :
Thank you. I have tested it to my website. Under 200 clients, my website is load time is almost 20s.
@Erawan – What are the specs of your VPS?
@Jack :
Here is my spec :
Memory 512 MB
Swap 768 MB
Disk Space 30 GB
Bandwidth 750 GB
Dual Core Xeon X3220 @ 2.40GHz
@hughesey could you give some info on how to set disk quota for the nginx user account
@LowEndAdmin: might be a good idea to have a page after all events that describes what you did to prevent this and how you made all the graphs….
To set the disk quota it is basically
and then set the number of kilobytes you give to this user under “soft” and “hard” column. Save the file — done!
I’m trying to set quotas but it says “No filesystems with quota detected.”
any ideas?
@philderbeast — OpenVZ or Xen? Make sure you have usrquota and grpquota as mount option in /etc/fstab for your root file system. With Xen you can edit your /etc/fstab and reboot, but not too sure with OpenVZ — for my OpenVZ VPS they are all already set by default.
its a zen system so a quick fstab change got it all running thanks hopefully this will help stop my box from suffering this same fate :)
Got an email from Ragnar at LoadImpact. He has looked at the logs and said that there are basically multiple paralleled requests to do “load testing” on LowEndBox.com 2 days ago — all from different IPs thus is capable of pulling that much traffic.
Darn.
Anyway. LoadImpact has also blocked my site from being tested. So hopefully we can all move on from now.
Xen isolation is really great that it actually never affected the host node, if that’s OpenVZ you might have been suspended by the host :)
@Joe – You mean to say that a VPS can be suspended for resource usage? I thought a VPS system had it’s isolated system with it’s own resources and coldn’t affect other containers on the node? Maybe I am wrong.
Jack xD
Hi Jackk,
VPS is still a shared resources and one VPS can still affect the whole node (OpenVZ is notorious on this but Xen provides better isolation but still not bullet proof)
regards,
Joe
Ah – OK. Thanks for clearing that up.
Jack xD
well, let me rephrase that, “a VPS is still on a shared resources” unlike on a dedicated where in you control and solely use all the machine’s resources .
@Mr. Joe
Maybe if my business growing bigger, I will buy a Dedicated Server from QuickWeb.
But for now, my budget is only enough for a VPS :D
In other words, if you envision yourself getting “load tested” often, Xen would get you further than OpenVZ…
Maybe LoadImpact should modify their code to only handle a single request at a time instead of allowing “multiple paralleled requests” to be processed.
Someone with a premium account on http://loadimpact.com used to crash one of my sites ( http://x-referat.ro ) connecting up to 500 users/second … loadimpact must do something about this…
@SorinM — you should really block them on the firewall level.