LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

Who The @#$% is Load Testing This Blog?! [Updated]

When I checked my Cacti installation just then and spotted the following graph.

Traffic graph

WTF?!!! About 1.5 hours ago the out bound bandwidth utilization of this low end VPS went from ~50kbps to almost 5Mbps! I thought, hurray I have finally made it to the Slashdot. But then I looked through the logs and spotted something else instead. Someone was using Load Impact to “load testing” this very site. Who the @#$% did that?!

I have never used Load Impact before but I’ll be amazed that you can do free denial of service “load testing” on any site you wish on the Internet — 50 simultaneous users are enough to crash some badly configured shared hosting sites. Anyway, this VPS at QuickWeb seems to cope pretty well. Here is the loadavg graph of the same period.

Loadavg graph

That’s it for today. Guys — don’t kill it while I am away.

Update 10 Hours Later…

Yup. LowEndBox got DDoS’ed.

Here is an updated graph of what was going on.

Traffic graph after DDoS

The big spike of in-bound traffic there 2 hours after someone’s “load test” is caused by floods invalid HTTP requests. It did not last very long, but this VPS becomes irresponsive shortly afterwards. Nginx’s error log got flooded with garbages so it filled up all the available disk space on this VPS (which only has 5GB storage). Access log also got truncated so it is not possible to get the full picture. The site did not come back properly until I free up more space.

  • DDoS starts at around 15:31 UTC.
  • 64,000+ requests after my logs died at 15:48 UTC.
  • Top offenders are from Germany, Uruguay, Vietnam, Russia and Brazil. Most are residential IP addresses.

Someone did asked me 2 days ago that “just wondering what you use for DDoS protection on such a little RAM VPS — or don’t you?”. Okay here is my answer Jack — no I don’t have DDoS protection on this small VPS. Not worthwhile — if I want to spend more effort on it I’ll get a bigger meatier server with more active monitoring.

Now someone can go and boast to his mates on IRC “hey my botnet brought down that site with just 80MB of memory, l33t huh?!”.

Update Another Day Later…

WTF. Can you guys stop?! Someone is dragging down 20Mbips continuously for 2 hours and pretty much disables the site due to too much files opening (I forgot to change the ulimit so it’s only at 1024 file descriptors). Here’s the graph from Cacti:

Traffic graph after DDoS #2

However the loadavg of the system was pretty constant at below 1. Didn’t run out of the disk space this time :) It’s someone doing a LoadImpact test again from their Stockholm server — but probably with much more severe setting this time.

Time to block them from the firewall.

LEA
Latest posts by LEA (see all)

56 Comments

  1. Oh! Don’t worry. We won’t let it be :) lol

    July 28, 2010 @ 3:40 pm | Reply
  2. flyah:

    I see Error 500 on LEB. Hmm.

    July 28, 2010 @ 4:00 pm | Reply
  3. foo:

    There’s Error 500 on every page except for the main one.

    July 28, 2010 @ 4:06 pm | Reply
  4. Did he/she at least have the decency to email you the results!? ;-)

    July 28, 2010 @ 4:13 pm | Reply
  5. I am also getting 500 bad gateway pages

    July 28, 2010 @ 8:37 pm | Reply
  6. C:

    new benchmark protection? got a 500 for a while :D

    July 28, 2010 @ 8:40 pm | Reply
  7. Damn I know it is going to happen when I am away. Someone sent in heaps of bad HTTP requests 2 hours after the “load test” — might not be the same person. Invalid HTTP requests with big payload as it’s all returning 400. Nginx not communicating with PHP because of load.

    I’ve got all the IP addresses and on the phone to the police and lawyers… Just kidding :) Probably a bit pointless as these are probably botnet addys.

    July 28, 2010 @ 10:03 pm | Reply
  8. Steve:

    Heh. I don’t really get why someone would want to DoS attack this place… What are they hoping to achieve?

    July 28, 2010 @ 10:46 pm | Reply
  9. http://loadimpact.com/result/freevps.tk-ff21cad86b43ff9f82405a4b98247718

    Interestingly, my VPS handled 50 users better than 10 users.

    July 28, 2010 @ 10:55 pm | Reply
  10. Post updated with more info on the DDoS.

    July 29, 2010 @ 12:41 am | Reply
  11. @Steve: maybe they are children =))

    July 29, 2010 @ 1:41 am | Reply
  12. You can prevent your error logs from filling the disk by setting a disk quota for the nginx user account. Should stop this happening in the future :).

    July 29, 2010 @ 4:36 am | Reply
  13. @hughesey — hey that’s a good idea. Implemented.

    July 29, 2010 @ 4:58 am | Reply
  14. Untruemailboxer:

    I had a **smurf** attack once on my dsl/router, request was amplified from an amazon EC2 server (204.236.224.255). I’ve been wise, I just emailed the Amazon EC2 service, guy got busted and they offered me free mp3 of surfer music I enjoyed listening with my fiancée, while drinking glasses of Merlot in the terrace of our house, comtemplating the seascape.

    Nay… Amazon never answered back and guy just stopped being an asshole.

    July 29, 2010 @ 5:49 am | Reply
  15. lol @ “hey my botnet brought down that site with just 80MB of memory, l33t huh?!”

    July 29, 2010 @ 6:09 am | Reply
  16. @LEA:
    Since you put the LoadImpact website link in the beginning of this thread, any visitors will also curious, and they will try to use that service to Load Testing.
    And thanks for the link, I have used it to test load my website which is using the VPS with QuickWeb.
    It didn’t show something necessary under the Geek XEN VPS package with 50 users. Maybe I will try to test it under 2000 visitors.

    July 29, 2010 @ 6:50 am | Reply
  17. My few suggestions to make LowEndBox.com more “DDoS resistant” :)
    – add LoadImpact IP’s to DROP list on your firewall (http://loadimpact.com/forum/viewtopic.php?id=133)
    – turn “limit_rate” feature in Nginx (about 4kb per connection should be OK).
    – install eAccelerator in your PHP instance (this will speedup php execution)
    – install “WP SuperCache+” plugin on your WordPress
    – think about using Squid as ReverseProxy (it uses about 10MB RAM per 1GB disc cache) before NginX
    – compress your jquery lib by gzip, and serve it as compressed (it’s 70kb now! :o) maybe use Google hosted libs?
    – add cache headers to all you static files (now is “max-age=0” <- this is BAD)
    – read & implement: http://vpslife.blogspot.com/2010/02/anti-ddos-nginx-server-configuration.html

    July 29, 2010 @ 7:39 am | Reply
  18. @Unknown — those are good suggestions although some of them will be limited by the memory I have on this VPS.

    – add LoadImpact’s IP to firewall — good idea :)
    – Nginx rate limiting — already done.
    – eAccelerator/Xcache — needs more memory.
    – WP-Cache/SuperCache — as seen in the loadavg graph the actual load is not an issue when LoadImpact hits.
    – Squad for reverse proxy — need more memory.
    – jQuery — yup will use Google’s
    – cache headers — will also need to implement versioning in query string just in case these files are changed.

    And setting “worker_processes 10” in nginx.conf is definitely unnecessary for an event based HTTP server (+2MB RSS each per worker process).

    Thanks for the suggestions. Those are good for popular sites and I’ve done a lot of them already for my other sites with a lot more page views.

    July 29, 2010 @ 7:56 am | Reply
  19. Tom:

    Not sure about ddos, but that test didn’t do anything then leech some traffic and used 300mhz ram over a few minutes, ram didn’t even move.

    July 29, 2010 @ 7:56 am | Reply
  20. @Tom — yeah the first load test wasn’t really a DDoS and the VPS handles the traffic without much issue. The actual DDoS came 2 hours later that killed the site for a few hours until I can get around to fix it. I did not show the loadavg on that one because it’s a bit “messy” :)

    July 29, 2010 @ 8:41 am | Reply
  21. A.:

    Hi. Might not even be a botnet. Did you scan the IPs to see if they were open proxies? I would be interested in the IP list, mail to CnZTFQmJZ648o3WdO0nf Áṭ hushmail.com if you would be so kind. ;)

    They could take you down much easier by mailing DMCA takedown requests, so I doubt it’s a pissed off vps provider. Probably just some random kid.

    July 29, 2010 @ 9:21 am | Reply
  22. @A — There are about 100 IP addresses. Could be all spoofed, and most are residential broadband/dialup addresses. Anyway, I took them out from the access log and they are here for all of you to enjoy:

    http://www.lowendbox.com/media/incident-access.log.gz

    July 29, 2010 @ 11:28 am | Reply
  23. Hello,

    You mention that I asked you what you use for DDoS protection. I was genuinely asking you as I couldn’t get DDoS Deflate to work on my Debian system and thought you may have some better suggestion.

    I didn’t DDoS your website – I don’t know how to run a DoS/DDoS.

    Just though’t I’d clear that up.

    Jack :-)

    July 29, 2010 @ 1:21 pm | Reply
  24. Ha. No problem :)

    July 29, 2010 @ 1:39 pm | Reply
  25. Would be great if one day you could provide a post on how to install Cacti. It looks like a great tool.

    July 29, 2010 @ 1:58 pm | Reply
  26. @Jackk I actually tried cacti but end up removing it because I didn’t knew how to run this up. Especially because I am using nginx or lighttpd as the web-server.

    July 29, 2010 @ 2:00 pm | Reply
  27. @Asim Zeeshan – I tried installing cacti myself but had some problems because I am using nginx. An article taking me through step by step would be really useful and much appreciated.

    July 29, 2010 @ 2:08 pm | Reply
  28. subscribing to this topic :).
    Hoping to get more info about anti ddos technique

    July 29, 2010 @ 3:19 pm | Reply
  29. @Bekanosky – If you have Iptables or APF installed on your machine then (D)DoS Deflate is a perfect little tool. Not sure if it runs on Debian machines though – I tried it and the cron wouldn’t run.

    http://deflate.medialayer.com/

    Jack xD

    July 29, 2010 @ 3:29 pm | Reply
  30. IWTFI:

    @Jackk it needs some tweaks to run on debian. it helped me lot of times!!

    July 29, 2010 @ 11:17 pm | Reply
  31. @IWTFI – Do you have any links to a tutorial for this?

    July 29, 2010 @ 11:20 pm | Reply
  32. IWTFI:

    don’t remember what changes i made.
    open ddos.sh and search and change it to work on debian

    like this: service cron restart do it /etc/init.d/cron restart
    don’t really remember

    July 29, 2010 @ 11:26 pm | Reply
  33. LoadImpact pulled ~20Mbps for two hours hammering the front page. Damn you! Can I please also have the report please? :)

    July 30, 2010 @ 12:57 am | Reply
  34. AZGuy:

    Wow. What a good DOS… er.. load testing….. tool.

    July 30, 2010 @ 2:03 am | Reply
  35. hm…
    it have been raising.
    I would like to learn how to do that to my website.
    Just curious, how my site configuration can handle that :(

    July 30, 2010 @ 2:07 am | Reply
  36. @Erawan – You can do it at http://www.loadimpact.com

    July 30, 2010 @ 10:45 am | Reply
  37. @Jack :

    Thank you. I have tested it to my website. Under 200 clients, my website is load time is almost 20s.

    July 30, 2010 @ 11:11 am | Reply
  38. @Erawan – What are the specs of your VPS?

    July 30, 2010 @ 11:15 am | Reply
  39. @Jack :

    Here is my spec :
    Memory 512 MB
    Swap 768 MB
    Disk Space 30 GB
    Bandwidth 750 GB
    Dual Core Xeon X3220 @ 2.40GHz

    July 30, 2010 @ 11:23 am | Reply
  40. Dirk:

    @hughesey could you give some info on how to set disk quota for the nginx user account

    @LowEndAdmin: might be a good idea to have a page after all events that describes what you did to prevent this and how you made all the graphs….

    July 30, 2010 @ 1:36 pm | Reply
  41. To set the disk quota it is basically

    # apt-get install quota
    # edquota -u www-data
    

    and then set the number of kilobytes you give to this user under “soft” and “hard” column. Save the file — done!

    July 30, 2010 @ 2:54 pm | Reply
  42. philderbeast:

    I’m trying to set quotas but it says “No filesystems with quota detected.”

    any ideas?

    July 31, 2010 @ 11:16 am | Reply
  43. @philderbeast — OpenVZ or Xen? Make sure you have usrquota and grpquota as mount option in /etc/fstab for your root file system. With Xen you can edit your /etc/fstab and reboot, but not too sure with OpenVZ — for my OpenVZ VPS they are all already set by default.

    July 31, 2010 @ 11:22 am | Reply
  44. philderbeast:

    its a zen system so a quick fstab change got it all running thanks hopefully this will help stop my box from suffering this same fate :)

    July 31, 2010 @ 11:33 am | Reply
  45. Got an email from Ragnar at LoadImpact. He has looked at the logs and said that there are basically multiple paralleled requests to do “load testing” on LowEndBox.com 2 days ago — all from different IPs thus is capable of pulling that much traffic.

    Darn.

    Anyway. LoadImpact has also blocked my site from being tested. So hopefully we can all move on from now.

    July 31, 2010 @ 11:46 am | Reply
  46. Xen isolation is really great that it actually never affected the host node, if that’s OpenVZ you might have been suspended by the host :)

    August 1, 2010 @ 9:08 am | Reply
  47. @Joe – You mean to say that a VPS can be suspended for resource usage? I thought a VPS system had it’s isolated system with it’s own resources and coldn’t affect other containers on the node? Maybe I am wrong.

    Jack xD

    August 1, 2010 @ 9:16 am | Reply
  48. Hi Jackk,

    VPS is still a shared resources and one VPS can still affect the whole node (OpenVZ is notorious on this but Xen provides better isolation but still not bullet proof)

    regards,
    Joe

    August 1, 2010 @ 9:34 am | Reply
  49. Ah – OK. Thanks for clearing that up.

    Jack xD

    August 1, 2010 @ 9:36 am | Reply
  50. well, let me rephrase that, “a VPS is still on a shared resources” unlike on a dedicated where in you control and solely use all the machine’s resources .

    August 1, 2010 @ 9:37 am | Reply
  51. @Mr. Joe

    Maybe if my business growing bigger, I will buy a Dedicated Server from QuickWeb.
    But for now, my budget is only enough for a VPS :D

    August 1, 2010 @ 9:40 am | Reply
  52. In other words, if you envision yourself getting “load tested” often, Xen would get you further than OpenVZ…

    August 1, 2010 @ 10:27 am | Reply
  53. Maybe LoadImpact should modify their code to only handle a single request at a time instead of allowing “multiple paralleled requests” to be processed.

    August 1, 2010 @ 2:00 pm | Reply
  54. Someone with a premium account on http://loadimpact.com used to crash one of my sites ( http://x-referat.ro ) connecting up to 500 users/second … loadimpact must do something about this…

    September 3, 2010 @ 11:59 am | Reply
  55. @SorinM — you should really block them on the firewall level.

    September 3, 2010 @ 1:20 pm | Reply

Leave a Reply

Some notes on commenting on LowEndBox:

  • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
  • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
  • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

Your email address will not be published. Required fields are marked *