LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

Protect Your VPS With an Electronic Pitbull: Active Firewalls

A firewall is a security tool that blocks network traffic, with many different configuration options.  For example, you could configure your firewall to block all traffic except attempts to connect to port 80 and 443 (for a web server) as well as your ssh port.  This is an example of the best practice of “blocking everything except what is explicitly allowed”. 

However, even with this sort of policy in place, the security gains for a basic firewall are limited.  If you only have a web server and sshd server running, then there is nothing to block because connecting to a different port will fail anyway.  Chiefly the benefit from this kind of simple firewall is to prevent inadvertent opening of ports.  For example, let’s say you install a piece of software that has a dependency that starts up a service you hadn’t intended to run, or perhaps you change configurations an accidentally set MySQL to expose itself on your public IP.  With a basic firewall, traffic to those port will still be blocked so you benefit from this kind of safety net.

Read more to continue…

However, the real power of firewalls becomes evident when we deploy an active firewall that intelligently monitors traffic and blocks traffic to/from specific clients.  For example:

  • If you are running an IMAP service, you have no choice but to have it publicly-facing where it may become a magnet for people trying to brute-force (guess passwords).  If you deploy an active firewall, the firewall software will watch the IMAP logs and after a configured number of failed logins, the client IP trying to connect will be temporarily blocked. 
  • Likewise, you could have someone trying passwords on your ssh ports all day but they’ll be quickly stopped by an active firewall after a few failed logins.
  • Although an active firewall cannot stop a distributed denial of service, it can end some limited denial of service attacks that are coming from a small number of attacking hosts.

Active firewalls are critical when you have less sophisticated users on your system, such as if you’re running a web host or mail server.  While you may conscientiously pick good passwords, use SSH keys, etc., your more naive users might not.   An active firewall helps protect you by eliminating brute-forcing.

Installing CSF

There are number of different products but one of the best known is ConfigServer Firewall.

To set it up, first install some required perl modules.  On Debian-based systems:

apt-get install libwww-perl liblwp-protocol-https-perl libgd-graph-perl

On CentOS-based systems:

yum install perl-libwww-perl.noarch perl-LWP-Protocol-https.noarch perl-GDGraph

If you’re running CentOS, that distro’s native firewall (firewalld) comes pre-enabled.  You’ll want to disable it before setting up CSF:

systemctl stop firewalld
systemctl disable firewalld

Now download and extract the tarball:

cd /usr/src
wget https://download.configserver.com/csf.tgz 
tar xzf csf.tgz
cd csf
sh install.sh

Next make sure you have all the required kernel modules:

# perl /usr/local/csf/bin/csftest.pl 
Testing ip_tables/iptable_filter...OK
Testing ipt_LOG...OK
Testing ipt_multiport/xt_multiport...OK
Testing ipt_REJECT...OK
Testing ipt_state/xt_state...OK
Testing ipt_limit/xt_limit...OK
Testing ipt_recent...OK
Testing xt_connlimit...OK
Testing ipt_owner/xt_owner...OK
Testing iptable_nat/ipt_REDIRECT...OK
Testing iptable_nat/ipt_DNAT...OK
RESULT: csf should function on this server

In this case, all required kernel modules were present.  On your VPS you might see an error or missing module but as long as you see the result that “csf should function on this server,” you’re good to go.

Configuring CSF

CSF starts in “TESTING” mode.  This is so you do not accidentally lock yourself out. 

Take a look at /etc/csf/csf.conf.  You probably want to adjust the following:

  • TCP_IN has a list of ports you allow.  Remove any services you’re not using.  For example, if you’re not running an FTP server, you can remove ports 20 and 21.  If you’ve changed your SSH port, make sure it is in this list and remove port 22.
  • TCP_OUT should match TCP_IN
  • You can probably pare down UDP_IN and UDP_OUT, perhaps just to port 53 (DNS)
  • If you’re not using IPv6, set IPV6 to 0.  Otherwise, adjust TCP6_IN, TCP6_OUT, UDP6_IN, and UDP6_OUT to match the ipv4 versions.
  • You will be mailed for each blocked IP.  You can modify the templates in /etc/csf/alerts to set an appropriate To: address, or you can set LF_ALERT_TO to one master email address.

There are many, many more ways to customize CSF.  You should take a look through the documentation on configserver.com to see the plethora of CSF capabilities.

Starting CSF

By default, CSF is in TESTING mode.  Before you start it for real, it’s handy to pre-set a time to turn it off.  You could use a crontab entry like this:

*/10 * * * *   systemctl stop lfd ; systemctl stop csf

Don’t forget to remove this crontab entry when you are ready to use CSF for real!

This will disable CSF entirely every 10 minutes on the 10s (so 1:00, 1:10, etc.)  You can then fire up CSF with the knowledge that if you lock yourself out, you will have to wait a maximum of 10 minutes to get back in.  Alternatively, you can login from your VPS provider’s console.

When you’re ready to go, modify /etc/csf/csf.conf and set TESTING to 0.  Then:

systemctl restart csf
systemctl restart lfd

Testing CSF

Here’s a test of CSF so you can see how it works.  I’ve replaced the IP of the server with ‘s.s.s.s’ and the IP of the client with ‘c.c.c.c’.

I ssh’d with a bogus user to the VPS:

$ ssh nonexistant@s.s.s.s
nonexistant@s.s.s.s's password:
$ ssh nonexistant@s.s.s.s
nonexistant@s.s.s.s's password:
$ ssh nonexistant@s.s.s.s
nonexistant@s.s.s.s's password:
$ ssh nonexistant@s.s.s.s
nonexistant@s.s.s.s's password:
$ ssh nonexistant@s.s.s.s
nonexistant@s.s.s.s's password:

CSF (specifically the lfd daemon) detected someone trying to brute-force a login by watching the system logs.  In /var/log/lfd.log this entry appeared:

Apr  7 16:31:23 debian10 lfd[6780]: (sshd) Failed SSH login from c.c.c.c (US/United States/some.example.com): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]

CSF then created a firewall rule to block the IP.  It also made a notation in /etc/csf/csf.deny (so that if the system restarts, the firewall rule is recreated):

c.c.c.c # lfd: (sshd) Failed SSH login from c.c.c.c (US/United States/some.example.com): 5 in the last 3600 secs - Tue Apr  7 16:31:23 2020

Everything after the # is a comment so you know why this IP was blocked.  CSF also looks up the location of the IP address (this is not 100% perfect) and notes it.

Behind the scenes, CSF is manipulating the kernel’s firewall rules to create blocks.  You can see the entire CSF ruleset at any time by issuing this command:

csf -l

You may also see them in dmesg and /var/log/messages, depending on your syslog config.

If you configured an email address as mentioned above, you’ll get an email like this one:

Time:     Tue Apr  7 16:31:23 2020 -0700
IP:       s.s.s.s (US/United States/some.example.com)
Failures: 5 (sshd)
Interval: 3600 seconds
Blocked:  Permanent Block [LF_SSHD]
Log entries:
Apr  7 16:31:07 debian10 sshd[6722]: Invalid user nonexistant from c.c.c.c port 42858
Apr  7 16:31:08 debian10 sshd[6722]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=c.c.c.c
Apr  7 16:31:10 debian10 sshd[6722]: Failed password for invalid user nonexistant from c.c.c.c port 42858 ssh2
Apr  7 16:31:13 debian10 sshd[6722]: Failed password for invalid user nonexistant from c.c.c.c port 42858 ssh2
Apr  7 16:31:18 debian10 sshd[6722]: Failed password for invalid user nonexistant from c.c.c.c port 42858 ssh2

After receiving this email I tried sshing from the same client:

$ ssh nonexistant@s.s.s.s

…and there was an infinite pause.  Here I am blocked not just from SSH but at the IP level, so nothing from my client would be able to connect to this server (web, FTP, etc.)

Enabling CSF

To enable the product, issue these commands:

systemctl enable csf
systemctl enable lfd

Again, if you’ve put any kind of temporary disabling command in cron during testing, be sure to remove it before relying on the firewall.

Living with CSF

An active firewall like CSF is a wonderful security tool but as always, security is the inverse of convenience.  There are two headaches that are common: a flood of block notifications and accidental blocks.

Reducing Block Notifications

At first, you may be surprised at how many emails you get about blocked IPs.  Welcome to the reality of the public Internet!  Previously you were in blissful ignorance – now you can see how many attempts to attack your box are routine every day.

If you’re running a web host, then there are many ports that you are kind of stuck with because users expect FTP on port 20/21, email on 25/587, etc.  However, you might consider changing your SSH port.

This advice is mildly controversial because changing your SSH port does not prevent someone from finding it and trying to login on that port.  However, it will radically cut down how many automated attacks you get.  Many attackers will mass-scan IPs and hone in on those that answer on port 22.  If your system doesn’t respond on port 22, they will move on to another host. 

Some sysadmins take the point of view that once they’ve secured the box by allowing logins only via ssh key and disabling root logins, there is little to fear from script kiddies wasting their time banging on the SSH port.  While this is true, the numerous emails you’ll receive daily are tedious and may drown out alerts you truly do want to review.

You can change sshd to run on any port, but ideally an unused high port (above 1024).

For example, let’s say you wanted to use port 32222.  To do this, modify /etc/ssh/sshd_config:

# Port 22
Port 32222

In this case, we’ve commented out the default and added our port. 

Next, be sure to update TCP_IN and TCP6_IN in /etc/csf/csf.conf to both remove port 22 and add your custom port.

Then restart sshd:

systemctl restart sshd

Your ssh command from your client will now look something like this if your account is ‘joe’:

ssh -p 32222 -i my_ssh_key joe@myvps.example.com

Handling Mistaken Blocks

When CSF blocks a client IP, it adds a firewall rule and also writes it to/etc/csf/csf.deny.  If a client is accidentally are blocked by CSF, you can immediately unblock them with this CSF command (here I’m using ‘c.c.c.c’ again for the client’s IP).  This will both remove the kernel’s firewall rule and clear it from /etc/csf/csf.deny:

# csf --denyrm c.c.c.c
Removing rule...
DROP  all opt -- in !lo out *  c.c.c.c  -> 0.0.0.0/0 
LOGDROPOUT  all opt -- in * out !lo  0.0.0.0/0  -> c.c.c.c

You can also whitelist any IP by adding it to /etc/csf/csf.allow.  Note that CSF whitelists the IP you’re connected from when you first set the product up.  You’ll see an entry like this in /etc/csf/csf.allow:

c.c.c.c # csf SSH installation/upgrade IP address - Tue Apr  7 22:56:37 2020

Consider whitelisting your home IP and/or any VPN IPs you use to connect.

raindog308

1 Comment

  1. Nice security blog post, Andrew.

    August 2, 2020 @ 1:54 am | Reply

Leave a Reply

Some notes on commenting on LowEndBox:

  • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
  • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
  • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

Your email address will not be published. Required fields are marked *