Your Linux VPS lives in a hostile, dangerous world: the Internet. Left unsecured, your system is vulnerable to compromise, which can result in lost data, incurred charges, and general unpleasantness. You should take every precaution to secure your system, but…how will you know if someone slips through? Blissful ignorance is dangerous.
Intrusion detection systems (IDS) are tools to detect if someone has compromised your system. Naturally it’s better to catch electronic criminals before they’re inside your server, but if they do get in, you certainly want to know about it. That’s where a “trip wire” system like AIDE comes in. A trip wire refers to military perimeter defenses where a thin, invisible wire would be setup to ring an alarm bell if someone broke the wire in the middle of the night. In a similar vein, AIDE can warn you if someone slips past your perimeter defenses.
Continue to read more…
What Does AIDE Do?
When installed, AIDE creates a database of all files on your system as well as metadata (modification time, checksum, etc.). Then every night it scans your system and reports if any files have changed.
Of course, some files change all the time. For example, logs are constantly being written to and there are many files that contain transient state information. AIDE can be configured to ignore those files. What you want to monitor with AIDE are system binaries (perhaps someone has installed a root kit), modified web content, etc.
Naturally, when you upgrade your system or do routine patching, many files will change – but in that case, you’ll remember doing it. On the other hand, if some morning you get a report that many files in /usr/bin have been changed and you don’t remember doing it, you’ll be alerted to investigate.
Installing AIDE
On Debian-based systems:
sudo apt-get install aide
Then initialize the AIDE database:
sudo aideinit --init
This process can take some time. On a bare-bones Debian 9 box running at Vultr, it took approximately 2 minutes to complete. You will see output such as this:
Start timestamp: 2020-04-07 17:04:45 -0700 (AIDE 0.16.1) AIDE initialized database at /var/lib/aide/aide.db.new Verbose level: 6 Number of entries: 42544 --------------------------------------------------- The attributes of the (uncompressed) database(s): --------------------------------------------------- /var/lib/aide/aide.db.new RMD160 : 34sejbbJk8NkYF2BJMGHd7+OkWM= TIGER : Xp7tp/LytruyViLSmSGS0lSNzeR4h9I+ SHA256 : T2SCINBXz7G1/W0PfcZrpe82RnWVYYBS wL40glZRM4o= SHA512 : kFziwj1myd8AJMj4WvXEWmaftHW1aiJl E5xLlBS7KwOBP49z2DpPSrqLbAhH2cQ3 LEYbJpWAX4gUV8hlZfCV1Q== CRC32 : NiWF3Q== HAVAL : +cs/EVPUkqStdNYfwd2UcRz5gWZiWLHM 4jBzWyT/FdY= GOST : LvnKAW+XrISqnftzVtA3QYsAtTbqoYJp TmoJfbQOvkk= End timestamp: 2020-04-07 17:06:25 -0700 (run time: 1m 40s)
Running AIDE
On Debian, there is a pre-generated job setup in /etc/cron.daily/aide that runs aide with all of the correct parameters and flags. If you wish to perform a check, simply execute this job:
/etc/cron.daily/aide
After a few minutes’ processing, AIDE will send an email. By default it goes to root. If you wish to change this, modify /etc/default/aide and change the MAILTO line.
The Daily Report
Here is an example of a daily AIDE report. Prior to running this report, I modified a couple files:
- I executed “touch /usr/bin/date” to change that files’ mtime
- I edited /etc/issue
His is what AIDE detected. I’ve trimmed it to focus on the interesting parts:
This is an automated report generated by the Advanced Intrusion Detection environment on server.example.com started at 2020-04-07 17:20:22. AIDE returned with exit code 7. Added, removed and changed entries detected! AIDE produced no errors. Output of the daily AIDE run (410 lines): Start timestamp: 2020-04-07 17:20:23 -0700 (AIDE 0.16.1) AIDE found differences between database and filesystem!! New AIDE database written to /var/lib/aide/aide.db.new Verbose level: 6 Summary: Total number of entries: 42553 Added entries: 5 Removed entries: 3 Changed entries: 24 --------------------------------------------------- Added entries: --------------------------------------------------- d++++++++++++++++: /tmp/aide f++++++++++++++++: /tmp/aide/aide.conf.autogenerated f++++++++++++++++: /tmp/aide/aide.db f++++++++++++++++: /tmp/aide/aide.db.new f++++++++++++++++: /var/lib/aide/aide.db --------------------------------------------------- Removed entries: --------------------------------------------------- f----------------: /run/systemd/sessions/11 p----------------: /run/systemd/sessions/11.ref l----------------: /run/systemd/units/invocation:session-11.scope --------------------------------------------------- Changed entries: --------------------------------------------------- d =.... mc.. .. .: /etc/aide d =.... mc.. .. .: /etc/cron.daily f >.... mc..C.. .: /etc/csf/csf.deny f >.... mci.C.. .: /etc/issue d =.... mc.. .. .: /etc/ssh f <.... mci.C.. .: /etc/ssh/sshd_config d =.... mc.. .. .: /root f >.... mc..C.. .: /root/.bash_history f >.... mci.C.. .: /root/.viminfo d ... n .. : /run d <.... mc.. .. : /run/systemd/units f =.... mc..... .: /usr/bin/date f >.... mc..C.. .: /var/lib/csf/csf.dnscache f >.... mc..C.. .: /var/lib/csf/csf.tempip f =.... mc..C.. .: /var/lib/csf/lock/AT_INTERVAL.lock f =.... mc..C.. .: /var/lib/csf/lock/LF_DIRWATCH.lock f =.... mc..C.. .: /var/lib/csf/lock/LF_EXPLOIT.lock f =.... mc..C.. .: /var/lib/csf/lock/PT_INTERVAL.lock f =.... mc..C.. .: /var/lib/csf/lock/PT_LOAD.lock f =.... mc..C.. .: /var/lib/csf/lock/ST_ENABLE_report.lock f =.... mc..C.. .: /var/lib/csf/lock/command.lock f >b... mc..C.. .: /var/lib/csf/stats/iptables_log f =.... mc..C.. .: /var/lib/csf/stats/lfdstats f >.... mc..C.. .: /var/log/lfd.log
So here we see it’s detected both /usr/bin/date and /etc/issue, as well as quite a lot of other things. We’ll talk next about how to trim down the noise. AIDE also gives you detailed information on the changes…here, I’ve trimmed the email down to just the two files we intentionally modified:
--------------------------------------------------- Detailed information about changes: --------------------------------------------------- File: /etc/issue Size : 27 | 32 Mtime : 2020-02-01 09:09:26 -0800 | 2020-04-07 17:18:08 -0700 Ctime : 2020-03-09 11:39:41 -0700 | 2020-04-07 17:18:08 -0700 Inode : 1731 | 33431 RMD160 : Bol7XoOiUO6aKpMaWAHqRIApxbs= | 7us/8GJhsItGIaiX+VKr8wT8oa4= TIGER : JSqTrZPw9d7S50HXSU8ZYggI/H6uCUBs | lNyNWwkBUEdCTRMJSeC1JsBQ28jix8xC SHA256 : ecDKUqYd+ifA4V9pC6ddPXQXXOEXK+/C | sgKqMRBIz9fJt5miZkDvRYPxk43bw2Oq 1lIEmp27DXI= | Be3LniYEOho= SHA512 : pV/mBESQb4rAo6JQ6yz5vo4dVnVMgdqC | K7Q7naUcNZOa6usQa7mjeqsMBnMhu77R hPb8D03E9ANL9fO5ROLS68HGjyOcqBtl | sTpzd0RHlhT5n8PHZBjLvSK5Ifd4idUd /WozNwQrNa47z0PrtBJDDA== | y3uaK374jhlNGChb1rS5GQ== CRC32 : Mk9Fiw== | rqebvQ== HAVAL : NSpBJniUuZPLndLFtFYgfAQNFCt261xZ | DNyN6a4YuSjQBJGrQsT5UfZx/2DpnXTS /GfU8iVjPSs= | 29XiFsU2/0M= GOST : ZIoi5Ot+3FxA9bpstZrKbAyGa+XAUmBO | Pz05vaNaaFp6+T1oPFT+agcupDdR/s/C 7pBHdKY/GGc= | f7xmSDu5X4U= File: /usr/bin/date Mtime : 2019-02-28 07:30:31 -0800 | 2020-04-07 17:13:45 -0700 Ctime : 2020-03-09 11:40:10 -0700 | 2020-04-07 17:13:45 -0700
AIDE then gives some information on the AIDE database itself, and concludes with:
End timestamp: 2020-04-07 17:22:13 -0700 (run time: 1m 50s) End of AIDE output. End of AIDE daily cron job at 2020-04-07 17:22, run time 111 seconds
Tuning AIDE
As you can see, out of the box AIDE gives a lot of extraneous reports. It’s doing its job, but it’s monitoring and reporting on things we don’t care about.
AIDE has a very flexible language for excluding files – you can even specify what sorts of metadata info you can are about and what you don’t. Debian ships with a lot of rule sets kept in /etc/aide/aide.conf.d. if you peruse those rules, you’ll see they’re by service. Adding your own rules is straightforward. Here we will add a rule to ignore /etc/csf/csf.deny. CSF is an active firewall and it notes every IP it blocks in that file, so it changes nearly every day. In /etc/aide/aide.conf.d, I create a file called 80_aide_csf and put this line inside:
!/etc/csf/csf.deny$
This means “ignore changes in a file called /etc/csf/csf.deny”.
You can read more about creating your own rules in the AIDE documentation.
Off-Site Database
One thing to consider is that if an attacker penetrates your system, he can run aide –init and generate a new DB after installing his root kit. You may wish to copy /var/lib/aide/* to an off-site location and periodically restore it. You could even keep it on a secure host (or cloud storage service such as Dropbox) and wget it prior to running the nightly aide report. If you do this, be sure to periodically refresh the off-site copy otherwise the list of differences will constantly grow.
Related Posts:
I also enjoy using LEBs! I have an empire of little guys for VPNs, larger guys for apps, and big guys for databases and storage. When I'm not in front a screen I'm into German Shepherd dogs, quality knives, target shooting, theology, tabletop roleplaying games, and forest hiking.
I enjoy writing technical articles here on LowEndBox to help people get more out of their systems. You can find me on LowEndTalk @raindog308
Leave a Reply