LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

Catching Intruders With a Trip Wire: The AIDE Package

Tags: , , , Date/Time: August 20, 2020 @ 9:55 am, by raindog308

Your Linux VPS lives in a hostile, dangerous world: the Internet.  Left unsecured, your system is vulnerable to compromise, which can result in lost data, incurred charges, and general unpleasantness.  You should take every precaution to secure your system, but…how will you know if someone slips through?  Blissful ignorance is dangerous.

Intrusion detection systems (IDS) are tools to detect if someone has compromised your system.  Naturally it’s better to catch electronic criminals before they’re inside your server, but if they do get in, you certainly want to know about it.  That’s where a “trip wire” system like AIDE comes in.  A trip wire refers to military perimeter defenses where a thin, invisible wire would be setup to ring an alarm bell if someone broke the wire in the middle of the night.  In a similar vein, AIDE can warn you if someone slips past your perimeter defenses.

Continue to read more…

What Does AIDE Do?

When installed, AIDE creates a database of all files on your system as well as metadata (modification time, checksum, etc.).  Then every night it scans your system and reports if any files have changed.

Of course, some files change all the time.  For example, logs are constantly being written to and there are many files that contain transient state information.  AIDE can be configured to ignore those files.  What you want to monitor with AIDE are system binaries (perhaps someone has installed a root kit), modified web content, etc.

Naturally, when you upgrade your system or do routine patching, many files will change – but in that case, you’ll remember doing it.  On the other hand, if some morning you get a report that many files in /usr/bin have been changed and you don’t remember doing it, you’ll be alerted to investigate.

Installing AIDE

On Debian-based systems:

sudo apt-get install aide

Then initialize the AIDE database:

sudo aideinit --init

This process can take some time.  On a bare-bones Debian 9 box running at Vultr, it took approximately 2 minutes to complete.  You will see output such as this:

Start timestamp: 2020-04-07 17:04:45 -0700 (AIDE 0.16.1)
AIDE initialized database at /var/lib/aide/aide.db.new
Verbose level: 6
Number of entries: 42544
---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------
/var/lib/aide/aide.db.new
RMD160   : 34sejbbJk8NkYF2BJMGHd7+OkWM=
TIGER    : Xp7tp/LytruyViLSmSGS0lSNzeR4h9I+
SHA256   : T2SCINBXz7G1/W0PfcZrpe82RnWVYYBS
wL40glZRM4o=
SHA512   : kFziwj1myd8AJMj4WvXEWmaftHW1aiJl
E5xLlBS7KwOBP49z2DpPSrqLbAhH2cQ3
LEYbJpWAX4gUV8hlZfCV1Q==
CRC32    : NiWF3Q==
HAVAL    : +cs/EVPUkqStdNYfwd2UcRz5gWZiWLHM
4jBzWyT/FdY=
GOST     : LvnKAW+XrISqnftzVtA3QYsAtTbqoYJp
TmoJfbQOvkk=
End timestamp: 2020-04-07 17:06:25 -0700 (run time: 1m 40s)

Running AIDE

On Debian, there is a pre-generated job setup in /etc/cron.daily/aide that runs aide with all of the correct parameters and flags.  If you wish to perform a check, simply execute this job:

/etc/cron.daily/aide

After a few minutes’ processing, AIDE will send an email.  By default it goes to root.  If you wish to change this, modify /etc/default/aide and change the MAILTO line.

The Daily Report

Here is an example of a daily AIDE report.  Prior to running this report, I modified a couple files:

  • I executed “touch /usr/bin/date” to change that files’ mtime
  • I edited /etc/issue

His is what AIDE detected.  I’ve trimmed it to focus on the interesting parts:

This is an automated report generated by the Advanced Intrusion Detection environment on server.example.com started at 2020-04-07 17:20:22.

AIDE returned with exit code 7. Added, removed and changed entries detected!

AIDE produced no errors.

Output of the daily AIDE run (410 lines):
Start timestamp: 2020-04-07 17:20:23 -0700 (AIDE 0.16.1)
AIDE found differences between database and filesystem!!
New AIDE database written to /var/lib/aide/aide.db.new
Verbose level: 6

Summary:
Total number of entries: 42553
Added entries: 5
Removed entries: 3
Changed entries: 24

---------------------------------------------------
Added entries:
---------------------------------------------------

d++++++++++++++++: /tmp/aide
f++++++++++++++++: /tmp/aide/aide.conf.autogenerated
f++++++++++++++++: /tmp/aide/aide.db
f++++++++++++++++: /tmp/aide/aide.db.new
f++++++++++++++++: /var/lib/aide/aide.db

---------------------------------------------------
Removed entries:
---------------------------------------------------

f----------------: /run/systemd/sessions/11
p----------------: /run/systemd/sessions/11.ref
l----------------: /run/systemd/units/invocation:session-11.scope

---------------------------------------------------
Changed entries:
---------------------------------------------------

d =.... mc.. .. .: /etc/aide
d =.... mc.. .. .: /etc/cron.daily
f >.... mc..C.. .: /etc/csf/csf.deny
f >.... mci.C.. .: /etc/issue
d =.... mc.. .. .: /etc/ssh
f <.... mci.C.. .: /etc/ssh/sshd_config
d =.... mc.. .. .: /root
f >.... mc..C.. .: /root/.bash_history
f >.... mci.C.. .: /root/.viminfo
d ... n .. : /run
d <.... mc.. .. : /run/systemd/units
f =.... mc..... .: /usr/bin/date
f >.... mc..C.. .: /var/lib/csf/csf.dnscache
f >.... mc..C.. .: /var/lib/csf/csf.tempip
f =.... mc..C.. .: /var/lib/csf/lock/AT_INTERVAL.lock
f =.... mc..C.. .: /var/lib/csf/lock/LF_DIRWATCH.lock
f =.... mc..C.. .: /var/lib/csf/lock/LF_EXPLOIT.lock
f =.... mc..C.. .: /var/lib/csf/lock/PT_INTERVAL.lock
f =.... mc..C.. .: /var/lib/csf/lock/PT_LOAD.lock
f =.... mc..C.. .: /var/lib/csf/lock/ST_ENABLE_report.lock
f =.... mc..C.. .: /var/lib/csf/lock/command.lock
f >b... mc..C.. .: /var/lib/csf/stats/iptables_log
f =.... mc..C.. .: /var/lib/csf/stats/lfdstats
f >.... mc..C.. .: /var/log/lfd.log

So here we see it’s detected both /usr/bin/date and /etc/issue, as well as quite a lot of other things.  We’ll talk next about how to trim down the noise.  AIDE also gives you detailed information on the changes…here, I’ve trimmed the email down to just the two files we intentionally modified:

---------------------------------------------------
Detailed information about changes:
---------------------------------------------------

File: /etc/issue
Size : 27 | 32
Mtime : 2020-02-01 09:09:26 -0800 | 2020-04-07 17:18:08 -0700
Ctime : 2020-03-09 11:39:41 -0700 | 2020-04-07 17:18:08 -0700
Inode : 1731 | 33431
RMD160 : Bol7XoOiUO6aKpMaWAHqRIApxbs= | 7us/8GJhsItGIaiX+VKr8wT8oa4=
TIGER : JSqTrZPw9d7S50HXSU8ZYggI/H6uCUBs | lNyNWwkBUEdCTRMJSeC1JsBQ28jix8xC
SHA256 : ecDKUqYd+ifA4V9pC6ddPXQXXOEXK+/C | sgKqMRBIz9fJt5miZkDvRYPxk43bw2Oq
1lIEmp27DXI= | Be3LniYEOho=
SHA512 : pV/mBESQb4rAo6JQ6yz5vo4dVnVMgdqC | K7Q7naUcNZOa6usQa7mjeqsMBnMhu77R
hPb8D03E9ANL9fO5ROLS68HGjyOcqBtl | sTpzd0RHlhT5n8PHZBjLvSK5Ifd4idUd
/WozNwQrNa47z0PrtBJDDA== | y3uaK374jhlNGChb1rS5GQ==
CRC32 : Mk9Fiw== | rqebvQ==
HAVAL : NSpBJniUuZPLndLFtFYgfAQNFCt261xZ | DNyN6a4YuSjQBJGrQsT5UfZx/2DpnXTS
/GfU8iVjPSs= | 29XiFsU2/0M=
GOST : ZIoi5Ot+3FxA9bpstZrKbAyGa+XAUmBO | Pz05vaNaaFp6+T1oPFT+agcupDdR/s/C
7pBHdKY/GGc= | f7xmSDu5X4U=

File: /usr/bin/date
Mtime : 2019-02-28 07:30:31 -0800 | 2020-04-07 17:13:45 -0700
Ctime : 2020-03-09 11:40:10 -0700 | 2020-04-07 17:13:45 -0700

AIDE then gives some information on the AIDE database itself, and concludes with:

End timestamp: 2020-04-07 17:22:13 -0700 (run time: 1m 50s)

End of AIDE output.

End of AIDE daily cron job at 2020-04-07 17:22, run time 111 seconds

Tuning AIDE

As you can see, out of the box AIDE gives a lot of extraneous reports.  It’s doing its job, but it’s monitoring and reporting on things we don’t care about.

AIDE has a very flexible language for excluding files – you can even specify what sorts of metadata info you can are about and what you don’t.  Debian ships with a lot of rule sets kept in /etc/aide/aide.conf.d.  if you peruse those rules, you’ll see they’re by service.  Adding your own rules is straightforward.  Here we will add a rule to ignore /etc/csf/csf.deny.  CSF is an active firewall and it notes every IP it blocks in that file, so it changes nearly every day.  In /etc/aide/aide.conf.d, I create a file called 80_aide_csf and put this line inside:

!/etc/csf/csf.deny$

This means “ignore changes in a file called /etc/csf/csf.deny”.

You can read more about creating your own rules in the AIDE documentation.

Off-Site Database

One thing to consider is that if an attacker penetrates your system, he can run aide –init and generate a new DB after installing his root kit.  You may wish to copy /var/lib/aide/* to an off-site location and periodically restore it.  You could even keep it on a secure host (or cloud storage service such as Dropbox) and wget it prior to running the nightly aide report.  If you do this, be sure to periodically refresh the off-site copy otherwise the list of differences will constantly grow.

 

I'm Andrew, techno polymath and long-time LowEndTalk community Moderator. My technical interests include all things Unix, perl, python, shell scripting, and relational database systems. I enjoy writing technical articles here on LowEndBox to help people get more out of their VPSes.

No Comments

    Leave a Reply

    Some notes on commenting on LowEndBox:

    • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
    • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
    • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

    Your email address will not be published. Required fields are marked *