LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

How to Audit Every Command Run on Your Linux System

auditPeriodically I’ve had auditors come to me and say “can you tell me what this user on this system did between such-and-such dates/times” and my answer is usually no. By default, Linux systems don’t log this info. But they can.

In this tutorial, I’ll show you how to use auditd, which is a daemon you can enable to capture every command entered.

There is one big disclaimer: a user with root can always hide his tracks. There are a couple techniques you can use to minimize this, such as using chattr (or *BSD’s securelevel) to set logs to append-only and echoing your logs to a different server. But in general, root access allows a cunning attacker to hide his tracks.

To setup auditing, you’ll need the auditd package. On Debian, to install this package, type

apt-get install auditd

Next, turn on auditing. There are many ways to filter what you want to audit but we’ll keep it simple here and audit all commands:

# auditctl -a exit,always -F arch=b32 -S execve -k allcmds
# auditctl -a exit,always -F arch=b64 -S execve -k allcmds

These commands only differ in the arch= specification – one for 32-bit syscalls and one for 64-bit.

Now you can use ausearch (audit search) to see what commands have been issued.

ausearch -k allcmds

For example, immediately after the two auditctl commands, I did a “ps -ef”. Here is what the audit record looks like:

time->Tue Oct 5 15:33:34 2021
type=PROCTITLE msg=audit(1633473214.211:33): proctitle=7073002D6566
type=PATH msg=audit(1633473214.211:33): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=1441854 dev=09:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PATH msg=audit(1633473214.211:33): item=0 name="/bin/ps" inode=11010174 dev=09:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1633473214.211:33): cwd="/root"
type=EXECVE msg=audit(1633473214.211:33): argc=2 a0="ps" a1="-ef"
type=SYSCALL msg=audit(1633473214.211:33): arch=c000003e syscall=59 success=yes exit=0 a0=edf6e8 a1=eca708 a2=fb7008 a3=59a items=2 ppid=17403 pid=18676 auid=1015 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=2532 comm="ps" exe="/bin/ps" key="allcmds"
----

You can see that the command entered was /bin/ps with the argument -ef. Interestingly, when I typed that ausearch command, the next command in the audit dump was a record for the ausearch command itself. An audit record was created before the ausearch command itself was run.

If you want to audit only a specific user, there are two approaches. You could audit all users and then search for only that user, like this:

ausearch -ue <user's uid>

Or you could setup an auditctl entry for just that user:

auditctl -a exit,always -F arch=b64 -F euid=X -S execve -k root-cmds
auditctl -a exit,always -F arch=b32 -F euid=X -S execve -k root-cmds

(where X is the uid)

To setup auditing permanently, edit /etc/audit/rules.d/audit.rules and add your rules in the same format as auditctl.

-a exit,always -F arch=b32 -S execve -k allcmds
-a exit,always -F arch=b64 -S execve -k allcmds

If you want to list active rules, use auditctl -l:

# auditctl -l
-a always,exit -F arch=b32 -S execve -F key=allcmds
-a always,exit -F arch=b64 -S execve -F key=allcmds

Restarting auditd will erase any rules that are active, though the /etc/audit/rules will be processed on startup. If you wanted to delete all rules immediately, you could type

auditctl -D

There are a ton of filtering options, both for what to capture and what to search for. Consult the auditctl(8) and ausearch(8) man pages.

2 Comments

  1. That was an interesting article, I’m going to play around with that. Curious if it captures crons too, etc.

    October 7, 2021 @ 5:40 pm | Reply
  2. Tony40:

    This is great! I always use.

    $history command to check system history.

    October 21, 2021 @ 10:39 am | Reply

Leave a Reply to Tony40 Cancel reply

Some notes on commenting on LowEndBox:

  • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
  • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
  • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

Your email address will not be published.