LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

How to Audit Every Command Run on Your Linux System

Tags: , , , , Date/Time: October 7, 2021 @ 12:00 am, by raindog308

auditPeriodically I’ve had auditors come to me and say “can you tell me what this user on this system did between such-and-such dates/times” and my answer is usually no. By default, Linux systems don’t log this info. But they can.

In this tutorial, I’ll show you how to use auditd, which is a daemon you can enable to capture every command entered.

There is one big disclaimer: a user with root can always hide his tracks. There are a couple techniques you can use to minimize this, such as using chattr (or *BSD’s securelevel) to set logs to append-only and echoing your logs to a different server. But in general, root access allows a cunning attacker to hide his tracks.

To setup auditing, you’ll need the auditd package. On Debian, to install this package, type

apt-get install auditd

Next, turn on auditing. There are many ways to filter what you want to audit but we’ll keep it simple here and audit all commands:

# auditctl -a exit,always -F arch=b32 -S execve -k allcmds
# auditctl -a exit,always -F arch=b64 -S execve -k allcmds

These commands only differ in the arch= specification – one for 32-bit syscalls and one for 64-bit.

Now you can use ausearch (audit search) to see what commands have been issued.

ausearch -k allcmds

For example, immediately after the two auditctl commands, I did a “ps -ef”. Here is what the audit record looks like:

time->Tue Oct 5 15:33:34 2021
type=PROCTITLE msg=audit(1633473214.211:33): proctitle=7073002D6566
type=PATH msg=audit(1633473214.211:33): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=1441854 dev=09:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PATH msg=audit(1633473214.211:33): item=0 name="/bin/ps" inode=11010174 dev=09:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1633473214.211:33): cwd="/root"
type=EXECVE msg=audit(1633473214.211:33): argc=2 a0="ps" a1="-ef"
type=SYSCALL msg=audit(1633473214.211:33): arch=c000003e syscall=59 success=yes exit=0 a0=edf6e8 a1=eca708 a2=fb7008 a3=59a items=2 ppid=17403 pid=18676 auid=1015 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=2532 comm="ps" exe="/bin/ps" key="allcmds"
----

You can see that the command entered was /bin/ps with the argument -ef. Interestingly, when I typed that ausearch command, the next command in the audit dump was a record for the ausearch command itself. An audit record was created before the ausearch command itself was run.

If you want to audit only a specific user, there are two approaches. You could audit all users and then search for only that user, like this:

ausearch -ue <user's uid>

Or you could setup an auditctl entry for just that user:

auditctl -a exit,always -F arch=b64 -F euid=X -S execve -k root-cmds
auditctl -a exit,always -F arch=b32 -F euid=X -S execve -k root-cmds

(where X is the uid)

To setup auditing permanently, edit /etc/audit/rules.d/audit.rules and add your rules in the same format as auditctl.

-a exit,always -F arch=b32 -S execve -k allcmds
-a exit,always -F arch=b64 -S execve -k allcmds

If you want to list active rules, use auditctl -l:

# auditctl -l
-a always,exit -F arch=b32 -S execve -F key=allcmds
-a always,exit -F arch=b64 -S execve -F key=allcmds

Restarting auditd will erase any rules that are active, though the /etc/audit/rules will be processed on startup. If you wanted to delete all rules immediately, you could type

auditctl -D

There are a ton of filtering options, both for what to capture and what to search for. Consult the auditctl(8) and ausearch(8) man pages.

I'm Andrew, techno polymath and long-time LowEndTalk community Moderator. My technical interests include all things Unix, perl, python, shell scripting, and relational database systems. I enjoy writing technical articles here on LowEndBox to help people get more out of their VPSes.

2 Comments

  1. That was an interesting article, I’m going to play around with that. Curious if it captures crons too, etc.

    October 7, 2021 @ 5:40 pm | Reply
  2. Tony40:

    This is great! I always use.

    $history command to check system history.

    October 21, 2021 @ 10:39 am | Reply

Leave a Reply

Some notes on commenting on LowEndBox:

  • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
  • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
  • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

Your email address will not be published. Required fields are marked *