LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

Instant Reactions to Filesystem Events with inotify

Tags: , , , , , Date/Time: May 20, 2021 @ 11:58 pm, by raindog308

The ability to run programs automatically is one of the great liberating powers of computers. In Unix, it’s common for users to schedule jobs using cron or at (or with systemd, timers). But what if you want to run a job not based on a schedule but rather in response to an event?

Imagine you had a directory and whenever you put a file in that directory, you wanted it to be compressed and put in a backup directory. Or perhaps you have a “share” directory where individuals can upload pictures and (after testing it if it’s a picture), you automatically move them to your web directory and add them to your hosting. Or perhaps you have a complicated backup system and after kicking off, it places a flag file in a certain spot.

In the old days, people would write cron jobs that ran every minute to see if files existed, etc. These are crude, resource intensive, and often bug prone – for example, you have to guard against multiple copies of these frequently-running scripts executing at the same time.

With Linux, we can use inotify (inode notifications), which react to filesystem changes at a very fine-grained level. Typically, you’d create a daemon that sets up filesystem watches, runs in the background, and reacts to events by calling some code or firing off a script.

There are several ways of using inotify:

  • If you’re a kernel programmer, you can use the C API
  • There are two command-line programs (inotifywatch and inotifywait, part of the inotify-tools package in Debian). I find them not very useful. inotifywatch gathers statistics and inotifywait simply waits until filesystem events occur. inotifywait does have a daemon option but it’s limited to logging events to a file.
  • Python has an inotify module, which is ideal for writing daemons.

Installing Python’s inotify

To install the module, we’ll use pip. Install pip if you don’t have it, and then install the inotify module:

apt -y install python-pip
pip install inotify

Using Python’s inotify

Let’s start with an example script so you get the feel of inotify. This script watches the directory /tmp/test and responds to file change events occurring there. First, setup the test:

mkdir /tmp/test

Now put the following in a file calleed test_inotify.py:

#!/usr/bin/python

import inotify.adapters

i = inotify.adapters.InotifyTree('/tmp/test')

for event in i.event_gen(yield_nones=False):
  (_, type_names, path, filename) = event
  print("PATH=[{}] FILENAME=[{}] EVENT_TYPES={}".format( path, filename, type_names))

Note: Because this is Python, be sure that the indented lines are indented with a single TAB character.

Now execute it:

chmod 755 test_inotify.py
./test_inotify.py

Initially, nothing happens. Now open a second session to your VPS and execute some filesystem commands in that directory. You’ll see events fire off as we create, populate, chmod, and delete files:

touch /tmp/test/LowEndFile
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_CREATE']
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_OPEN']
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_ATTRIB']
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_CLOSE_WRITE']

echo 'put something in' >> /tmp/test/LowEndFile
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_OPEN']
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_MODIFY']
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_CLOSE_WRITE']

chmod 600 /tmp/test/LowEndFile 
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_ATTRIB']

rm /tmp/test/LowEndFile
PATH=[/tmp/test] FILENAME=[LowEndFile] EVENT_TYPES=['IN_DELETE']

You can hit control-C to stop test_inotify.py

As you can see, each change fires off one or more events. Some such as touching a file (creating it) fire off multiple events.

Creating an inotify-based Daemon

Let’s do something more useful. We’ll create a daemon that we can start at system time with systemd and have it react to events by calling a shell script. This is just a skeleton you can take and extend.

First, install Python’s daemonize module:

pip install daemonize

In /usr/local/bin, create the following file as watchfs.py:

#!/usr/bin/python

# CONFIG
directory_to_watch = '/tmp/test'
script_to_call = '/usr/local/bin/react.sh'
event_type_to_watch = 'IN_CREATE'
# END CONFIG

from daemonize import Daemonize
import inotify.adapters
import subprocess

pid = "/run/watchfs.pid"

def main():

i = inotify.adapters.InotifyTree(directory_to_watch)

for event in i.event_gen(yield_nones=False):
  (_, type_names, path, filename) = event
  for event_type in type_names:
    if event_type == event_type_to_watch: 
      subprocess.call([script_to_call,filename])

daemon = Daemonize(app="watchfs",pid=pid, action=main)
daemon.start()

Again, remember that each indented line must be indented with the right number of TABs.

If this looks like Greek to you, don’t sweat it. The only lines you need to modify to customize it are in the lines between CONFIG and END_CONFIG:

  • which directory you want to watch (directory_to_watch)
  • which script you want to call in response (/usr/local/bin/react.sh)
  • which filesystem event you want to watch for (here, IN_CREATE)
Now create this file in /usr/local/bin/react.sh:

#!/bin/bash

FILENAME="${1}"
echo "new file: $FILENAME" >> /tmp/reaction.log

All this script does is write the name of the file to /tmp/reaction.log. You can easily extend it to do whatever you want with the file name it receives.

Make both files executable:

chmod 755 /usr/local/bin/watchfs.py
chmod 755 /usr/local/bin/react.sh

Let’s set it up in systemd. Place the following file in /etc/systemd/system/watchfs.service:

[Unit]
Description=watchfs
After=network.target

[Service]
PIDFile=/run/watchfs.pid
ExecStart=/usr/local/bin/watchfs.py

[Install]
WantedBy=multi-user.target

Now:

systemctl daemon-reload
systemctl enable watchfs
systemctl start watchfs

Let’s test it:

# touch /tmp/test/tutorial
# cat /tmp/reaction.log
new file: tutorial

This is a basic example but it should be obvious how you can customize it and extend it to more watches. Since it’s just calling a shell script, you can tailor react.sh to do whatever you want when new files are created. If you know (or want to learn) a little Python, you can easily watch multiple directories with a table of events and actions. Enjoy!

I'm Andrew, techno polymath and long-time LowEndTalk community Moderator. My technical interests include all things Unix, perl, python, shell scripting, and relational database systems. I enjoy writing technical articles here on LowEndBox to help people get more out of their VPSes.

1 Comment

  1. You might find Watchman useful – It lets you do this sort of watching without having to write a Python script. It uses inotify on Linux, but also supports MacOS (FSEvents), BSD (kqueue) and Windows.

    https://facebook.github.io/watchman/
    https://facebook.github.io/watchman/docs/watchman-make.html

    May 23, 2021 @ 7:08 pm | Reply

Leave a Reply

Some notes on commenting on LowEndBox:

  • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
  • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
  • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

Your email address will not be published. Required fields are marked *