Controlled Files

If you want to perform a task every time a specific event occurs, you could employ various techniques for polling or log watching; however, Linux provides a more general solution in the form of a tool called incron. Incron uses the inotify subsystem [1] to listen for events that affect a filesystem, such as opening, creating, or deleting a file; accessing a directory; or changing an attribute.

The name incron suggests the common cron utility [2]; however, cron jobs are triggered by a moment in time (every Friday, once a day at 3 am, in August, etc.), whereas incron is triggered by file or directory events. The gears in your head are probably already spinning, thinking of all the potential uses of incron. So, in this article, I describe how to set up incron and put it to work in some simple examples.

Getting incron

The incron tool set is not usually preinstalled by default in most distros, but because it is an implementation of the inotify subsystem (see the box "The Origin of incron"), it is likely in your distro's repository, so you can install it through the package manager. However, if for some reason you can't install the necessary packages from your distro's repos, you can download the latest version [3] and build from source.

Note that incron is a relatively recent technology – the inotify notification framework was first implemented in kernel 2.6.13 – so if you're running an older kernel, incron will not work.

Incron comprises several bits and pieces, of which the main component is the incrond daemon. This daemon installs into /etc/init.d. Depending on your system, it can be started with one of the following commands

# /etc/init.d/incrond start
# /etc/init.d/incron start

or, if you're using systemd,

# systemctl status incron.service

or any variation thereof.

Hello incron!

Once the daemon is running, you can create your own particular "Hello World!" In this first project, I'll create a logfile and add a line to it every time a file is added to a monitored directory.

First, I create the directory I want to monitor:

$ mkdir my_dir

The incron system is similar to cron in that it saves a file for each user. This file, called incrontab, contains the events you want to monitor linked to the actions you want to run. Although you could edit this file by hand (I'll look at that a bit later), the done way of doing things is to use the incrontab instruction with, in this case, the -e (for edit) argument:

incrontab -e

This command opens the file for the current user in the preconfigured default editor – that is, whatever your $EDITOR environment variable points to (or vi if it isn't pointing to anything). As with cron, each event/task pair has to be on one line and comprises:

1. <a path> – the path to the directory or file to monitor.
2. <a mask> – the events to monitor (see Table 1).
3. <a command> – the action to run. It can be an individual command or a script that runs from the shell.

In this example, incrontab will contain a single line:

/home/<user>/my_dir IN_CREATE /home/<user>/bin/hello.sh

The hello.sh file contains the very short script,

#!/bin/bash
echo "File created." >> /home/<user>/file_log

where <user> is the username of interest. For this command to work, you must make hello.sh executable with:

chmod a+x bin/

Once you have saved and closed the user's incrontab file, the program will show the message table updated, indicating the changes have been registered. Now, if you create a file in my_dir by typing, for example,

$ touch my_dir/file1.txt

the file_log file will appear in your home directory containing the line File created.

Note the first limitation of incron: Compound operations, such as the echo instruction that redirects its output to a file in hello.sh, must be wrapped in shell script, or incron will choke. Because incron is not a shell command interpreter, it does not understand such things as the redirections, pipes, and globbings that make up a compound instruction. The only thing incron can use in the command section of the line is the instruction's name and its parameters. For example, this command

/home/paul/my_dir IN_CREATE echo "File created." >> /home/paul/file_log

would not work as expected, because it would be interpreted as follows:

• echo is the instruction to run (correct).
• "File created." is a parameter for echo (also correct).
• >> is another argument for echo (incorrect).
• /home/paul/file_log is also an argument for echo (also incorrect).

This behavior is the default for incron, so it won't generate any errors in its logs. (You can track incron's errors or lack thereof by checking /var/log/cron.) Incron simply ignores what it doesn't understand and carries on.

This example also illustrates the second limitation of incron: In the incrontab line, you have to specify the complete absolute path to the executable that, in this case, lives in the user's own bin/ directory. Although this directory might be included in the user's own $PATH environment variable, the incron daemon runs as superuser, so the executable would have to be in the root's $PATH for the daemon to find it; otherwise, you have to specify the absolute path. Thus, you must make sure your executable is in /bin, /usr/bin, /usr/local/bin, or somewhere equally accessible, or point to the program with an absolute path.

Wild Cards

This "Hello World" incron program is as silly as any other, but with a couple of small changes you can modify it to make it much more useful. For example, registering the name of a created file in the logfile is easy. Incron provides a series of default variables (for some reason called "wildcards" in incron jargon) that contain this kind of information. (See Table 2 for a full list.) For the time being, $#, which contains the name of the file affected by the event, is of interest.

Open your incrontab file again (incrontab -e) and change your monitoring line to:

/home/paul/my_dir IN_CREATE /home/paul/bin/hello.sh $#

Next, modify your hello.sh script to:

#!/bin/bash
echo "File $1 created." >> /home/paul/file_log

Now try it out by creating a new file in my_dir:

$ touch my_dir/file2.txt

If you look in file_log, you'll see that it contains a new line that says File file2.txt created..

Automatic Apache

A more serious use for incron would be to monitor a server's configuration files and order a reboot if anything changes, such as a modified Apache web server httpd.conf or apache2.con file. Begin by finding out which user has privileges to stop and restart Apache on your system and edit their incrontab.

For this exercise, assume that user is root, become superuser, and open root's incrontab for editing:

$ su
# incrontab -e

If you get an error that reads user 'root' is not allowed to use incron, edit /etc/incron.allow and add root to the list of allowed users. Then, insert the following line into root's incrontab file:

/etc/apache2/apache2.conf IN_MODIFY /etc/init.d/apache2 restart

After modifying httpd.conf, or apache2.conf (e.g., by changing the Timeout value), and saving the file, look at /var/log/apache2/error.log (Figure 1). You'll see that Apache was restarted automatically when you hit the Save button (or typed :wq).

However, you probably know that it's been a long time since Apache has had only one configuration file. Apache's configuration has been modularized and sometimes approaches tens of files spread out over several nested directories. To monitor the whole lot, you can add the following to your root's incrontab:

/etc/apache2/conf.d/ IN_CREATE /etc/init.d/apache2 restart

This method is not very subtle, but, again, it'll work. You've probably figured out that this code restarts the web server every time a new file is added to the /etc/apache2/conf.d/ directory. It doesn't check whether it is a *.conf file or not, but because Apache's conf.d directory was created for those kinds of files, it doesn't really matter.

Furthermore, you're going to want to monitor the files within the conf.d directory to see whether any of the files change. To do that, you can modify the incrontab to:

/etc/apache2/conf.d/ IN_CREATE,IN_CLOSE_WRITE /etc/init.d/apache2 restart

As you can see, if you want to monitor more than one event, you have to separate the triggers with commas. In this case, you are monitoring for both the  IN_CREATE and IN_CLOSE_WRITE events, because you also want to restart Apache when a configuration file closes after it has been modified.

Note that your root's incrontab file is now two lines long, one for each directory. You're probably thinking that using two lines for a directory and one of its subdirectories is inefficient, and that it would be more elegant if you had a way to monitor the upper directory and then drill down.

You'd be wrong.

Incron does not support recursive directory monitoring, nor are there any plans for it to do so, and there's a good reason for this decision: infinite recursive loops.

To Infinity and Beyond

Imagine that you want to monitor the files in a directory and write an entry in a log each time a file is modified. You save the log for convenience in the same directory you're monitoring.

The incrontab could look like this:

</path/to/directory/> IN_CLOSE_WRITE log_changes $#

where log_changes is the following Bash script:

#!/bin/bash
echo "`date` File $1 modified" >> </path/to/directory/>my.log

Do you start to see the problem? You've created an infinite loop.

When you finish writing the entry in my_log, the IN_CLOSE_WRITE event fires because, you know, my_log is a file in the monitored directory, which was closed after being written to. A new entry is written in my_log, and the IN_CLOSE_WRITE event fires again, because my_log is a file in the monitored directory, which was closed after being written to. IN_CLOSE_WRITE fires again and  … . You get the picture. Bad things would happen if you let something like that run unchecked.

To avoid this kind of situation, every sys admin worth his salt has two tools available: (1) common sense and (2) the IN_NO_LOOP wildcard event. This element blocks the incrontab instruction until it's completely finished, avoiding the other event from firing in the same iteration, and thus avoiding loops.

Although the following is a lousy, lazy, shoddy solution (the best move would be to store your logfile elsewhere), you can solve the problem described above with the following incrontab line:

</path/to/directory/> IN_CLOSE_WRITE, IN_NO_LOOP log_changes $#

So, you can see that drilling down into subdirectories of subdirectories of subdirectories (including soft-linked directories), if it were implemented in incron, would increase significantly the chances of infinite loops, so no recursive monitoring is allowed in incron.

Silent Alarm

In the next example, I'm going to implement a security system against intruders who try to access the system and directories containing sensitive documents. I'm not talking about that folder called secret_CIA_documents that everybody has in their home directory.

If you have learned anything from the likes of the NSA, LulzSec, and Anonymous, it is that the files most coveted by attackers are email messages. Email contains usernames, passwords, bank accounts, credit card numbers, personal images, business contracts, and on and on. A frequently used email folder is a true treasure trove for your friendly neighborhood snooper.

However, monitoring a real email folder would be a nightmare, with so many writes, rewrites, deletions, and changes making it impossible to filter out false positives. Also, are you sure you want an attacker to get as far as being able to read your messages? No, you don't. What you'll do instead is construct a honeypot by creating a .Mail directory hanging off your home directory and filling it with messages culled from, say, your spam folder. That should make for fun reading. If your email client already points to that folder to store messages, CHANGE IT! It's way too exposed.

Incron will monitor your honeypot directory as before, but with a twist: Your action will depend on what the intruder does. If the intruder opens a file for reading, you'll activate the silent alarm and warn the system administrator. However, if the intruder starts writing in files, you'll disconnect the machine from the network with, for example

ifconfig eth0 down

The problem with this solution is that, apparently, you need two lines to call two different scripts, depending on the triggered event. Something like:

/home/<user>/.Mail IN_ACCESS warn_admin.sh
/home/<user>/.Mail IN_CLOSE_WRITE close_connection.sh

But that won't work because incron doesn't allow you to monitor the same thing twice. If you try to run these lines to trigger one of the events, you will get an error in /var/log/cron that says:

Jul 24 21:17:54
host
incrond[9454]:
cannot create
watch for user root:
(16) Device or resource busy

To solve this problem, you can combine both events onto one line and use the predefined incron variable $%:

/home/paul/.Mail IN_ACCESS, IN_CLOSE_WRITE access_control.sh $%

The $% wildcard is passed to your script as an argument, telling it which event was triggered, so you can then deal with it as shown in Listing 1.

Dynamic incrontab

Something else you can do is use incron to tell users when they have new documents on a server waiting to be processed. For example, imagine you're the editor in chief of a famous Linux magazine, and you need to tell your editors when articles are available for editing, correcting, proofreading, and so on. Instead of copying the documents to the server, firing up your email client, and manually shooting off a message to the concerned party, you just want to copy the document and be done with it. You could hope that your staff will check their online folders regularly, but you can't afford to assume that will happen. That is why we script.

Instead of dumping all the documents willy-nilly into one directory, you want to be able to monitor each of the users' home directories on the server and then monitor the subdirectories that are created for each magazine and issue within those directories.

This sets another challenge: As mentioned before, incron does not implement recursivity, so you can't just monitor the top level and bore down when a folder for a new issue is created. If you monitor the users' top-level directories, incron won't see when a file is added to an issue subdirectory, so it seems you will have to create an incrontab line for each new subdirectory you create in a user's home folder, and that seems as bothersome as emailing each editor by hand – unless you write a script to do it for you.

The trick is to get your script to write directly into the users' incrontab files without having to use incrontab -e. The inrontab files live at /var/spool/incron/, and each user has their own with their name on it. Thus, user joe's incrontab will be /var/spool/incron/joe, user jane's file will be /var/spool/incron/jane, and so on. To accomplish this, you use a global, superuser incrontab with one line per editor to write in each file. The rules in root's incrontab will look something like this:

/home/joe IN_CREATE /usr/local/bin/makeIncrontab.sh $@
/home/jane IN_CREATE /usr/local/bin/makeIncrontab.sh $@
/home/jed IN_CREATE /usr/local/bin/makeIncrontab.sh $@
...

Yes, you do have to create one rule per editor, but unless you're continually hiring and firing editorial staff, you won't have to change this file very often. Each line monitors the users' home directories and calls the makeIncrontab.sh script (Listing 2) whenever a subdirectory is created in any of the monitored directories.

The makeIncrontab.sh script takes as an argument the name of the user from the user's monitored home directory (culled from the $@ variable included in each of root's incrontab rules) and removes the existing incrontab file for that user from the /var/spool/incron/ directory. Then, it loops over all the subdirectories in the user's home directory and creates a new file with rules for each subdirectory, indicating that the user must be mailed if a new file is added to any of those subdirectories.

The user's email address is extracted from the file .emailaddress that you will have created previously and placed in the user's home directory. The beauty of removing the user's incrontab file and then creating it anew every time the script is run is that the rules that refer to old directories that have been deleted get flushed out of the incrontab file.

Say, over time, you have created subdirectories in Joe's home directory for Linux Magazine 75 (lm75), Linux Magazine 76 (lm76), Shell Special 02 (shell02), and Ubuntu User 04 (uu04). His personal incrontab file will end up looking like Listing 3.

The send_mail.sh script, by the way, you'll have to write yourself. Mine is actually a Python (not Bash) script, and it sends a message with instructions on how to download the files via SSH, complete with tailored command lines (thanks again to the $@ variable) that the editors can copy and paste into a terminal.

Conclusion

Incron is fun! It gives you a very complete and powerful set of tools for monitoring and reacting to changes in your filesystem, saving you a ton of work and trouble by allowing you to link events and tasks.