If your existence depends on extreme security, you could put your computers in a bunker and avoid external network connections, but unless you happen to be dealing with nuclear missiles, the realities of doing business might require you to configure some form of Internet access, and the price for joining the information superhighway is a need to watch for attackers attempting to penetrate your network.

IT professionals deploy a dizzying variety of tools for protecting the modern network, from firewalls to sophisticated intrusion prevention frameworks. (See the box titled "NIDS, IPS, HIDS.") One useful class of tools that receives less attention than it used to is the humble host-based intrusion detection system (HIDS). HIDS systems perform checks on a single host computer to look for indications of a recent attack.

HIDS works on the principle that every attacker leaves tracks. In forensics, this assumption is known as Locard's exchange principle: Interaction always causes change. The trick is to identify this change and draw the right conclusions [1].

The HIDS was once a primary line of defense, although today's enterprise networks often opt for bigger and more comprehensive network-based systems. In the spirit of Unix, however, a simple tool with a specific purpose is often a healthy addition to the admin toolkit. A typical HIDS system takes one of the following approaches for detecting the footprint of a past attack:

• track changes to the filesystem
• analyze logfiles
• search for malware

In this article, I describe some popular HIDS tools that highlight each of these approaches. You'll also learn about some all-in-one tools that combine these techniques with a single tool.

Changes to the Filesystem

An attacker gets extreme pleasure from breaking into a third-party system through a buffer overflow. But once the intruder is inside, the work becomes much more monotonous. The attacker starts by installing a rootkit to provide convenient access. To set up a rootkit, the intruder adds software to the system and executes this software – which is something a system administrator might notice when running ps. To use a backdoor to control the target system, the attacker needs a network listener, resulting in an open port that would normally be visible to netstat. Attackers need to prevent their programs from appearing in the output of typical admin tools. Traditionally, intruders replaced the executables for common admin utilities like ps and netstat with fake versions that conceal their presence. More modern approaches, such as kernel rootkits, install a separate kernel module that blocks the visibility of rootkit processes in the kernel itself.

Administrators can identify Linux tools replaced by intruders on the basis of the different modification and creation times, the file sizes, or the fact that binaries use different inodes. In practical terms, administrators who are not using some form of automated security system will never notice the change because manual checks are far too time-consuming. Classical HIDS tools like Tripwire [2], AIDE [3], or the newer AFICK [4] provide an automated means for finding filesystem changes.

The syntax these free tools use in their configuration files is very similar. AFICK, which is written in Perl, actually advertises its Tripwire compatibility. All three tools create a list of critical files and directories, along with their modification and creation times, sizes, inode numbers, and permissions. Comparing these parameters with the results of later checks helps identify attacks.

Of course, capable attackers are likely to build their malware so that the file sizes are identical; they will set identical access times using touch, keep the file permissions, and bundle the modified files into the same inodes (cat newps > ps). The HIDS, however, does not just keep records of the visible data but also of the files' hash values.

AIDE even offers the ability to apply multiple hash processes in parallel to make collisions even less likely: For MD5, the probability is less than 264, and less than 2128 for SHA-256. The probability of a collision in both procedures is thus less than 2192, that is about 1.5x1058. The comparison: If you play the lottery, your chances of winning a jackpot are 7x109.

AIDE is a good example of an HIDS tool that looks for file changes. On Debian-based systems, install AIDE with

apt-get install aide

You can modify the configuration to suit your needs. To change the configuration, edit the /etc/aide/aide.conf file, and for special cases, also /etc/aide/aide.conf.d. The objective is to make sure AIDE keeps an eye on all the relevant programs and configuration files.

The aide.conf file is also where your own macro definitions will reside. Studying the macro definitions can save you a huge amount of work when it comes to writing rules. AIDE detects any new files created – for example, as a consequence of a core dump, which is sometimes associated with a buffer overflow. (Core dumping is disabled by default on most current flavors of Linux [5].)

The default configuration offers meaningful rulesets for logfiles and for determining the system status. The configuration file even contains a good explanation for how to derive your own ruleset.

AIDE uses update-aide.conf to update its configuration; you can type aide -D to perform a syntax check. The command

aideinit -y -f

initializes the database. AIDE now computes the hashes – a process that took some 10 minutes on the virtual machine in my lab. Because a cron job is automatically installed when you install AIDE, the tool checks the changes and mails them to the root user. AIDE can also output the results to any URL (aide.conf report_url entry) so you can collect reports in a central location.

AIDE emulates its role models so closely – you can easily migrate the setup I just described to Tripwire.

Logs, Logs, Logs

Any admin's daily chore list should include checking the log files. Did a process unexpectedly terminate? Was the process restarted – either by a watchdog or by an attacker attempting to conceal their tracks? Do any other clues indicate attempted intrusions? Regularly investigating the logs helps you identify problems. Fortunately, some tools help administrators combat any tendency toward laziness – these tools analyze logfiles, filter out potentially critical messages, and mail the results to the admin.

Logsentry [6] (formerly called Logcheck) is a useful analysis tool: This simple but complete HIDS consists entirely of shell scripts and configuration files. Logsentry saves administrators the trouble of thinking about the right grep commands to detect attacks on the basis of logfile analysis. Download Logsentry, then install it with the following commands:

tar -xzf logcheck-1.3.17.tar.gz
cd logcheck-1.3.17
make linux

It is a good idea to set up a cron job to check your logs regularly. Still working as root, type

crontab -e

to open the crontab for editing. The line

makes sure the logcheck.sh test script runs every hour. If you want to check more than the three logfiles configured by default, modify logcheck.sh from line 172 using the preceding lines as an example. Logcheck defines its rules in multiple configuration files below /etc/logcheck.

Everything in logcheck.hacking appears right at the top of the mail message to root. The logcheck.violations follow slightly lower down. Everything that is not excluded by ignore ends up in the message – including Unusual system events. The only important thing is to make sure you don't leave logcheck.ignore empty – this would be equivalent to a wildcard that ignores everything, thus rendering the tool useless.

Logsentry and its siblings save administrators some work, but this is actually all they do. Anyone who needs to inspect their logfiles regularly is likely to have programmed a simple search script of their own that filters out non-critical messages. Logsentry does exactly the opposite: When you set up the tool, you need to know exactly what critical messages look like. The tool only filters for known suspects, not for assumed non-suspects.

Also, the practice of checking logfiles relies on the attacker lacking ability. Any capable attacker will clean the logfiles on the guest system before leaving, and Logsentry will not find anything suspicious. The attacker will probably have enough time to clean up, depending on how often cron runs Logsentry. In general, a tool that looks for attackers by checking logfiles is only useful if it runs in combination with other systems.

Searching for Malware

If log analysis and file checks reveal the aftermath of an infection, it makes sense to look for the malware code. This task is handled by malware scanners, which also exist for Linux. ClamAV [7] is one example. The demarcation between malware scanners and host-based intrusion detection systems is not clearly defined; most antivirus scanners use both signatures and heuristics to detect attacks.

Programs such as Chkrootkit [8] and RKHunter [9] (a tool written in Perl) use a more specific approach; both of these tools were last updated in 2014, and you can install both on Ubuntu as root with the following commands:

apt-get install chkrootkit
apt-get install rkhunter

Chkrootkit is a shell script that attempts to identify rootkits based on various parameters. You can run Chkrootkit without specifying any parameters. Among other things, the Chkrootkit script uses strings to detect suspicious strings, checks the lastlog to see if anyone has removed entries, checks to see if ps is not showing certain process IDs, and looks for missing information in /proc/ (Figure 1).

Figure 1: Chkrootkit making various attempts to detect signs of typical rootkit activity.

RKHunter has a more sophisticated approach. The hunter's configuration file is named /etc/rkhunter.conf. (I had some issues setting up RKHunter on my Ubuntu system – see the box titled "Ubuntu Fix.")

cpan Mock::LWP::Request

You can initialize RKHunter and check the system for rootkits and conspicuous files:

rkhunter --propupd
rkhunter --check

Figure 2 gives an impression of what RKHunter considers to be a potential attack. You can run both Chkrootkit and RKHunter on a daily basis using cron, and they will thus become part of your HIDS.

Figure 2: One part of RKHunter's work is checking for illegal material.