Detecting when you need to system rescue
System Rescue
Kurt provides some tips and recommends some tools to help you detect signs of network intrusion and data corruption.
System rescue – it's definitely an important topic with lots of considerations. Do you go with "bare-metal restore" or just back up the data and all the configs? What about your database? Do you snapshot it, or replicate it and keep a transaction log? What about all the new NoSQL things? More to the point, how do you know when you need to do a system rescue?
Sometimes it's pretty obvious, like when some water spilled onto one of my machines; I stared in horror as the machine made a loud "pop" and the power supply killed the motherboard and then itself. Luckily, I didn't lose any data. Sometimes, however, it's not so clear when you have to do a system rescue. For example, if you find a corrupted file on your system, do you have other corrupted files? Short of opening them all and checking them, you don't know whether you have just one bad file or a completely corrupted filesystem.
File Integrity to the Rescue
Such problems have plagued administrators, well, since computers have had read/write data storage. The good news is that several mature tools can help you address the problems of managing files and ensuring that they are not modified or corrupted. Certain strategies are also helpful when designing and architecting systems to make things more robust. Ultimately, the goal is to prevent data corruption or improper modification as much as possible – by using file permissions, robust filesystems with journaling, and so on. Then, you need to ensure that you can detect file corruption and improper modification and, finally, restore things to a known good state. The two main tools for these tasks are Open Source Tripwire [1] and AIDE [2]. Neither has undergone major changes for a few years, mostly because they are fairly feature complete.
Tripwire
Tripwire, first written in 1992, is the granddaddy of file integrity tools. It quickly became popular and was eventually taken commercial, with an open source version remaining available. Open Source Tripwire hasn't undergone an update since late 2011. As I mentioned, it's pretty feature complete – except for hashing algorithms: Open Source Tripwire supports CRC-32 (trivial for an attacker to bypass), HAVAL (weaknesses were found as far back as 2004, so it's probably not a good choice), MD5, and SHA (both of which are showing their age).
Basically Open Source Tripwire doesn't support any modern hashing algorithms (e.g., SHA256 or SHA512). Although MD5 and SHA are hard to break, the skills of attackers keep improving, and it's unlikely that Open Source Tripwire will ever get support for modern hashing algorithms. It also seems to lack support for checking extended file attributes (xattr). Although it can check the basic file permissions (user, group, other), it can't check xattrs, meaning attackers can potentially add themselves to a file or directory and remain undetected. As such, if you have strong security requirements, you should probably consider moving away from Open Source Tripwire. Commercial versions of Tripwire are available, but I've never tried them because I'm not a big fan of closed source security.
AIDE
Luckily, you have a second option, AIDE. AIDE was created as a replacement for Tripwire and has had somewhat more active development. AIDE does support modern hashing algorithms such as SHA256 and SHA512, so the chances of an attacker modifying a file and managing to keep the hash the same on it are pretty nonexistent at this time (and probably for the next 10-20 years). AIDE also supports extended attributes, which is pretty important, because most Linux distributions now default to filesystems like ext4, XFS, and Btrfs, all of which support xattr by default.
Open Source Tripwire and AIDE operate in largely the same manner. You configure them to check certain files and directories, and they create a database of the file and directory permissions, ownership, size, access and modification times, a hash value of the data (if it's a file), and so on. You then run these tools periodically, and they recheck all the files to see whether anything has changed. If it has, the changes are logged, and you can configure the tools to email you a report.
I won't go into installation, because the tools are available as packages for virtually every distribution. Also, I won't cover configuration, because they have pretty solid default policies. I will, however, discuss where things can go horribly wrong and how to prevent that.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Endless OS 6 has Arrived
After more than a year since the last update, the latest release of Endless OS is now available for general usage.
-
Fedora Asahi 40 Remix Available for Macs with Apple Silicon
If you've been anticipating KDE's Plasma 6 for your Apple Silicon-powered Mac, then you're in luck.
-
Red Hat Adds New Deployment Option for Enterprise Linux Platforms
Red Hat has re-imagined enterprise Linux for an AI future with Image Mode.
-
OSJH and LPI Release 2024 Open Source Pros Job Survey Results
See what open source professionals look for in a new role.
-
Proton 9.0-1 Released to Improve Gaming with Steam
The latest release of Proton 9 adds several improvements and fixes an issue that has been problematic for Linux users.
-
So Long Neofetch and Thanks for the Info
Today is a day that every Linux user who enjoys bragging about their system(s) will mourn, as Neofetch has come to an end.
-
Ubuntu 24.04 Comes with a “Flaw"
If you're thinking you might want to upgrade from your current Ubuntu release to the latest, there's something you might want to consider before doing so.
-
Canonical Releases Ubuntu 24.04
After a brief pause because of the XZ vulnerability, Ubuntu 24.04 is now available for install.
-
Linux Servers Targeted by Akira Ransomware
A group of bad actors who have already extorted $42 million have their sights set on the Linux platform.
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.