Broadly speaking, there are two types of software that run on a machine: interactive software that the user launches and controls and background software that runs quietly doing what it needs to do. The interactive software is what you think you're using most often – it's the web browsers and word processors and all the other bits and pieces you launch when you want to do something. However, if you take a look at the processes on your computer (just type ps ax in a terminal window), you'll see that most of the software running isn't anything you launched, but the stuff that runs in the background quietly getting on with what it's doing. For the most part, you don't have to think about this – the software just works, and you get to focus on the interactive software and use the computer in the way you want; however, every once in a while, you have to delve a little deeper and see how this background software is running.

Because this software runs in the background, you should never see an error message, a progress bar, or any of the other user interface clues that tell you how interactive software is performing. Instead, it sends all the output to logfiles. These are text files that usually live in the /var/log directory (see the "Init Systems" box). These logfiles are designed to be readable by both humans and machines, so they follow a fixed text format, but one that's sufficiently verbose to understand. The main log on most Linux boxes is the system log, which is /var/log/syslog on Debian and Ubuntu-based machines and /var/log/messages on many others. This is a text file, so you could open it in a text editor. However, it can be very large, so it's usually best to just view the end of it. You can do this from a terminal by typing:

tail /var/log/syslog

journalctl | lnav

On CentOS and other Red Hat-based machines, you'll have to change syslog for messages. You'll see that each line corresponds to a single event that a piece of background software wants to record. For example, it might be the software that handles your network connection logging some information about the interface, or cron running a periodic command. Each line can be useful, but it can be hard to discern what's going on by just looking at a logfile. Useful information can be hidden by pages and pages of routine output. The simplest way of pulling information out of these logfiles is to use grep, which searches text and only outputs lines that match. For example, If you want to find all the lines that contain the text "cron," you can run:

grep cron /var/log/syslog

There's far more power here than I have space to cover, but things quickly gets complex, especially if you want to analyze dates or numbers, because that isn't easy to do in Linux text processing (see the "Regular Expressions" box).

A New Hope

Lnav is a tool that hopes to make it easier to extract useful information from logfiles by making it easy to import them into a SQLite-based data store and then perform advanced queries on them simply and efficiently (for larger setups, see the "Alternative Options" box). You may find the tool in your distro's repositories; otherwise, you can grab a statically linked binary that should run on any Linux system from the lnav website [2]. Unzip this file, and you can run this tool from the command line by navigating to the unzipped directory and entering ./lnav.

There's no setup or configuration for lnav, and it automatically detects most log formats, so you only need to point this tool at the logfile you want to investigate. For example:

lnav /var/log/syslog

If you're running the statically linked version that you've just downloaded, you'll need to include a path to the executable (e.g., prepend the above command with ./ if you're in the same directory as the file).

At first glance, it'll look just like you've opened the text file in a text editor, but unlike a general purpose text application, lnav understands a little about the logfiles, so it can help you find what's happening. For example, press Shift+W to jump back to the previous warning message and Shift+E to jump to the last error message (without the Shift, W and E move forward to the next message, but until you've moved back in the file, there won't be anything forward).

If you want to display only lines matching a particular pattern, you can do this with filters. First enter command mode by pressing the colon, and then enter filter-in cron to show only the lines containing the text "cron" (see Figure 1). Similarly, you can use filter-out to show everything except these lines. These filters will be applied to the whole file, so until you cancel them (with Ctrl+R), any other action takes place on the filtered log, not the log as a whole.

Figure 1: In addition to all the searching options, the colored highlighting in lnav makes logs easier to understand.

Perhaps the biggest advantage of lnav over traditional text processing methods for looking through logfiles is that it makes it easy to search by date. You can scroll through by pressing D and Shift+D to move forward and backward by 24 hours (see the "Log Rotation" box if you can't find the day's data you are looking for), whereas pressing the numbers 1 to 6 go to the next nth minute past the hour (with Shift held down, they go to the previous nth minute).

lnav /var/log/syslog*

As well as navigating by time, you can also amalgamate data by time. Press I to enter histogram mode (see Figure 2). Here, you'll get a line for each time period with the length of the highlighting varying depending on how many log messages there were at the time and the color of the highlighting showing the number of info, warning, and logfiles. In this mode, Z zooms in and Shift+Z zooms out. (See Figure 3 for other lnav hot keys.)

Figure 2: The highlighting in Histogram mode makes it easy to see how much activity has been going on over the log's period.

Figure 3: Lnav has a lot of features, but they're well documented [3] if you are having trouble.

By far the most powerful part of lnav is its ability to query the logfile using SQL. Press the semicolon to enter query mode, and then you can analyze the logfile using SQL statements. Everything is held in a single table called logline, so you can get everything with:

;select * from logline

If you look carefully, you'll see that this doesn't return all the lines in the table. Since the text lines in a logfile don't all hold the same type of data, lnav can't put them all in the same SQLite table. For example, a cron line might include details of what is run and by which user, while a dhclient line might include details of what IP addresses are assigned to a particular network interface. Lnav gets around this by only matching lines that match the pattern of the currently selected line (i.e., the top line in the terminal window) when you run a command. Press P before entering query mode to see the available columns for this log type (see Figure 4).

Figure 4: After pressing P, you can see the details lnav has managed to glean from the current log line.

I don't have time to delve fully into SQL here, but many of you will already be familiar with this database language, and SQLite supports all the major features, including aggregation, which can be a really useful tool for building up a high-level overview of what's going on in your logfiles. If you don't know SQL, then you can still find out most useful information using filters and the histogram mode in lnav.

One of the best features of the Linux command line is the ability to pipe the output of one command into the input of another. A really simple example of this pipes the bottom 10 lines of a logfile into a search for the text "cron," which you can get by running the command:

tail /var/log/syslog | grep cron

Fortunately, you don't loose this ability when using lnav, and you can send output to other programs. There are two commands for this: pipe-to and pipe-line-to. The first sends all the lines that you've bookmarked, and the second sends just the current line. Before looking at how to use these, you first need to know how to bookmark lines. The basic commands for this are m (which marks the current line), Shift+M (which marks all the lines between the last marked line and the top of the screen), and Shift+J (which marks the next line). When you mark a line, it becomes highlighted in the user interface, so you can see at a glance which lines are selected. Once you've selected the lines you are interested in, you can enter command mode by pressing the colon. The pipe-to command can be followed by any Linux commands, and it will send the data to them. For example, if you want to know how many lines you've highlighted, you can pipe them to the wc (word count) command with the l flag:

pipe-to wc -l

The pipe-line-to command works in much the same way, but it only sends the current line, regardless of what is or isn't marked.

Once you've found the lines you're looking for, you may well want to extract them for future use. This could be as simple as keeping a copy of them to use in the future, or you might be writing a report about a particular event and need to document what's happened. Whatever the purpose, you can get data out of lnav in a couple of ways. One way is to extract the bookmarked lines to another text file using the append-to <filename> or write-to <filename> commands. Alternatively, you can output the results of an SQL query as either CSV or JSON with write-to-csv <filename> or write-to-json <filename>.

This article has been a whirlwind tour of the main features of lnav and should give you a valuable new tool for finding out what's going on deep in the heart of your Linux box. Hopefully you'll find that next time something goes wrong, it should be a little easier to figure out.