Even in the age of newfangled buzzwords like "Kubernetes," veteran system administrators still appreciate command-line tools like good old top, which displays running processes in real time. Since I also want to keep up with the young bloods (even at my age, which can already be described as advanced), I recently created a small, terminal-based monitor that shows Docker containers coming and going on a system.

The standard Docker client, docker, is written in Go and communicates via a web interface with the Docker daemon to query the status of running containers, start new ones, or terminate existing ones. In addition to a Python interface, Docker also offers a Go SDK. Since Go also has excellent libraries for displaying in the terminal, it was my choice for implementing the dockertop monitor in this Programming Snapshot column.

The idea is simple: The program asks the Docker daemon at regular intervals for all the containers running on the system, displays their names in a list, and refreshes them every second – just like top. As an additional treat, the monitor in the right part of the split screen also displays a rolling history of the containers. Each time it detects a new one, the program writes New: <Name> into it. If a container has been lost since the last call – for example, because it has died in the meantime – the log entry reads: Gone: <Name> (Figure 1).

Figure 1: The left column shows the active Docker containers, while the right column has the history of lost and found containers.

This gives the sys admin an impression, even on a busy system with many containers, of how the individual instances are doing. Depending on how fast the log changes, you can guesstimate whether there is a problem causing the started container to fail immediately or if everything is within bounds.

Courtesy of Google

This Programming Snapshot column has introduced terminal user interfaces (UIs) [1] several times in previous editions, including termui [2] and promptui [3]. This time it's a Google framework based on termui: Termdash, which is especially suitable for the dashboards of the data world.

Listing 1 [4] implements the graphical components of the terminal UI in Figure 1 and adds a whole litany of Go libraries from GitHub. Since the widget named container in line 7 would collide with the Docker containers used later, the code fetches the component under the name tco. The somewhat verbose handling of individual errors in Go is accelerated by the panicOnError() function starting at line 16. On a production system, the code would probably handle errors explicitly and in a dedicated way, instead of immediately aborting the program if something goes wrong. But in our example, this saves us a long listing.

01 package main
02
03 import (
04   "context"
05   "fmt"
06   "github.com/mum4k/termdash"
07   tco "github.com/mum4k/termdash/container"
08   "github.com/mum4k/termdash/linestyle"
09   "github.com/mum4k/termdash/terminal/termbox"
10   "github.com/mum4k/termdash/terminal/terminalapi"
11   "github.com/mum4k/termdash/widgets/text"
12   "strings"
13   "time"
14 )
15
16 func panicOnError(err error) {
17   if err != nil {
18     panic(err)
19   }
20 }
21
22 func main() {
23   t, err := termbox.New()
24   panicOnError(err)
25   defer t.Close()
26
27   ctx, cancel :=
28    context.WithCancel(context.Background())
29
30   top, err := text.New()
31   panicOnError(err)
32
33   rolled, err := text.New(
34     text.RollContent(), text.WrapAtWords())
35   panicOnError(err)
36
37   go updater(top, rolled)
38
39   c, err := tco.New(
40     t,
41     tco.Border(linestyle.Light),
42     tco.BorderTitle(" PRESS Q TO QUIT "),
43     tco.SplitVertical(
44       tco.Left(
45         tco.PlaceWidget(top),
46       ),
47       tco.Right(
48         tco.Border(linestyle.Light),
49         tco.BorderTitle(" History "),
50         tco.PlaceWidget(rolled),
51       ),
52     ),
53   )
54   panicOnError(err)
55
56   quit := func(k *terminalapi.Keyboard) {
57     if k.Key == 'q' || k.Key == 'Q' {
58       cancel()
59     }
60   }
61
62   err = termdash.Run(ctx, t, c,
63     termdash.KeyboardSubscriber(quit))
64   panicOnError(err)
65 }
66
67 func updater(top *text.Text,
68              rolled *text.Text) {
69   items_saved := []string{}
70   for {
71     err, items, _ := dockerList()
72     panicOnError(err)
73
74     add, remove :=
75       diff(items_saved, items)
76
77     for _, item := range add {
78       err := rolled.Write(
79         fmt.Sprintf("New: %s\n", item))
80       panicOnError(err)
81     }
82     for _, item := range remove {
83       err := rolled.Write(
84         fmt.Sprintf("Gone: %s\n", item))
85       panicOnError(err)
86     }
87
88     content := strings.Join(items, "\n")
89     if len(content) == 0 {
90       content = " " // can't be empty
91     }
92     err = top.Write(content,
93       text.WriteReplace())
94     panicOnError(err)
95
96     items_saved = items
97     time.Sleep(time.Second)
98   }
99 }

The context construct created in line 28 is a kind of remote control that subroutines pass on to each other in Go. If the main program calls the returned cancel() function, this signals the end to the context, and all subroutines get the message and can initiate cleanup actions.

The application's main window contains two text windows side by side, as seen in Figure 1. The top widget displays the list of active containers similar to the top Unix utility, while the rolling log window (rolled) to the right provides the historical view of containers coming and going. To arrange them side by side, the code employs the helpers Left() and Right() with a call to SplitVertical() in the terminal. When the user presses Q, you want Go to clear the UI and abort the program. This is why line 56 defines in quit a callback of the keyboard watchdog that triggers when the user presses the corresponding key. Once in action, the callback in line 58 calls the cancel() function of the previously created context, which in turn triggers lower-level cleanup functions.

For the UI to be able to react to changes in the context, the object is passed to the UI main loop starting with Run() in line 62, along with a list of all widgets to be used. When it's time to close shop, the UI's internal main event loop detects this via the passed in context and neatly winds down the UI. Without a controlled exit, the program would leave the terminal in graphics mode, in which case the user would no longer be able to enter shell commands or get a proper prompt. Closing the terminal window and opening a new one is usually the only way out of a mess like this.

Groundhog Day

The Go routine updater() called asynchronously from line 37 defines the time loop that refreshes the UI with the latest data from the Docker daemon every second. Starting at line 67, it fetches the list of containers via dockerList(), which I'll get to in a bit in Listing 2. The left subwindow with the top view refreshes itself with the call to top.Write() in line 92 in Listing 1 with a long content string containing the individual container names with 10 characters of their ID, separated by line breaks.

01 package main
02
03 import (
04   "context"
05   "fmt"
06   "github.com/docker/docker/api/types"
07   "github.com/docker/docker/client"
08 )
09
10 func dockerList() (error, []string,
11     map[string]types.Container) {
12   items := []string{}
13   containerMap :=
14     make(map[string]types.Container)
15
16   opt :=
17     client.WithAPIVersionNegotiation()
18   cli, err :=
19     client.NewClientWithOpts(opt)
20   if err != nil {
21     return err, nil, nil
22   }
23   defer cli.Close()
24
25   containers, err := cli.ContainerList(
26       context.Background(),
27       types.ContainerListOptions{})
28   if err != nil {
29     return err, nil, nil
30   }
31
32   for _, container := range containers {
33     name := fmt.Sprintf("%s-%s",
34       container.Image, container.ID[:10])
35     items = append(items, name)
36     containerMap[name] = container
37   }
38
39   return nil, items, containerMap
40 }

Containers that the monitor sees for the first time are reported by the diff() function called in line 75 of Listing 1. You'll see its inner workings later in Listing 4, but for now it just returns two array slices, add and remove, which are generated from the difference between the last container listing (items_saved) and the current one (items). All these steps are embedded in an endless for loop, at the end of which, in line 97, the call to time.Sleep() pauses for one second before it enters the next round. The loop and the sleep command run in a Go routine (i.e., asynchronously), and thus the UI remains fully responsive.

01 $ go get -u github.com/docker/docker/client
02 $ go build dockertop.go dockerlist.go dockerdiff.go

package main
import "github.com/yudai/golcs"
func diff(old []string,
          new []string) (add []string,
            remove []string) {
  left := make([]interface{}, len(old))
  for i, v := range old {
    left[i] = v
  }
  right := make([]interface{}, len(new))
  for i, v := range new {
    right[i] = v
  }
  l := lcs.New(left, right)
  leftidx := 0
  rightidx := 0
  for _, pair := range l.IndexPairs() {
    for leftidx < len(left) &&
        leftidx <= pair.Left {
      if leftidx < pair.Left {
        remove =
          append(remove, old[leftidx])
      }
      leftidx++
    }
    for rightidx < len(right) &&
        rightidx <= pair.Right {
      if rightidx < pair.Right {
        add = append(add, new[rightidx])
      }
      rightidx++
    }
  }
  for leftidx < len(left) {
    remove = append(remove, old[leftidx])
    leftidx++
  }
  for rightidx < len(right) {
    add = append(add, new[rightidx])
    rightidx++
  }
  return add, remove
}

That was it for the UI, whose implementation neatly fits into 99 lines. So how does the Go program get access to the active containers' names on the system? The Docker API's individual components and their functions are described in great detail on the project's website, which has a link to automatically generated documentation from comments in the Go source code [5].

However, with its open source Moby project, Docker has cooked up a strange brew here and does not follow the versioning common in the Go community. Consequently, the otherwise successful go mod init, which is used to prepare Listing 2 for compilation by fetching the source code from GitHub during the build phase, does not work. Instead, the user has to install the library (Listing 3, line 1) and repeat the process with all libraries pulled in by import statements in the listings. Only then can you build the dockertop binary (line 2). If you used the modern module method, it would fail, because the Docker API delivers an ancient version that does not support some functions used in the listings.

Hello Daemon, Client Speaking

As a simple Docker client, which fetches the list of all containers from the daemon, docker ps called from the shell would also be useful; its standard output would dump out the names. Instead, I'm using the Docker Client API – because I can, and because it can later be extended at will – but it takes a little more effort.

Line 19 in Listing 2 creates a new client object and passes the parameter WithAPIVersionNegotiation to it. This is enormously important: Without it, the client on a somewhat outdated Ubuntu system complains that the server is rejecting it, because the client version number is supposedly too high. But passing the version negotiation parameter fixes the problem, and both start talking to each other. ContainerList() returns a list of active container objects, sorted by start date. The Docker image for each container can be found in the .Image attribute and will be displayed in the UI alongside the container ID.

In order for the client to be able to distinguish between several Ubuntu containers running in parallel, line 34 uses container.ID[:10] to add the first 10 characters of the container's unique ID. The names of all containers found in this way are appended to a slice of strings in line 35, so that the original order in which the server reported them is retained.

Additional information on each container ends up in the containerMap attribute under items. This allows other program parts to access the correctly sorted list, as well as more details if required. dockerList() returns both data structures to the caller.