It's not only hyperactive millennials; even veteran Internet users lose patience when it takes longer than a few seconds for a website to load in the browser. What is especially annoying is when there isn't a clue to what is going on and how long it's going to take. Some 40 years ago, this prompted a smart programmer to invent the progress bar [1], reassuring the user: "Already 10 percent down, 90 to go, and we'll make it through the rest at the following speed."

Hollywood thrillers also love progress bars (Figure 1). When the movie spy downloads sensitive data onto a USB stick, it seems to take forever, and the progress bar keeps ticking really slowly, while the bad guys are approaching, just about to barge in at any moment and blow the spy's cover!

Figure 1: Some Hollywood movies would only be half as exciting without the progress bar.

Some Unix command-line tools already feature built-in progress bars. For example, curl typically learns at the outset how many bytes a web page contains and, thanks to the -# (or --progress-bar) option, shows you the real-time data flow:

$ curl -# -o data http://...
########### 50.6%

As another example, the dd tool, which is often used to copy disk data, recently started showing the progress (as of the GNU Coreutils v8.24) if the user sets the status=progress option (Figure 2).

Figure 2: Since Coreutils v8.24, dd starts showing progress with the status=progress option.

Bash & Co.

Friends of shell programming use the Linux pv tool, which helps out utilities without built-in progress bars. Slotted in as an adapter between two sections of a pipe, it uses an ASCII bar to indicate the progress of the data through the pipe by counting the bytes flowing through it. To be able to discover what fraction of the expected total amount has flowed through, and what still needs to be done, it needs to know the total amount of data in advance, to then accurately update the progress bar in regular intervals. It simply calculates the percentage value from the division of the bytes counted so far by the total quantity.

Simply inserted between two pipe sections, however, pv knows nothing about the total byte count to be expected and can therefore only count bytes that have already flowed so far (Figure 3, top). If you want to help out pv in this role as a "pipe gauge," you can specify the total expected amount of data (if known in advance) with the -s bytes option, in which case pv draws and updates a nice progress bar.

Figure 3: pv shows the progress of reading a file as a bar.

But if you give pv the name of a file, it acts as a cat command and can determine how large the file is, before forwarding its data byte by byte, and will display the progress bar correctly without further assistance, as shown by the last backup command in Figure 3.

Doing It Yourself

If you like to put together your own tools, chances are you will find a suitable progress bar library for your programming language on GitHub. For Go, for example, you can use the simple ASCII art displaying tool progressbar. The command

$ go get github.com/schollz/progressbar

will retrieve it directly from GitHub and install it in your Go path. Listing 1 [2] shows a simple web client titled webpgb, which fetches a URL from the web and at the same time shows in a progress bar how far the download has progressed:

$ ./webpgb http://...
13%[##----------][16s:1m49s]

01 package main
02
03 import (
04   "os"
05   "net/http"
06   "io"
07   pb "github.com/schollz/progressbar"
08 )
09
10 func main() {
11   resp, err := http.Get(os.Args[1])
12   buffer := make([]byte, 4096)
13
14   if err != nil {
15     panic(err)
16   }
17
18   if resp.StatusCode != 200 {
19     panic(resp.StatusCode)
20     return
21   }
22
23   bar := pb.NewOptions(
24     int(resp.ContentLength),
25     pb.OptionSetTheme(
26       pb.Theme{Saucer: "#",
27         SaucerPadding: "-",
28         BarStart:      "[",
29         BarEnd:        "]"}),
30     pb.OptionSetWidth(30))
31
32   bar.RenderBlank()
33
34   defer resp.Body.Close()
35
36   for {
37     n, err := resp.Body.Read(buffer)
38     if err == nil {
39       bar.Add(n)
40     } else if err == io.EOF {
41       return
42     } else {
43       panic(err)
44     }
45   }
46 }

Line 7 imports the progress bar library as pb into the main program, which uses os.Args[1] to gobble up the first command-line parameter passed to it, the URL, which it then fetches off the web with the Get() function from the standard net/http package.

For piecing the data together from the incoming chunks, line 12 defines a buffer as a Go slice with a length of 4096 bytes; the infinite loop starting in line 36 uses Read() to fill it with up to 4,096 characters from the arriving HTTP response part, until the web server is done and sends an EOF, which line 40 detects and goes ahead to exit from the main function. Meanwhile, line 39 uses Add() to refresh the progress bar display by sending it the number of bytes received in the buffer.

Previously, line 23 defined a new bar structure in the bar variable and initialized its maximum length to the total number of bytes expected from the web request. Lines 24 to 30 also define cosmetic settings, such as the ASCII character for the saucer, that is, the previously unidentified flying object that illustrates the progress (# in this case), the bar frame as [], and the fill character for the empty bar as -.

Since the data bytes from the web request arrive in chunks from the web anyway, the progress bar and the logic to refresh it can be organically integrated into the code. On the other hand, if long running system calls determine the program's run time, they need to be rewritten so that the bar can progress step by step, instead of pausing until shortly before the end, before jumping to the finish at warp speed.

Retro Look

If you are looking for more eye candy than just command-line characters, you can impress your users with a terminal UI, such as the termui project I introduced in a previous article [3]. Figure 4 illustrates how Listing 2 displays its progress while copying a large file.

01 package main
02
03 import (
04     ui "github.com/gizak/termui"
05     "io/ioutil"
06     "os"
07     "fmt"
08     "log"
09 )
10
11 func main() {
12   file := os.Args[1];
13   err := ui.Init()
14   if err != nil {
15     panic(err)
16   }
17   defer ui.Close()
18
19   g := ui.NewGauge()
20   g.Percent = 0
21   g.Width = 50
22   g.Height = 7
23   g.BorderLabel = "Copying"
24   g.BarColor = ui.ColorRed
25   g.BorderFg = ui.ColorWhite
26   g.BorderLabelFg = ui.ColorCyan
27   ui.Render(g)
28
29   update := make(chan int)
30   done := make(chan bool)
31
32     // wait for completion
33   go func() {
34     <-done
35     ui.StopLoop()
36   }()
37
38    // process updates
39   go func() {
40     for {
41       g.Percent = <-update
42       ui.Render(g)
43     }
44   }()
45
46   go backup(file, fmt.Sprintf("%s.bak", file),
47             update, done)
48
49   ui.Handle("/sys/kbd/q", func(ui.Event) {
50     ui.StopLoop()
51   })
52
53   ui.Loop()
54 }
55
56 func backup(src string, dst string,
57  update chan int, done chan bool) error {
58
59  input, err := ioutil.ReadFile(src)
60  if err != nil {
61    log.Println(err)
62    done <- true
63  }
64  total := len(input)
65  total_written := 0
66
67  out, err := os.Create(dst)
68  if err != nil {
69    log.Println(err)
70    done <- true
71  }
72
73  lim := 4096
74  var chunk []byte
75
76  for len(input) >= lim {
77    chunk, input = input[:lim], input[lim:]
78    out.Write(chunk)
79    total_written += len(chunk)
80    update<- total_written*100/total
81  }
82  out.Write(input)
83
84  done <- true
85  return nil
86 }

Figure 4: Listing 2 updates a progress bar while copying a large file in Go with the termui library.

Since GUIs and thus the progress bar are running in an event loop, data must be read and written in a non-blocking, asynchronous fashion, such as with Node.js. When data arrives in this programming paradigm, the code regularly triggers callbacks when more data becomes available. In addition to gobbling up and processing data, these callbacks are handy to update our progress bar, to reflect the amount of data read or written so far.