Seeing More Progress with dd and pv

By Paulus, 26 July, 2018

In cloning my PS3 and PS4 hard drives using the dd command, and using the status=progress I was only able to see how much has been transferred, how long the command has been running, and the speed. The only thing that was missing was a estimate of how long it would take to complete. Additionally, a progress bar would have been nice. While waiting for the process to complete I looked for a tool that would provide what I was missing. The only tool that I found that is still maintained is pv (Pipe Viewer).

The real nice thing about pv is that it's really simple to use, with almost everything turned on by default.

Running

pv /dev/sda > /dev/null
661MiB 0:00:11 [33.9MiB/s] [>                                               ]  0% ETA 0:38:32

Will print an ASCII progress bar, timer, how long it's going to take, current transfer rate, and the amount of data transferred. Using -a will display the average transfer rate, and -T will show the the percentage of the transfer buffer in use.

In addition to what is displayed by default, the amount of the transfer buffer being used (displayed in the curly brackets) and in the average transfer speed (the second set of square brackets) are added.

 /dev/null
732MiB {100%} 0:00:22 [33.8MiB/s] [33.3MiB/s] [>                            ]  0% ETA 0:37:49

To use a different format use -F followed by the format string:

 /dev/null
[35.1MiB/s] [33.8MiB/s] {100%} 0:01:09 ETA 0:36:30 ETA 01:14:52 [>          ]  3%   
From the man page
%p Progress bar. Expands to fill the remaining space. Should only be specified once. Equivalent to -p.
%t Elapsed time. Equivalent to -t.
%e ETA as time remaining. Equivalent to -e.
%I ETA as local time of completion. Equivalent to -I.
%r Current data transfer rate. Equivalent to -r.
%a Average data transfer rate. Equivalent to -a.
%b Bytes transferred so far (or lines if -l was specified). Equivalent to -b.
%T Percentage of the transfer buffer in use. Equivalent to -T. Shows "{----}" if the transfer is being done with splice(2), since splicing to or from pipes does not use the buffer.
%nA Show the last n bytes written (e.g. %16A for the last 16 bytes). Shows only dots if the transfer is being done with splice(2), since splicing to or from pipes does not use the buffer.
%N Name prefix given by -N. Padded to 9 characters with spaces, and suffixed with :.
%% A single %.

The format string equivalent of turning on all display switches is `%N %b %T %t %r %a %p %e'.

pv < /dev/zero > /dev/sdn
pv /dev/urandom > /dev/sdn

I've seen a lot of examples like the the two below online, where pv and dd are used together, but that's not necessary as you can use pv on its own.

dd if=/dev/sdn | pv -S 76319M | dd of=/dev/null bs=64k
pv -tpreb /dev/sdn | dd of=/dev/null bs=64k

The only time that it would make sense to use the pv command between two other dd commands would be if you wanted to stop at a certain point. Although, you could specify how much data you want to write using the bs and count arguments, pv makes it a bit easier and more flexible.

To use pv with the dialog

(pv -n /dev/sdn > /dev/null) 2>&1 | dialog --gauge "Looking for bad sectors on /dev/sdn..." 10 70 0
(tar cf - . | pv -n -s $(du -sb . | awk '{print $1}') | gzip -9 > out.tgz) 2>&1 | dialog --gauge 'Progress' 7 70