Chapter 2. Have a Good Command

In this chapter, we will cover the following recipes:

Concatenating with cat
Recording and playing back terminal sessions
Finding files and file listing
Playing with xargs
Translating with tr
Checksum and verification
Cryptographic tools and hashes
Sorting unique and duplicate lines
Temporary file naming and random numbers
Splitting files and data
Slicing filenames based on extensions
Renaming and moving files in bulk
Spell–checking and dictionary manipulation
Automating interactive input
Making commands quicker by running parallel processes
Examining a directory, files and subdirectories in it

Introduction

Unix-like systems have the best command-line tools. Each command performs a simple function to make our work easier. These simple functions can be combined with other commands to solve complex problems. Combining simple commands is an art; you will get better at it as you practice and gain experience. This chapter introduces some of the most interesting and useful commands, including grep, awk, sed, and find.

Concatenating with cat

The cat command displays or concatenates the contents of a file, but cat is capable of more. For example, cat can combine standard input data with data from a file. One way of combining the stdin data with file data is to redirect stdin to a file and then append two files. The cat command can do this in a single invocation. The next recipes show basic and advanced usages of cat.

How to do it...

The cat command is a simple and frequently used command and it stands for conCATenate.

The general syntax of cat for reading contents is as follows:

$ cat file1 file2 file3 ...

This command concatenates data from the files specified as command-line arguments and sends that data to stdout.

To print contents of a single file, execute the following command:

$ cat file.txt
        This is a line inside file.txt
        This is the second line inside file.txt

To print contents of more than one file, execute the following command:

$ cat one.txt two.txt 
        This line is from one.txt
...

Recording and playing back terminal sessions

Recording a screen session as a video is useful, but a video is an overkill for debugging terminal sessions or providing a shell tutorial.

The shell provides another option. The script command records your keystrokes and the timing of keystrokes as you type, and saves your input and the resulting output in a pair of files. The scriptreplay command will replay the session.

Getting ready

The script and scriptreplay commands are available in most GNU/Linux distributions. You can create tutorials of command-line hacks and tricks by recording the terminal sessions. You can also share the recorded files for others to playback and see how to perform a particular task with the command line. You can even invoke other interpreters and record the keystrokes sent to that interpreter. You cannot record vi, emacs, or other applications that map characters to particular locations on the screen.

How to do it...

Start recording the terminal session with the following...

Finding files and file listing

The find command is one of the great utilities in the Unix/Linux command-line toolbox. It is useful both at the command line and in shell scripts. Like cat and ls, find has many features, and most people do not use it to its fullest. This recipe deals with some common ways to utilize find to locate files.

Getting ready

The find command uses the following strategy: find descends through a hierarchy of files, matches files that meet the specified criteria, and performs some actions. The default action is to print the names of files and folders, which can be specified with the -print option.

How to do it...

To list all the files and folders descending from a given directory, use this syntax:

$ find base_path

The base_path can be any location from which find should start descending (for example, /home/slynux/).

Here's an example of this command:

$ find . -print
.history
Downloads
Downloads/tcl.fossil
Downloads/chapter2.doc
…

The . specifies the current directory and ....

Playing with xargs

Unix commands accept data either from the standard input (stdin) or as command line arguments. Previous examples have shown how to pass data from one application's standard output to another's standard input with a pipe.

We can invoke applications that accept command-line arguments in other ways. The simplest is to use the back-tic symbol to run a command and use its output as a command line:

$ gcc `find '*.c'`

This solution works fine in many situations, but if there are a lot of files to be processed, you'll see the dreaded Argument list too long error message. The xargs program solves this problem.

The xargs command reads a list of arguments from stdin and executes a command using these arguments in the command line. The xargs command can also convert any one-line or multiple-line text inputs into other formats, such as multiple lines (specified number of columns) or a single line, and vice versa.

Getting ready

The xargs command should be the first command to appear after...

Translating with tr

The tr command is a versatile tool in the Unix command–warrior's kit. It is used to craft elegant one-liner commands. It performs substitution of characters, deletes selected characters, and can squeeze repeated characters from the standard input. Tr is short for translate, since it translates a set of characters to another set. In this recipe, we will see how to use tr to perform basic translation between sets.

Getting ready

The tr command accepts input through stdin (standard input) and cannot accept input through command-line arguments. It has this invocation format:

tr [options] set1 set2

Input characters from stdin are mapped from the first character in set1 to the first character in set2, and so on and the output is written to stdout (standard output). set1 and set2 are character classes or a set of characters. If the length of sets is unequal, set2 is extended to the length of set1 by repeating the last character; otherwise if the length of set2 is greater than that...

Checksum and verification

Checksum programs are used to generate a relatively small unique key from files. We can recalculate the key to confirm that a file has not changed. Files may be modified deliberately (adding a new user changes the password file), accidentally (a data read error from a CD-ROM drive), or maliciously (a virus is inserted). Checksums let us verify that a file contains the data we expect it to.

Checksums are used by backup applications to check whether a file has been modified and needs to be backed up.

Most software distributions also have a checksum file available. Even robust protocols such as TCP can allow a file to be modified in transit. Hence, we need to know whether the received file is the original one or not by applying some kind of test.

By comparing the checksum of the file we downloaded with the checksum calculated by the distributer, we can verify that the received file is correct. If the checksum calculated from the original file at the source location matches...

Cryptographic tools and hashes

Encryption techniques are used to protect data from unauthorized access. Unlike the checksum algorithms we just discussed, encryption programs can reconstruct the original data with no loss. There are many algorithms available and we will discuss those most commonly used in the Linux/Unix world.

How to do it...

Let's see how to use tools such as crypt, gpg, and base64:

The crypt command is not commonly installed on Linux systems. It's a simple and relatively insecure cryptographic utility that accepts input from stdin, requests a passphrase, and sends encrypted output to stdout:

$ crypt <input_file >output_file
        Enter passphrase:

We can provide a passphrase on the command line:

$ crypt PASSPHRASE <input_file >encrypted_file

In order to decrypt the file, use this:

$ crypt PASSPHRASE -d <encrypted_file >output_file

gpg (GNU privacy guard) is a widely used tool for protecting files to ensure that data is not read until it reaches its intended destination...

Sorting unique and duplicate lines

Sorting text files is a common task. The sort command sorts text files and stdin. It can be coupled with other commands to produce the required output. uniq is often used with sort to extract unique (or duplicate) lines. The following recipes illustrate some sort and uniq use cases.

Getting ready

The sort and uniq commands accept input as filenames or from stdin (standard input) and output the result by writing to stdout.

How to do it...

We can sort a set of files (for example, file1.txt and file2.txt), like this:

$ sort file1.txt file2.txt > sorted.txt

Alternatively, use this:

$ sort file1.txt file2.txt -o sorted.txt

For a numerical sort, we use this:

$ sort -n file.txt

To sort in the reverse order, we use the following command:

$ sort -r file.txt

To sort by months (in the order Jan, Feb, March,...), use this:

$ sort -M months.txt

To merge two already sorted files, use this command:

$ sort -m sorted1 sorted2

To find the unique lines from a sorted file, use this:

$...

Temporary file naming and random numbers

Shell scripts often need to store temporary data. The most suitable location to do this is /tmp (which will be cleaned out by the system on reboot). There are two methods to generate standard filenames for temporary data.

How to do it...

The mktemp command will create a unique temporary file or folder name:

Create a temporary file:

$ filename=`mktemp`
        $ echo $filename
        /tmp/tmp.8xvhkjF5fH

This creates a temporary file, stores the name in filename, and then displays the name.

Create a temporary directory:

$ dirname=`mktemp -d`
        $ echo $dirname
        tmp.NI8xzW7VRX

This creates a temporary directory, stores the name in filename, and displays the name.

To generate a filename without creating a file or directory, use this:

$ tmpfile=`mktemp -u`
                $ echo $tmpfile
                /tmp/tmp.RsGmilRpcT

Here, the filename is stored in $tmpfile, but the file won't be created.

To create the temporary filename based on a template, use...

Splitting files and data

Splitting a large file into smaller pieces is sometimes necessary. Long ago, we had to split files to transport large datasets on floppy disks. Today, we split files for readability, for generating logs, or for working around size-restrictions on e-mail attachments. These recipes will demonstrate ways of splitting files in different chunks.

How to do it...

The split command was created to split files. It accepts a filename as an argument and creates a set of smaller files in which the first part of the original file is in the alphabetically first new file, the next set in the alphabetically next file, and so on.

For example, a 100 KB file can be divided into smaller files of 10k each by specifying the split size. The split command supports M for MB, G for GB, c for byte, and w for word.

$ split -b 10k data.file
$ ls
data.file  xaa  xab  xac  xad  xae  xaf  xag  xah  xai  xaj

The preceding code will split data.file into ten files of 10k each. The new files are named xab...

Slicing filenames based on extensions

Many shell scripts perform actions that involve modifying filenames. They may need to rename the files and preserve the extension, or convert files from one format to another and change the extension, while preserving the name, extracting a portion of the filename, and so on.

The shell has built-in features for manipulating filenames.

How to do it...

The % operator will extract the name from name.extension. This example extracts sample from sample.jpg:

file_jpg="sample.jpg" 
name=${file_jpg%.*} 
echo File name is: $name

The output is this:

File name is: sample

The # operator will extract the extension:

Extract .jpg from the filename stored in the file_jpg variable:

extension=${file_jpg#*.} 
echo Extension is: jpg

The output is as follows:

Extension is: jpg

How it works...

To extract the name from the filename formatted as name.extension, we use the % operator.

${VAR%.*} is interpreted as follows:

Remove the string match from $VAR for the wildcard pattern that appears...

Renaming and moving files in bulk

We frequently need to move and perhaps rename a set of files. System housekeeping often requires moving files with a common prefix or file type to a new folder. Images downloaded from a camera may need to be renamed and sorted. Music, video, and e-mail files all need to be reorganized eventually.

There are custom applications for many of these operations, but we can write our own custom scripts to do it our way.

Let's see how to write scripts to perform these kinds of operation.

Getting ready

The rename command changes filenames using Perl regular expressions. By combining the find, rename, and mv commands, we can perform a lot of things.

How to do it...

The following script uses find to locate PNG and JPEG files, then uses the ## operator and mv to rename them as image-1.EXT, image-2.EXT, and so on. This changes the file's name, but not its extension:

#!/bin/bash 
#Filename: rename.sh 
#Desc: Rename jpg and png files 

count=1; 
for img in `find . -iname '*.png...

Spell–checking and dictionary manipulation

Most Linux distributions include a dictionary file. However, very few people are aware of this, thus spelling errors abound. The aspell command-line utility is a spell checker. Let's go through a few scripts that make use of the dictionary file and the spell checker.

How to do it...

The /usr/share/dict/ directory contains one or perhaps more dictionary files, which are text files with a list of words. We can use this list to check whether a word is a dictionary word or not:

$ ls /usr/share/dict/ 
american-english  british-english

To check whether the given word is a dictionary word, use the following script:

#!/bin/bash 
#Filename: checkword.sh 
word=$1 
grep "^$1$" /usr/share/dict/british-english -q  
if [ $? -eq 0 ]; then 
  echo $word is a dictionary word; 
else 
  echo $word is not a dictionary word; 
fi

The usage is as follows:

$ ./checkword.sh ful 
ful is not a dictionary word 

$ ./checkword.sh fool 
fool is a dictionary word

How it works...

In grep...

Automating interactive input

We looked at commands that accept arguments on the command line. Linux also supports many interactive applications ranging from passwd to ssh.

We can create our own interactive shell scripts. It's easier for casual users to interact with a set of prompts rather than remember command line flags and the proper order. For instance, a script to back up a user's work, but not to back up and lock files, might look like this:

$ backupWork.sh

What folder should be backed up? notes
What type of files should be backed up? .docx

Automating interactive applications can save you time when you need to rerun the same application and frustration while you're developing one.

Getting ready

The first step to automating a task is to run it and note what you do. The script command discussed earlier may be of use.

How to do it...

Examine the sequence of interactive inputs. From the previous code, we can formulate the steps of the sequence like this:

notes[Return]docx[Return]

In addition to...

Making commands quicker by running parallel processes

Computing power constantly increases not only because processors have higher clock cycles but also because they have multiple cores. This means that in a single hardware processor there are multiple logical processors. It's like having several computers, instead of just one.

However, multiple cores are useless unless the software makes use of them. For example, a program that does huge calculations may only run on one core while the others will sit idle. The software has to be aware and take advantage of the multiple cores if we want it to be faster.

In this recipe, we will see how we can make our commands run faster.

How to do it...

Let's take an example of the md5sum command we discussed in the previous recipes. This command performs complex computations, making it CPU-intensive. If we have more than one file that we want to generate a checksum for, we can run multiple instances of md5sum using a script like this:

#/bin/bash 
#filename:...

Examining a directory, files and subdirectories in it

One of the commonest problems we deal with is finding misplaced files and sorting out mangled file hierarchies. This section will discuss tricks for examining a portion of the filesystem and presenting the contents.

Getting ready

The find command and loops we discussed give us tools to examine and report details in a directory and its contents.

How to do it...

The next recipes show two ways to examine a directory. First we'll display the hierarchy as a tree, then we'll see how to generate a summary of files and folders under a directory.

Generating a tree view of a directory.

Sometimes it's easier to visualize a file system if it's presented graphically.

The next recipe pulls together several of the tools we discussed. It uses the find command to generate a list of all the files and sub-folders under the current folder.

The -exec option creates a subshell which uses echo to send the filenames to the tr command's stdin. There are two tr commands...