Linux Shell Scripting Cookbook: Do amazing things with the shell and automate tedious tasks , Third Edition

Unix-like systems have the best command-line tools. Each command performs a simple function to make our work easier. These simple functions can be combined with other commands to solve complex problems. Combining simple commands is an art; you will get better at it as you practice and gain experience. This chapter introduces some of the most interesting and useful commands, including grep, awk, sed, and find.

The cat command displays or concatenates the contents of a file, but cat is capable of more. For example, cat can combine standard input data with data from a file. One way of combining the stdin data with file data is to redirect stdin to a file and then append two files. The cat command can do this in a single invocation. The next recipes show basic and advanced usages of cat.

$ cat file1 file2 file3 ...

$ cat file.txt
        This is a line inside file.txt
        This is the second line inside file.txt

$ cat one.txt two.txt 
        This line is from one.txt
...

Recording a screen session as a video is useful, but a video is an overkill for debugging terminal sessions or providing a shell tutorial.

The shell provides another option. The script command records your keystrokes and the timing of keystrokes as you type, and saves your input and the resulting output in a pair of files. The scriptreplay command will replay the session.

The find command is one of the great utilities in the Unix/Linux command-line toolbox. It is useful both at the command line and in shell scripts. Like cat and ls, find has many features, and most people do not use it to its fullest. This recipe deals with some common ways to utilize find to locate files.

$ find base_path

$ find . -print
.history
Downloads
Downloads/tcl.fossil
Downloads/chapter2.doc
…

Unix commands accept data either from the standard input (stdin) or as command line arguments. Previous examples have shown how to pass data from one application's standard output to another's standard input with a pipe.

We can invoke applications that accept command-line arguments in other ways. The simplest is to use the back-tic symbol to run a command and use its output as a command line:

$ gcc `find '*.c'`

This solution works fine in many situations, but if there are a lot of files to be processed, you'll see the dreaded Argument list too long error message. The xargs program solves this problem.

The xargs command reads a list of arguments from stdin and executes a command using these arguments in the command line. The xargs command can also convert any one-line or multiple-line text inputs into other formats, such as multiple lines (specified number of columns) or a single line, and vice versa.

The tr command is a versatile tool in the Unix command–warrior's kit. It is used to craft elegant one-liner commands. It performs substitution of characters, deletes selected characters, and can squeeze repeated characters from the standard input. Tr is short for translate, since it translates a set of characters to another set. In this recipe, we will see how to use tr to perform basic translation between sets.

tr [options] set1 set2

Checksum programs are used to generate a relatively small unique key from files. We can recalculate the key to confirm that a file has not changed. Files may be modified deliberately (adding a new user changes the password file), accidentally (a data read error from a CD-ROM drive), or maliciously (a virus is inserted). Checksums let us verify that a file contains the data we expect it to.

Checksums are used by backup applications to check whether a file has been modified and needs to be backed up.

Most software distributions also have a checksum file available. Even robust protocols such as TCP can allow a file to be modified in transit. Hence, we need to know whether the received file is the original one or not by applying some kind of test.

By comparing the checksum of the file we downloaded with the checksum calculated by the distributer, we can verify that the received file is correct. If the checksum calculated from the original file at the source location matches...

Encryption techniques are used to protect data from unauthorized access. Unlike the checksum algorithms we just discussed, encryption programs can reconstruct the original data with no loss. There are many algorithms available and we will discuss those most commonly used in the Linux/Unix world.

$ crypt <input_file >output_file
        Enter passphrase:

$ crypt PASSPHRASE <input_file >encrypted_file

$ crypt PASSPHRASE -d <encrypted_file >output_file

Sorting text files is a common task. The sort command sorts text files and stdin. It can be coupled with other commands to produce the required output. uniq is often used with sort to extract unique (or duplicate) lines. The following recipes illustrate some sort and uniq use cases.

$ sort file1.txt file2.txt > sorted.txt

$ sort file1.txt file2.txt -o sorted.txt

$ sort -n file.txt

$ sort -r file.txt

$ sort -M months.txt

$ sort -m sorted1 sorted2

$...

Shell scripts often need to store temporary data. The most suitable location to do this is /tmp (which will be cleaned out by the system on reboot). There are two methods to generate standard filenames for temporary data.

$ filename=`mktemp`
        $ echo $filename
        /tmp/tmp.8xvhkjF5fH

$ dirname=`mktemp -d`
        $ echo $dirname
        tmp.NI8xzW7VRX

$ tmpfile=`mktemp -u`
                $ echo $tmpfile
                /tmp/tmp.RsGmilRpcT

Splitting a large file into smaller pieces is sometimes necessary. Long ago, we had to split files to transport large datasets on floppy disks. Today, we split files for readability, for generating logs, or for working around size-restrictions on e-mail attachments. These recipes will demonstrate ways of splitting files in different chunks.

$ split -b 10k data.file
$ ls
data.file  xaa  xab  xac  xad  xae  xaf  xag  xah  xai  xaj

Many shell scripts perform actions that involve modifying filenames. They may need to rename the files and preserve the extension, or convert files from one format to another and change the extension, while preserving the name, extracting a portion of the filename, and so on.

The shell has built-in features for manipulating filenames.

file_jpg="sample.jpg" 
name=${file_jpg%.*} 
echo File name is: $name 

File name is: sample

extension=${file_jpg#*.} 
echo Extension is: jpg 

Extension is: jpg

We frequently need to move and perhaps rename a set of files. System housekeeping often requires moving files with a common prefix or file type to a new folder. Images downloaded from a camera may need to be renamed and sorted. Music, video, and e-mail files all need to be reorganized eventually.

There are custom applications for many of these operations, but we can write our own custom scripts to do it our way.

Let's see how to write scripts to perform these kinds of operation.

#!/bin/bash 
#Filename: rename.sh 
#Desc: Rename jpg and png files 

count=1; 
for img in `find . -iname '*.png...

Most Linux distributions include a dictionary file. However, very few people are aware of this, thus spelling errors abound. The aspell command-line utility is a spell checker. Let's go through a few scripts that make use of the dictionary file and the spell checker.

$ ls /usr/share/dict/ 
american-english  british-english

#!/bin/bash 
#Filename: checkword.sh 
word=$1 
grep "^$1$" /usr/share/dict/british-english -q  
if [ $? -eq 0 ]; then 
  echo $word is a dictionary word; 
else 
  echo $word is not a dictionary word; 
fi 

$ ./checkword.sh ful 
ful is not a dictionary word 

$ ./checkword.sh fool 
fool is a dictionary word

We looked at commands that accept arguments on the command line. Linux also supports many interactive applications ranging from passwd to ssh.

We can create our own interactive shell scripts. It's easier for casual users to interact with a set of prompts rather than remember command line flags and the proper order. For instance, a script to back up a user's work, but not to back up and lock files, might look like this:

$ backupWork.sh

Automating interactive applications can save you time when you need to rerun the same application and frustration while you're developing one.

notes[Return]docx[Return] 

Computing power constantly increases not only because processors have higher clock cycles but also because they have multiple cores. This means that in a single hardware processor there are multiple logical processors. It's like having several computers, instead of just one.

However, multiple cores are useless unless the software makes use of them. For example, a program that does huge calculations may only run on one core while the others will sit idle. The software has to be aware and take advantage of the multiple cores if we want it to be faster.

In this recipe, we will see how we can make our commands run faster.

#/bin/bash 
#filename:...

One of the commonest problems we deal with is finding misplaced files and sorting out mangled file hierarchies. This section will discuss tricks for examining a portion of the filesystem and presenting the contents.