Linux Shell Scripting Cookbook: Do amazing things with the shell and automate tedious tasks , Third Edition

Commands are typed and executed in a terminal session. When a terminal is opened, a prompt is displayed. The prompt can be configured in many ways, but frequently resembles this:

username@hostname$

Alternatively, it can also be configured as root@hostname # or simply as $ or #.

The $ character represents regular users and # represents the administrative user root. Root is the most privileged user in a Linux system.

A shell script typically begins with a shebang:

#!/bin/bash

Shebang is a line on which #! is prefixed to the interpreter path. /bin/bash is the interpreter command path for Bash. A line starting with a # symbol is treated by the bash interpreter as a comment. Only the first line of a script can have a shebang to define the interpreter to be used to evaluate the script.

A script can be executed in two ways:

        bash myScript.sh

        chmod 755 myScript.sh
        ./myScript.sh.

If a script is run as a command-line argument for bash, the shebang is not required. The shebang facilitates running the script on its own. Executable scripts use the interpreter path that follows the shebang to interpret a script.

Scripts are made executable with the chmod command:

$ chmod a+x sample.sh

This command makes a script executable by all users. The script can be executed as follows:

$ ./sample.sh #./ represents the current directory

Alternatively, the script can be executed like this:

$ /home/path/sample.sh # Full path of the script is used

The kernel will read the first line and see that the shebang is #!/bin/bash. It will identify /bin/bash and execute the script as follows:

$ /bin/bash sample.sh

When an interactive shell starts, it executes a set of commands to initialize settings, such as the prompt text, colors, and so on. These commands are read from a shell script at ~/.bashrc (or ~/.bash_profile for login shells), located in the home directory of the user. The Bash shell maintains a history of commands run by the user in the ~/.bash_history file.

A comment starts with # and proceeds up to the end of the line. The comment lines are most often used to describe the code, or to disable execution of a line of code during debugging:

# sample.sh - echoes "hello world"
echo "hello world"

Now let's move on to the basic recipes in this chapter.

The echo command is the simplest command for printing in the terminal.

By default, echo adds a newline at the end of every echo invocation:

$ echo "Welcome to Bash"
Welcome to Bash

Simply, using double-quoted text with the echo command prints the text in the terminal. Similarly, text without double quotes also gives the same output:

$ echo Welcome to Bash
Welcome to Bash

Another way to do the same task is with single quotes:

$ echo 'text in quotes'

These methods appear similar, but each has a specific purpose and side effects. Double quotes allow the shell to interpret special characters within the string. Single quotes disable this interpretation.

Consider the following command:

$ echo "cannot include exclamation - ! within double quotes"

This returns the following output:

bash: !: event not found error

If you need to print special characters such as !, you must either not use any quotes, use single quotes, or escape the special characters with a backslash (\):

$ echo Hello world !

Alternatively, use this:

$ echo 'Hello world !'

Alternatively, it can be used like this:

$ echo "Hello World\!" #Escape character \ prefixed.

When using echo without quotes, we cannot use a semicolon, as a semicolon is the delimiter between commands in the Bash shell:

echo hello; hello 

From the preceding line, Bash takes echo hello as one command and the second hello as the second command.

Variable substitution, which is discussed in the next recipe, will not work within single quotes.

Another command for printing in the terminal is printf. It uses the same arguments as the C library printf function. Consider this example:

$ printf "Hello world"

The printf command takes quoted text or arguments delimited by spaces. It supports formatted strings. The format string specifies string width, left or right alignment, and so on. By default, printf does not append a newline. We have to specify a newline when required, as shown in the following script:

#!/bin/bash
#Filename: printf.sh

printf  "%-5s %-10s %-4s\n" No Name  Mark
printf  "%-5s %-10s %-4.2f\n" 1 Sarath 80.3456
printf  "%-5s %-10s %-4.2f\n" 2 James 90.9989
printf  "%-5s %-10s %-4.2f\n" 3 Jeff 77.564

We will receive the following formatted output:

No    Name       Mark
1     Sarath     80.35
2     James      91.00
3     Jeff       77.56

The %s, %c, %d, and %f characters are format substitution characters, which define how the following argument will be printed. The %-5s string defines a string substitution with left alignment (- represents left alignment) and a 5 character width. If - was not specified, the string would have been aligned to the right. The width specifies the number of characters reserved for the string. For Name, the width reserved is 10. Hence, any name will reside within the 10-character width reserved for it and the rest of the line will be filled with spaces up to 10 characters total.

For floating point numbers, we can pass additional parameters to round off the decimal places.

For the Mark section, we have formatted the string as %-4.2f, where .2 specifies rounding off to two decimal places. Note that for every line of the format string, a newline (\n) is issued.

While using flags for echo and printf, place the flags before any strings in the command, otherwise Bash will consider the flags as another string.

echo -e "1\t2\t3"
1  2  3

echo -e "\e[1;31m This is red text \e[0m"

echo -e "\e[1;42m Green Background \e[0m"

Variables are named as a sequence of letters, numbers, and underscores with no whitespace. Common conventions are to use UPPER_CASE for environment variables and camelCase or lower_case for variables used within a script.

All applications and scripts can access the environment variables. To view all the environment variables defined in your current shell, issue the env or printenv command:

$> env 
PWD=/home/clif/ShellCookBook 
HOME=/home/clif 
SHELL=/bin/bash 
# ... And many more lines

To view the environment of other processes, use the following command:

cat /proc/$PID/environ

Set PID with a process ID of the process (PID is an integer value).

Assume an application called gedit is running. We obtain the process ID of gedit with the pgrep command:

$ pgrep gedit
12501

We view the environment variables associated with the process by executing the following command:

$ cat /proc/12501/environ
GDM_KEYBOARD_LAYOUT=usGNOME_KEYRING_PID=1560USER=slynuxHOME=/home/slynux

To make a human-friendly report, pipe the output of the cat command to tr, to substitute the \0 character with \n:

$ cat /proc/12501/environ  | tr '\0' '\n'

Assign a value to a variable with the equal sign operator:

varName=value

The name of the variable is varName and value is the value to be assigned to it. If value does not contain any space character (such as space), it need not be enclosed in quotes, otherwise it must be enclosed in single or double quotes.

Access the contents of a variable by prefixing the variable name with a dollar sign ($).

var="value" #Assign "value" to var
echo $var

You may also use it like this:

echo ${var}

This output will be displayed:

value

Variable values within double quotes can be used with printf, echo, and other shell commands:

#!/bin/bash
#Filename :variables.sh
fruit=apple
count=5
echo "We have $count ${fruit}(s)"

The output will be as follows:

We have 5 apple(s)

Because the shell uses a space to delimit words, we need to add curly braces to let the shell know that the variable name is fruit, not fruit(s).

Environment variables are inherited from the parent processes. For example, HTTP_PROXY is an environment variable that defines which proxy server to use for an Internet connection.

Usually, it is set as follows:

HTTP_PROXY=192.168.1.23:3128
export HTTP_PROXY

The export command declares one or more variables that will be inherited by child tasks. After variables are exported, any application executed from the current shell script, receives this variable. There are many standard environment variables created and used by the shell, and we can export our own variables.

For example, the PATH variable lists the folders, which the shell will search for an application. A typical PATH variable will contain the following:

$ echo $PATH 
/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Directory paths are delimited by the : character. Usually, $PATH is defined in /etc/environment, /etc/profile or ~/.bashrc.

To add a new path to the PATH environment, use the following command:

export PATH="$PATH:/home/user/bin"

Alternatively, use these commands:

$ PATH="$PATH:/home/user/bin" 
$ export PATH 
$ echo $PATH 
/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/user/bin

Here we have added /home/user/bin to PATH.

Some of the well-known environment variables are HOME, PWD, USER, UID, and SHELL.

The shell has many more built-in features. Here are a few more:

length=${#var}

$ var=12345678901234567890$
echo ${#var}
20

echo $SHELL

echo $0

$ echo $SHELL
/bin/bash

$ echo $0
/bin/bash

If [ $UID -ne 0 ]; then 
  echo Non root user. Please run as root. 
else 
  echo Root user 
fi

if test $UID -ne 0:1 
  then 
    echo Non root user. Please run as root 
  else 
    echo Root User 
fi

        $ cat ~/.bashrc | grep PS1 
        PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '

        slynux@localhost: ~$ PS1="PROMPT> " # Prompt string changed 
        PROMPT> Type commands here.

This example shows how to add new paths to the beginning of an environment variable. The first example shows how to do this with what's been covered so far, the second demonstrates creating a function to simplify modifying the variable. Functions are covered later in this chapter.

export PATH=/opt/myapp/bin:$PATH 
export LD_LIBRARY_PATH=/opt/myapp/lib;$LD_LIBRARY_PATH

The PATH and LD_LIBRARY_PATH variables should now look something like this:

PATH=/opt/myapp/bin:/usr/bin:/bin 
LD_LIBRARY_PATH=/opt/myapp/lib:/usr/lib;/lib

We can make adding a new path easier by defining a prepend function in the .bashrc file.

prepend() { [ -d "$2" ] && eval $1=\"$2':'\$$1\" && export $1; }

This can be used in the following way:

prepend PATH /opt/myapp/bin 
prepend LD_LIBRARY_PATH /opt/myapp/lib

The prepend() function first confirms that the directory specified by the second parameter to the function exists. If it does, the eval expression sets the variable, with the name in the first parameter equal to the second parameter string, followed by : (the path separator), and then the original value for the variable.

If the variable is empty when we try to prepend, there will be a trailing : at the end. To fix this, modify the function to this:

prepend() { [ -d "$2" ] && eval $1=\"$2\$\{$1:+':'\$$1\}\" && export $1 ; }

        #!/bin/bash
        no1=4;
        no2=5;

        let result=no1+no2
        echo $result

                  Other uses of let command are as follows:

                $ let no1++

                $ let no1--

                let no+=6
                let no-=6

                These are equal to let no=no+6 and let no=no-6, respectively.

              The [] operator is used in the same way as the let command:

                    result=$[ no1 + no2 ]

              Using the $ prefix inside the [] operator is legal; consider this example:

                    result=$[ $no1 + 5 ]

      The (( )) operator can also be used. The prefix variable names                        with a $ within the (( )) operator:

                    result=$(( no1 + 50 ))

             The expr expression can be used for basic operations:

                    result=`expr 3 + 4`
                    result=$(expr $no1 + 5)

      The preceding methods do not support floating point numbers,      and operate on integers only.

        echo "4 * 0.56" | bc
        2.24
        no=54;
        result=`echo "$no * 1.5" | bc`
        echo $result
        81.0

The bc application accepts prefixes to control the operation. These are separated from each other with a semicolon.

                echo "scale=2;22/7" | bc
                3.14

                #!/bin/bash
                Desc: Number conversion
                no=100
                echo "obase=2;$no" | bc
                1100100
                no=1100100
                echo "obase=10;ibase=2;$no" | bc
                100

                echo "sqrt(100)" | bc #Square root
                echo "10^10" | bc #Square

Shell scripts frequently use standard input (stdin), standard output (stdout), and standard error (stderr). A script can redirect output to a file with the greater-than symbol. Text generated by a command may be normal output or an error message. By default, both normal output (stdout) and error messages (stderr) are sent to the display. The two streams can be separated by specifying a specific descriptor for each stream.

File descriptors are integers associated with an opened file or data stream. File descriptors 0, 1, and 2 are reserved, as given here:

        $ echo "This is a sample text 1" > temp.txt

This stores the echoed text in temp.txt. If temp.txt already exists, the single greater-than sign will delete any previous contents.

        $ echo "This is sample text 2" >> temp.txt

        $ cat temp.txt
        This is sample text 1
        This is sample text 2

The next recipes demonstrate redirecting stderr. A message is printed to the stderr stream when a command generates an error message. Consider the following example:

$ ls +
ls: cannot access +: No such file or directory

Here + is an invalid argument and hence an error is returned.

The following command prints the stderr text to the screen rather than to a file (and because there is no stdout output, out.txt will be empty):

$ ls + > out.txt
ls: cannot access +: No such file or directory 

In the following command, we redirect stderr to out.txt with 2> (two greater-than):

$ ls + 2> out.txt # works

You can redirect stderr to one file and stdout to another file.

$ cmd 2>stderr.txt 1>stdout.txt

It is also possible to redirect stderr and stdout to a single file by converting stderr to stdout using this preferred method:

$ cmd 2>&1 allOutput.txt

This can be done even using an alternate approach:

$ cmd &> output.txt 

If you don't want to see or save any error messages, you can redirect the stderr output to /dev/null, which removes it completely. For example, consider that we have three files a1, a2, and a3. However, a1 does not have the read-write-execute permission for the user. To print the contents of all files starting with the letter a, we use the cat command. Set up the test files as follows:

$ echo A1 > a1
$ echo A2 > a2
$ echo A3 > a3
$ chmod 000 a1  #Deny all permissions

Displaying the contents of the files using wildcards (a*), will generate an error message for the a1 file because that file does not have the proper read permission:

$ cat a*
cat: a1: Permission denied
A2
A3

Here, cat: a1: Permission denied belongs to the stderr data. We can redirect the stderr data into a file, while sending stdout to the terminal.

$ cat a* 2> err.txt #stderr is redirected to err.txt
A2
A3

$ cat err.txt
cat: a1: Permission denied

Some commands generate output that we want to process and also save for future reference or other processing. The stdout stream is a single stream that we can redirect to a file or pipe to another program. You might think there is no way for us to have our cake and eat it too.

However, there is a way to redirect data to a file, while providing a copy of redirected data as stdin to the next command in a pipe. The tee command reads from stdin and redirects the input data to stdout and one or more files.

command | tee FILE1 FILE2 | otherCommand

In the following code, the stdin data is received by the tee command. It writes a copy of stdout to the out.txt file and sends another copy as stdin for the next command. The cat -n command puts a line number for each line received from stdin and writes it into stdout:

$ cat a* | tee out.txt | cat -n
cat: a1: Permission denied
     1 A2
     2 A3

Use cat to examine the contents of out.txt:

$ cat out.txt
A2
A3

By default, the tee command overwrites the file. Including the -a option will force it to append the new data.

$ cat a* | tee -a out.txt | cat -n

Commands with arguments follow the format: command FILE1 FILE2 ... or simply command FILE.

To send two copies of the input to stdout, use - for the filename argument:

$ cmd1 | cmd2 | cmd -

Consider this example:

$ echo who is this | tee -
who is this
who is this

Alternately, we can use /dev/stdin as the output filename to use stdin.Similarly, use /dev/stderr for standard error and /dev/stdout for standard output. These are special device files that correspond to stdin, stderr, and stdout.

The redirection operators (> and >>) send output to a file instead of the terminal. The > and >> operators behave slightly differently. Both redirect output to a file, but the single greater-than symbol (>) empties the file and then writes to it, whereas the double greater-than symbol (>>) adds the output to the end of the existing file.

By default, the redirection operates on standard output. To explicitly take a specific file descriptor, you must prefix the descriptor number to the operator.

The > operator is equivalent to 1> and similarly it applies for >> (equivalent to 1>>).

When working with errors, the stderr output is dumped to the /dev/null file. The ./dev/null file is a special device file where any data received by the file is discarded. The null device is often known as a black hole, as all the data that goes into it is lost forever.

Commands that read input from stdin can receive data in multiple ways. It is possible to specify file descriptors of our own, using cat and pipes. Consider this example:

$ cat file | cmd
$ cmd1 | cmd2

$ cmd < file

#!/bin/bash 
cat<<EOF>log.txt
This is a generated file. Do not edit. Changes will be overwritten. 
EOF

$ cat log.txt
This is a generated file. Do not edit. Changes will be overwritten. 

$ exec 3<input.txt # open for reading with descriptor number 3

$ echo this is a test line > input.txt
$ exec 3<input.txt

$ cat<&3
this is a test line

$ exec 4>output.txt # open for writing

$ exec 4>output.txt
$ echo newline >&4
$ cat output.txt
newline

$ exec 5>>input.txt

$ exec 5>>input.txt
$ echo appended line >&5
$ cat input.txt
newline
appended line

To use associate arrays, you must have Bash Version 4 or higher.

Arrays can be defined using different techniques:

        array_var=(test1 test2 test3 test4)
        #Values will be stored in consecutive locations starting 
        from index 0.

Alternately, define an array as a set of index-value pairs:

        array_var[0]="test1"
        array_var[1]="test2"
        array_var[2]="test3"
        array_var[3]="test4"
        array_var[4]="test5"
        array_var[5]="test6"

        echo ${array_var[0]}
        test1
        index=5
        echo ${array_var[$index]}
        test6

        $ echo ${array_var[*]}
        test1 test2 test3 test4 test5 test6

  Alternately, you can use the following command:

        $ echo ${array_var[@]}
        test1 test2 test3 test4 test5 test6

        $ echo ${#array_var[*]}6

Associative arrays have been introduced to Bash from Version 4.0. When the indices are a string (site names, user names, nonsequential numbers, and so on), an associative array is easier to work with than a numerically indexed array.

$ declare -A ass_array

        $ ass_array=([index1]=val1 [index2]=val2)

        $ ass_array[index1]=val1
        $ ass_array'index2]=val2

$ declare -A fruits_value
$ fruits_value=([apple]='100 dollars' [orange]='150 dollars')

$ echo "Apple costs ${fruits_value[apple]}"
Apple costs 100 dollars

$ echo ${!array_var[*]}

$ echo ${!array_var[@]}

$ echo ${!fruits_value[*]}
orange apple

These are the operations you can perform on aliases:

        $ alias new_command='command sequence'

This example creates a shortcut for the apt-get install command:

        $ alias install='sudo apt-get install'

Once the alias is defined, we can type install instead of sudo apt-get install.

        $ echo 'alias cmd="command seq"' >> ~/.bashrc

        alias rm='cp $@ ~/backup && rm $@'

When running as a privileged user, aliases can be a security breach. To avoid compromising your system, you should escape commands.

$ \command

$ aliasalias lc='ls -color=auto'
alias ll='ls -l'
alias vi='vim'

The tput and stty commands are utilities used for terminal manipulations.

Here are some capabilities of the tput command:

        tput cols
        tput lines

        tput longname

        tput cup 100 100

        tput setb n

The value of n can be a value in the range of 0 to 7

        tput setf n

The value of n can be a value in the range of 0 to 7

        tput bold

        tput smul
        tput rmul

        tput ed

        #!/bin/sh
        #Filename: password.sh
        echo -e "Enter password: "
        # disable echo before reading password
        stty -echo
        read password
        # re-enable echo
        stty echo
        echo
        echo Password read.

Dates can be printed in a variety of formats. Internally, dates are stored as an integer number of seconds since 00:00:00 1970-01-01. This is called epoch or Unix time.

The system's date can be set from the command line. The next recipes demonstrate how to read and set dates.

It is possible to read the dates in different formats and also to set the date.

        $ date
        Thu May 20 23:09:04 IST 2010

        $ date +%s
        1290047248

The date command can convert many formatted date strings into the epoch time. This lets you use dates in multiple date formats as input. Usually, you don't need to bother about the date string format you use if you are collecting the date from a system log or any standard application generated output.Convert the date string into epoch:

        $ date --date "Wed mar 15 08:09:16 EDT 2017" +%s
        1489579718

The --date option defines a date string as input. We can use any date formatting options to print the output. The date command can be used to find the day of the week given a date string:

        $ date --date "Jan 20 2001" +%A
        Saturday

The date format strings are listed in the table mentioned in the How it works... section

        $ date "+%d %B %Y"
        20 May 2010

        # date -s "Formatted date string"
        # date -s "21 June 2009 11:01:22"

        #!/bin/bash
        #Filename: time_take.sh
        start=$(date +%s)
        commands;
        statements;
        end=$(date +%s)
        difference=$(( end - start))
        echo Time taken to execute commands is $difference seconds.

The Unix epoch is defined as the number of seconds that have elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds. Epoch time is useful when you need to calculate the difference between two dates or times. Convert the two date strings to epoch and take the difference between the epoch values. This recipe calculates the number of seconds between two dates:

secs1=`date -d "Jan 2 1970" 
secs2=`date -d "Jan 3 1970" 
echo "There are `expr $secs2 - $secs1` seconds between Jan 2 and Jan 3" 
There are 86400 seconds between Jan 2 and Jan 3 

Displaying a time in seconds since midnight of January 1, 1970, is not easily read by humans. The date command supports output in human readable formats.

The following table lists the format options that the date command supports.

Producing time intervals is essential when writing monitoring scripts that execute in a loop. The following examples show how to generate time delays.

#!/bin/bash 
#Filename: sleep.sh 
echo Count: 
tput sc 

# Loop for 40 seconds 
for count in `seq 0 40` 
do 
  tput rc 
  tput ed 
  echo -n $count 
  sleep 1 
done

We can either use Bash's inbuilt debugging tools or write our scripts in such a manner that they become easy to debug; here's how:

        $ bash -x script.sh

Running the script with the -x flag will print each source line with the current status.

        #!/bin/bash 
        #Filename: debug.sh 
        for i in {1..6}; 
        do 
            set -x 
            echo $i 
            set +x 
        done 
        echo "Script executed"

In the preceding script, the debug information for echo $i will only be printed, as debugging is restricted to that section using -x and +x.The script uses the {start..end} construct to iterate from a start to end value, instead of the seq command used in the previous example. This construct is slightly faster than invoking the seq command.

Look at the following example code:

        #!/bin/bash 
        function DEBUG() 
        { 
            [ "$_DEBUG" == "on" ] && $@ || : 
        } 
        for i in {1..10} 
        do 
          DEBUG echo "I is $i" 
        done

Run the preceding script with debugging set to "on":

        $ _DEBUG=on ./script.sh

We prefix DEBUG before every statement where debug information is to be printed. If _DEBUG=on is not passed to the script, debug information will not be printed. In Bash, the command : tells the shell to do nothing.

The -x flag outputs every line of script as it is executed. However, we may require only some portions of the source lines to be observed. Bash uses a set builtin to enable and disable debug printing within the script:

We can also use other convenient ways to debug scripts. We can make use of shebang in a trickier way to debug scripts.

PS4='$LINENO: ' 

sh -x testScript.sh 2> debugout.txt

exec 6> /tmp/debugout.txt 
BASH_XTRACEFD=6

A function is defined with the function command, a function name, open/close parentheses, and a function body enclosed in curly brackets:

        function fname() 
        { 
            statements; 
        }  

Alternatively, it can be defined as:

        fname() 
        { 
            statements; 
        } 

It can even be defined as follows (for simple functions):

        fname() { statement; }

        $ fname ; # executes function

        fname arg1 arg2 ; # passing args

The following is the definition of the function fname. In the fname function, we have included various ways of accessing the function arguments.

        fname() 
        { 
           echo $1, $2; #Accessing arg1 and arg2 
           echo "$@"; # Printing all arguments as list at once 
           echo "$*"; # Similar to $@, but arguments taken as single  
           entity 
           return 0; # Return value 
         }

Arguments passed to scripts can be accessed as $0 (the name of the script):

        $> alias lsg='ls | grep' 
        $> lsg txt 
          file1.txt 
          file2.txt 
          file3.txt

        $> alias wontWork='/sbin/ifconfig | grep' 
        $> wontWork eth0 
        eth0  Link  encap:Ethernet  HWaddr 00:11::22::33::44:55

        $> function getIP() { /sbin/ifconfig $1 | grep 'inet ';  } 
        $> getIP eth0 
        inet addr:192.168.1.2 Bcast:192.168.255.255 Mask:255.255.0.0

Let's explore more tips on Bash functions.

export -f fname
$> function getIP() { /sbin/ifconfig $1 | grep 'inet '; }
$> echo "getIP eth0" >test.sh
$> sh test.sh
  sh: getIP: No such file or directory
$> export -f getIP
$> sh test.sh
  inet addr: 192.168.1.2 Bcast: 192.168.255.255 Mask:255.255.0.0

cmd;
echo $?;

#!/bin/bash 
#Filename: success_test.sh 
# Evaluate the arguments on the command line - ie success_test.sh 'ls | grep txt' 
eval $@ 
if [ $? -eq 0 ]; 
then 
    echo "$CMD executed successfully" 
else 
    echo "$CMD terminated unsuccessfully" 
fi

echo $1 $2 $3

$ cat showArgs.sh
for i in `seq 1 $#`
do
echo $i is $1
shift
done
$ sh showArgs.sh a b c
1 is a
2 is b
3 is c

The input is usually fed into a command through stdin or arguments. The output is sent to stdout or stderr. When we combine multiple commands, we usually supply input via stdin and generate output to stdout.

In this context, the commands are called filters. We connect each filter using pipes, sympolized by the piping operator (|), like this:

$ cmd1 | cmd2 | cmd3 

Here, we combine three commands. The output of cmd1 goes to cmd2, the output of cmd2 goes to cmd3, and the final output (which comes out of cmd3) will be displayed on the monitor, or directed to a file.

Pipes can be used with the subshell method for combining outputs of multiple commands.

        $ ls | cat -n > out.txt

The output of ls (the listing of the current directory) is passed to cat -n, which in turn prepends line numbers to the input received through stdin. The output is redirected to out.txt.

        cmd_output=$(COMMANDS)

This is called the subshell method. Consider this example:

        cmd_output=$(ls | cat -n)
        echo $cmd_output

Another method, called back quotes (some people also refer to it as back tick) can also be used to store the command output:

        cmd_output=`COMMANDS`

Consider this example:

        cmd_output=`ls | cat -n`
        echo $cmd_output

Back quote is different from the single-quote character. It is the character on the ~ button on the keyboard.

There are multiple ways of grouping commands.

        $> pwd 
        / 
        $> (cd /bin; ls) 
        awk bash cat... 
        $> pwd 
        /

$ cat text.txt
1
2
3

$ out=$(cat text.txt)
$ echo $out
1 2 3 # Lost \n spacing in 1,2,3

$ out="$(cat text.txt)"
$ echo $out
1
2
3

You can use various options of the read command to obtain different results, as shown in the following steps:

        read -n number_of_chars variable_name

Consider this example:

        $ read -n 2 var
        $ echo $var

        read -s var

        read -p "Enter input:"  var

        read -t timeout var

Consider the following example:

        $ read -t 2 var
        # Read the string that is typed within 2 seconds into
        variable var.

        read -d delim_char var

 Consider this example:

        $ read -d ":" var
        hello:#var is set to hello

Define a function in the following way:

repeat() 
{ 
  while true 
  do 
    $@ && return 
  done 
}

Alternatively, add this to your shell's rc file for ease of use:

repeat() { while true; do $@ && return; done }

This repeat function has an infinite while loop, which attempts to run the command passed as a parameter (accessed by $@) to the function. It returns if the command was successful, thereby exiting the loop.

We saw a basic way to run commands until they succeed. Let's make things more efficient.

repeat() { while :; do $@ && return; done }

repeat wget -c http://www.example.com/software-0.1.tar.gz

repeat() { while :; do $@ && return; sleep 30; done }

Consider the case of CSV data:

data="name,gender,rollno,location" 
To read each of the item in a variable, we can use IFS. 
oldIFS=$IFS 
IFS=, # IFS is now a , 
for item in $data; 
do 
    echo Item: $item 
done 

IFS=$oldIFS

This generates the following output:

Item: name
Item: gender
Item: rollno
Item: location

The default value of IFS is a white-space (newline, tab, or a space character).

When IFS is set as , the shell interprets the comma as a delimiter character, therefore, the $item variable takes substrings separated by a comma as its value during the iteration.

If IFS is not set as , then it will print the entire data as a single string.

Let's go through another example usage of IFS to parse the /etc/passwd file. In the /etc/passwd file, every line contains items delimited by :. Each line in the file corresponds to an attribute related to a user.

Consider the input: root:x:0:0:root:/root:/bin/bash. The last entry on each line specifies the default shell for the user.

Print users and their default shells using the IFS hack:

#!/bin/bash 
#Desc: Illustration of IFS 
line="root:x:0:0:root:/root:/bin/bash"  
oldIFS=$IFS; 
IFS=":" 
count=0 
for item in $line; 
do 

     [ $count -eq 0 ]  && user=$item; 
     [ $count -eq 6 ]  && shell=$item; 
    let count++ 
done; 
IFS=$oldIFS 
echo $user's shell is $shell;

The output will be as follows:

root's shell is /bin/bash

Loops are very useful in iterating through a sequence of values. Bash provides many types of loops.

        for var in list; 
        do 
            commands; # use $var 
        done 

A list can be a string or a sequence of values.

We can generate sequences with the echo command:

echo {1..50} ;# Generate a list of numbers from 1 to 50.
echo {a..z} {A..Z} ;# List of lower and upper case letters.

We can combine these to concatenate data.In the following code, in each iteration, the variable i will hold a character in the a to z range:

      for i in {a..z}; do actions; done;

        for((i=0;i<10;i++)) 
        { 
           commands; # Use $i 
        }

The while loop continues while a condition is true, the until loop runs until a condition is true:

        while condition 
        do 
            commands; 
        done

For an infinite loop, use true as the condition:

A special loop called until is available with Bash. This executes the loop until the given condition becomes true. Consider this example:

        x=0; 
        until [ $x -eq 9 ]; # [ $x -eq 9 ] is the condition 
        do 
            let x++; echo $x; 
        done

Here are some methods used for comparisons and performing tests:

        if condition; 
        then 
            commands; 
        fi

        if condition;  
        then 
            commands; 
        else if condition; then 
            commands; 
        else 
            commands; 
        fi 

[$var -eq 0 ] or [ $var -eq 0]

Perform mathematical tests on variables and values, like this:

[ $var -eq 0 ]  # It returns true when $var equal to 0. 
[ $var -ne 0 ] # It returns true when $var is not equal to 0

Other important operators include the following:

The -a operator is a logical AND and the -o operator is the logical OR. Multiple test conditions can be combined:

[ $var1 -ne 0 -a $var2 -gt 2 ]  # using and -a 
[ $var1 -ne 0 -o var2 -gt 2 ] # OR -o

Filesystem-related tests are as follows:

Test different filesystem-related attributes using different condition flags

Consider this example:

fpath="/etc/passwd" 
if [ -e $fpath ]; then 
    echo File exists;  
else 
    echo Does not exist;  
fi

String comparisons: When using string comparison, it is best to use double square brackets, since the use of single brackets can sometimes lead to errors

Test if two strings are identical:

Test if two strings are not identical:

Test for an empty string:

It is easier to combine multiple conditions using logical operators such as && and ||, as in the following code:

if [[ -n $str1 ]] && [[ -z $str2 ]] ;
   then
       commands;
   fi

Consider this example:

str1="Not empty " 
str2="" 
if [[ -n $str1 ]] && [[ -z $str2 ]]; 
then 
    echo str1 is nonempty and str2 is empty string. 
fi

This will be the output:

str1 is nonempty and str2 is empty string.

The test command can be used for performing condition checks. This reduces the number of braces used and can make your code more readable. The same test conditions enclosed within [] can be used with the test command.

Consider this example:

if  [ $var -eq 0 ]; then echo "True"; fi 
can be written as 
if  test $var -eq 0 ; then echo "True"; fi

These files are evaluated when a user logs into a shell:

/etc/profile, $HOME/.profile, $HOME/.bash_login, $HOME/.bash_profile /

If a .bash_profile or .bash_login file is present, a .profile file will not be read.

These files will be read by an interactive shell such as a X11 terminal session or using ssh to run a single command like: ssh 192.168.1.1 ls /tmp.

/etc/bash.bashrc $HOME/.bashrc

Run a shell script like this:

$> cat myscript.sh  
#!/bin/bash 
echo "Running"

None of these files will be sourced unless you have defined the BASH_ENV environment variable:

$> export BASH_ENV=~/.bashrc 
$> ./myscript.sh

Use ssh to run a single command, as with the following:

ssh 192.168.1.100 ls /tmp

This will start a bash shell which will evaluate /etc/bash.bashrc and $HOME/.bashrc, but not /etc/profile or .profile.

Invoke a ssh login session, like this:

ssh 192.168.1.100

This creates a new login bash shell, which will evaluate the following:

/etc/profile 
/etc/bash.bashrc 
$HOME/.profile or .bashrc_profile

Use these files to define non-exported items such as aliases desired by all users. Consider this example:

alias l "ls -l"
/etc/bash.bashrc /etc/bashrc

Use these files to hold personal settings. They are useful for setting paths that must be inherited by other bash instances. They might include lines like these:

CLASSPATH=$CLASSPATH:$HOME/MyJavaProject; export CLASSPATH
$HOME/.bash_login $HOME/.bash_profile $HOME/.profile

Use these files to hold your personal values that need to be defined whenever a new shell is created. Define aliases and functions here if you want them available in an X11 terminal session:

$HOME/.bashrc, /etc/bash.bashrc

This file is evaluated when a user logs out of a session:

$HOME/.bash_logout

For example, if the user logs in remotely they should clear the screen when they log out.

$> cat ~/.bash_logout 
# Clear the screen after a remote login/logout. 
clear