Processing NETCONF using classic Expect/TCL

Don Libes' Expect, extending the ever-flexible Tool Command Language (TCL), forms one of the original ways of automating I/O interaction with terminal-based UNIX processes.

It has been used for numerous applications, from managing the login process on modem dial-up systems, to automating the interaction with network elements in ISP networks in a programmatic way. While this activity of so-called screen-scraping-reading and parsing output -- meant for humans in a machine-compatible way -- can be limited and subject to future-proofing problems, it still represents a significant capability, and sometimes it can be useful to make use of Expect with NETCONF-based network elements.

In this recipe, we explore using a simplistic Expect skeleton program to make RPC calls to our JUNOS OS devices in order to execute commands and extract data.

Getting ready

To complete this recipe, you should have completed the previous recipe, JUNOS NETCONF- over-SSH setup for your device, particularly with respect to establishing SSH key-pairs.

You should ideally make use of Expect 5.45, or a compatible version on your management host. At the time of writing, this version was available in the built-in software package systems of OpenBSD 6.0 and Ubuntu Linux 16.04. Expect is particularly mature and stable however, so if you can't match the exact version, it's unlikely that you'll run into trouble with the example code that we have here.

Our Expect program, netconf.tcl, will be comprised of three main parts, which are as follows:

• Some initialization routines to read the command-line arguments
• Set up of the NETCONF-over-SSH session
• Interaction with the NETCONF-over-SSH session to make an RPC call, and output the response

How to do it...

The steps for the recipe are as follows:

1. Create the interaction procedure first. To do this, create a TCL procedure that accepts a string argument that will represent the command to run:

     proc cmdrpc { cmd } {
        send -- "<rpc><command format=\"text\">[join $cmd]</command>    
    </rpc>\r\n"
      set output ""
      expect {

            -re {<error-message>([^<]+)</error-message>} {
              send_error "Command RPC for $cmd caused error:    
       $expect_out(1,string)\r\n"
              return
          }

            -re {<(configuration-)?output[^>]*>} {
               expect {
                  -re {^[^<]+} {
                     append output $expect_out(0,string)
                     exp_continue
                 }
                   -re "</(configuration-)?output>" {}
              }
                regsub -all "&lt;" $output "<" output
                regsub -all "&gt;" $output ">" output
                regsub -all "&amp;" $output "&" output
                return $output
             }

            default {
               send_error "Timeout waiting for RPC [join $cmd]\r\n"
               send_error [
                  concat "\t" [
                      regsub -all {[\r\n]+} $expect_out(buffer)      
                          "\r\n\t"
                     ]
                 ]
                  return
              }
           }
        }

2. Read the environment command-line arguments in order to determine a hostname and a command:

      if { [ llength $argv ] != 2 } {
          send_user "Usage: netconf.tcl hostname command\r\n"
           exit 1
      }
      set hostname [lrange $argv 0 0]
      set command [lrange $argv 1 1]

3. Establish a NETCONF-over-SSH session and call the previously defined interaction procedure to send the RPC and extract the results:

      set DELIMITER {]]>]]>}
       if [ spawn -noecho ssh -p 830 -i JUNOS_auto_id_rsa      
       auto@$hostname -s netconf ] {
       expect {
           $DELIMITER {
           set result [ cmdrpc $command ]
               if {$result ne ""} {
                  send_user $result
               }
          }
            default {
            send_error "SSH protocol error (check       
            authorized_keys?)\r\n"
               exit 1
          }
        }
       } {
        send_error "Unable to start SSH client for connection to   
       $hostname\r\n"
        exit 1
       }
       close
       exit

How it works...

First of all, the command-line arguments are analyzed to get a hostname and a command to run. Then we use the spawn command to start up a regular SSH client with the necessary parameters to connect to the hostname. Note that we're using the auto username and the key that we explicitly generated in the previous recipes.

The hard work happens in the interaction procedure, cmdrpc. It's comprised of two nested expect loops. First of all, it open the dialog with the NETCONF host by sending the command RPC along with the textual command that we want to execute. The first expect loop runs, which attempts to determine if the RPC was successful or otherwise. If the successful RPC branch is chosen, a second expect loop runs, which accumulates the lines of output in a variable, ready to return. The second expect loop determines the end of the output by looking for the appropriate XML closing tag. Finally the resulting output is scanned to expand some special XML tokens, as per the JUNOS OS specification, and we print the output for the user to see.

Depending on your familiarity with TCL and Expect, you might have a little bit of trouble following the example code. If so, take heart. TCL can seem a little bit daunting because of the quoting and escaping rules that are implemented using braces. In the table, there's a handy phrase - book to compare an example to the typical UNIX shell, which might be a little more widely understood.

TCL	Shell	Description
"text with $variable"	"text with $variable"	The double quotes group together the textual output along with white space, but expand any variables preceded with dollar signs ($)
{ literal string block }	{ literal string block }	Literal string block, including white space and not performing variable expansion.
[ command ]	$(command), or `command`	Sub-shell or command invocation expansion. Used to substitute the evaluation of an expression or the result of a command or procedure call