Go Standard Library Cookbook: Over 120 specific ways to make full use of the standard library components in Golang

Install and verify the Go installation. The following steps could help:

The following steps cover the solution:

        package main
        import (
          "log"
          "runtime"
        )
        const info = `
          Application %s starting.
          The binary was build by GO: %s`

        func main() {
          log.Printf(info, "Example", runtime.Version())
        }

The runtime package contains a lot of useful functions. To find out the Go runtime version, the Version function could be used. The documentation states that the function returns the hash of the commit, and the date or tag at the time of the binary build.

The Version function, in fact, returns the runtime/internal/sys .The Version constant. The constant itself is located in the $GOROOT/src/runtime/internal/sys/zversion.go file.

This .go file is generated by the go dist tool and the version is resolved by the findgoversion function in the go/src/cmd/dist/build.go file, as explained next.

The $GOROOT/VERSION takes priority. If the file is empty or does not exist, the $GOROOT/VERSION.cache file is used. If the $GOROOT/VERSION.cache is also not found, the tool tries to resolve the version by using the Git information, but in this case, you need to initialize the Git repository for the Go source.

        package main
        import (
          "fmt"
          "os"
        )

        func main() {

          args := os.Args

          // This call will print
          // all command line arguments.
          fmt.Println(args)

          // The first argument, zero item from slice,
          // is the name of the called binary.
          programName := args[0]
          fmt.Printf("The binary name is: %s \n", programName)

          // The rest of the arguments could be obtained
          // by omitting the first argument.
          otherArgs := args[1:]
          fmt.Println(otherArgs)

          for idx, arg := range otherArgs {
            fmt.Printf("Arg %d = %s \n", idx, arg)

          }
        }

The Go standard library offers a few ways to access the arguments of the program call. The most generic way is to access the arguments by the Args variable from the OS package.

This way you can get all the arguments from the command line in a string slice. The advantage of this approach is that the number of arguments is dynamic and this way you can, for example, pass the names of the files to be processed by the program.

The preceding example just echoes all the arguments that are passed to the program. Finally, let's say the binary is called test and the program run is executed by the Terminal command ./test arg1 arg2.

In detail, the os.Args[0] will return ./test. The os.Args[1:] returns the rest of the arguments without the binary name. In the real world, it is better to not rely on the number of arguments passed to the program, but always check the length of the argument array. Otherwise, naturally, if the argument on a given index is not within the range, the program panics.

If the arguments are defined as flags, -flag value, additional logic is needed to assign the value to the flag. In this case, there is a better way to parse these by using the flag package. This approach is part of the next recipe.

        package main
        import (
          "flag"
          "fmt"
          "log"
          "os"
          "strings"
        )

        // Custom type need to implement
        // flag.Value interface to be able to
        // use it in flag.Var function.
        type ArrayValue []string

        func (s *ArrayValue) String() string {
          return fmt.Sprintf("%v", *s)
        }

        func (a *ArrayValue) Set(s string) error {
          *a = strings.Split(s, ",")
          return nil
        }

        func main() {

          // Extracting flag values with methods returning pointers
          retry := flag.Int("retry", -1, "Defines max retry count")

          // Read the flag using the XXXVar function.
          // In this case the variable must be defined
          // prior to the flag.
          var logPrefix string
          flag.StringVar(&logPrefix, "prefix", "", "Logger prefix")

          var arr ArrayValue
          flag.Var(&arr, "array", "Input array to iterate through.")

          // Execute the flag.Parse function, to
          // read the flags to defined variables.
          // Without this call the flag
          // variables remain empty.
          flag.Parse()

          // Sample logic not related to flags
          logger := log.New(os.Stdout, logPrefix, log.Ldate)

          retryCount := 0
          for retryCount < *retry {
            logger.Println("Retrying connection")
            logger.Printf("Sending array %v\n", arr)
            retryCount++
          }
        }

For the flag definition in code, the flag package defines two types of functions.

The first type is the simple name of the flag type such as Int. This function will return the pointer to the integer variable where the value of the parsed flag is.

The XXXVar functions are the second type. These provide the same functionality, but you need to provide the pointer to the variable. The parsed flag value will be stored in the given variable.

The Go library also supports a custom flag type. The custom type must implement the Value interface from the flag package.

As an example, let's say the flag retry defines the retry limit for reconnecting to the endpoint, the prefix flag defines the prefix of each row in a log, and the array is the array flag that will be send as an payload to server. The program call from the Terminal will look like ./util -retry 2 -prefix=example array=1,2.

The important part of the preceding code is the Parse() function which parses the defined flags from Args[1:]. The function must be called after all flags are defined and before the values are accessed.

The preceding code shows how to parse some data types from the command-line flags. Analogously, the other built-in types are parsed. 

The last flag, array, demonstrates the definition of the custom type flag. Note that the ArrayType implements the Value interface from the flag package.

The flag package contains more functions to design the interface with flags. It is worth reading the documentation for FlagSet.

By defining the new FlagSet, the arguments could be parsed by calling the myFlagset.Parse(os.Args[2:]). This way you can have flag subsets based on, for example, the first flag.

        package main

        import (
          "log"
          "os"
        )

        func main() {
          connStr := os.Getenv("DB_CONN")
          log.Printf("Connection string: %s\n", connStr)
        }

        package main

        import (
          "log"
          "os"
        )

        func main() {

          key := "DB_CONN"

          connStr, ex := os.LookupEnv(key)
          if !ex {
            log.Printf("The env variable %s is not set.\n", key)
          }
          fmt.Println(connStr)
        }

        package main
        import (
          "log"
          "os"
        )

        func main() {

          key := "DB_CONN"
          // Set the environmental variable.
          os.Setenv(key, "postgres://as:[email protected]/pg?
                          sslmode=verify-full")
          val := GetEnvDefault(key, "postgres://as:as@localhost/pg?
                                     sslmode=verify-full")
          log.Println("The value is :" + val)

          os.Unsetenv(key)
          val = GetEnvDefault(key, "postgres://as:[email protected]/pg?
                                    sslmode=verify-full")
          log.Println("The default value is :" + val)

        }

        func GetEnvDefault(key, defVal string) string {
          val, ex := os.LookupEnv(key)
          if !ex {
            return defVal
          }
          return val
        }

The environment variables are accessed by the Getenv and Setenv functions  in the os package. The names of the functions are self-explanatory and do not need any further description. 

There is one more useful function in the os package. The LookupEnv function provides two values as a result; the value of the variable, and the boolean value which defines if the variable was set or not in the environment.

The disadvantage of the os.Getenv function is that it returns an empty string, even in cases where the environment variable is not set.

This handicap could be overcome by the os.LookupEnv function, which returns the string as a value of the environment variable and the boolean value that indicates whether the variable was set or not.

 To implement the retrieval of the environment variable or the default one, use the os.LookupEnv function. Simply, if the variable is not set, which means that the second returned value is false, then the default value is returned. The use of the function is part of step 9.

        package main

        import (
          "fmt"
          "os"
          "path/filepath"
        )

        func main() {
          ex, err := os.Executable()
          if err != nil {
            panic(err)
          }

          // Path to executable file
          fmt.Println(ex)

          // Resolve the direcotry
          // of the executable
          exPath := filepath.Dir(ex)
          fmt.Println("Executable path :" + exPath)

          // Use EvalSymlinks to get
          // the real path.
          realPath, err := filepath.EvalSymlinks(exPath)
          if err != nil {
            panic(err)
          }
          fmt.Println("Symlink evaluated:" + realPath)
        }

Since Go 1.8, the Executable function from the os package is the preferred way of resolving the path of the executable. The Executable function returns the absolute path of the binary that is executed (unless the error is returned).

To resolve the directory from the binary path, the Dir from the filepath package is applied. The only pitfall of this is that the result could be the symlink or the path it pointed to.

To overcome this unstable behavior, the EvalSymlinks from the filepath package could be applied to the resultant path. With this hack, the returned value would be the real path of the binary.

The information about the directory where the binary is located could be obtained with the use of the Executable function in the os library.

Note that if the code is run by the command go run, the actual executable is located in a temporary directory.

        package main

        import (
          "fmt"
          "os"
          "os/exec"
          "strconv"
        )

        func main() {

          pid := os.Getpid()
          fmt.Printf("Process PID: %d \n", pid)

          prc := exec.Command("ps", "-p", strconv.Itoa(pid), "-v")
          out, err := prc.Output()
          if err != nil {
            panic(err)
          }

          fmt.Println(string(out))
        }

The function Getpid from the os package returns the PID of a process. The sample code shows how to get more information on the process from the operating system utility ps.

It could be useful to print the PID at the start of the application, so at the time of the crash, the cause could also be investigated by the retrieved PID.

        package main

        import (
          "fmt"
          "os"
          "os/signal"
          "syscall"
        )

        func main() {

          // Create the channel where the received
          // signal would be sent. The Notify
          // will not block when the signal
          // is sent and the channel is not ready.
          // So it is better to
          // create buffered channel.
          sChan := make(chan os.Signal, 1)

          // Notify will catch the
          // given signals and send
          // the os.Signal value
          // through the sChan.
          // If no signal specified in 
          // argument, all signals are matched.
          signal.Notify(sChan,
            syscall.SIGHUP,
            syscall.SIGINT,
            syscall.SIGTERM,
            syscall.SIGQUIT)

          // Create channel to wait till the
          // signal is handled.
          exitChan := make(chan int)
          go func() {
            signal := <-sChan
            switch signal {
              case syscall.SIGHUP:
                fmt.Println("The calling terminal has been closed")
                exitChan <- 0

              case syscall.SIGINT:
                fmt.Println("The process has been interrupted by CTRL+C")
                exitChan <- 1

              case syscall.SIGTERM:
                fmt.Println("kill SIGTERM was executed for process")
                exitChan <- 1

              case syscall.SIGQUIT:
                fmt.Println("kill SIGQUIT was executed for process")
                exitChan <- 1
            }
          }()

          code := <-exitChan
          os.Exit(code)
        }

In an application, where the resources are acquired, a resource leak could happen in the case of an instant termination. It is better to handle the signals and take some necessary steps to release the resources. The preceding code shows the concept of how to do that.

The Notify function from the signal package would be the one that helps us to handle the received signals.

Note that the Notify function of the signal package is communicating with the goroutine by the sChan channel. Notify then catches the defined signals and sends these to goroutine to be handled. Finally, exitChan is used to resolve the exit code of the process.

The important information is that the Notify function will not block the signal if the assigned channel is not ready. This way the signal could be missed. To avoid missing the signal, it is better to create the buffered channel.

Test if the following commands work in your Terminal:

The following steps cover the solution:

        package main

        import (
          "bytes"
          "fmt"
          "os/exec"
        )

        func main() {

          prc := exec.Command("ls", "-a")
          out := bytes.NewBuffer([]byte{})
          prc.Stdout = out
          err := prc.Run()
          if err != nil {
            fmt.Println(err)
          }

          if prc.ProcessState.Success() {
            fmt.Println("Process run successfully with output:\n")
            fmt.Println(out.String())
          }
        }

        package main

        import (
          "fmt"
          "os/exec"
        )

        func main() {

          prc := exec.Command("ls", "-a")
          err := prc.Start()
          if err != nil {
            fmt.Println(err)
          }

          prc.Wait()

          if prc.ProcessState.Success() {
            fmt.Println("Process run successfully with output:\n")
            fmt.Println(out.String())
          }
        }

The Go standard library provides a simple way of calling the external process. This could be done by the Command function of the os/exec package.

The simplest way is to create the Cmd struct and call the Run function. The Run function executes the process and waits until it completes. If the command exited with an error, the err value is not null.

This is more suitable for calling the OS utils and tools, so the program does not hang too long.

The process could be executed asynchronously too. This is done by calling the Start method of the Cmd structure. In this case, the process is executed, but the main goroutine does not wait until it ends. The Wait method could be used to wait until the process ends. After the Wait method finishes, the resources of the process are released.

This approach is more suitable for executing long-running processes and services that the program depends on.

This recipe describes how to simply execute the child process. There are Retrieve child process information and Reading/writing from the child process recipes in this chapter that also provide the steps on how to read from and write to the child process, and get useful information about the process.

Test if the sleep (timeout for Windows) command exists in the Terminal.

        package main

        import (
          "fmt"
          "os/exec"
          "runtime"
        )

        func main() {

          var cmd string
          if runtime.GOOS == "windows" {
            cmd = "timeout"
          } else {
            cmd = "sleep"
          }
          proc := exec.Command(cmd, "1")
          proc.Start()

          // No process state is returned
          // till the process finish.
          fmt.Printf("Process state for running process: %v\n",
                     proc.ProcessState)

          // The PID could be obtain
          // event for the running process
          fmt.Printf("PID of running process: %d\n\n", 
                     proc.Process.Pid)
        }

        func main() {

          var cmd string
          if runtime.GOOS == "windows" {
            cmd = "timeout"
          } else {
            cmd = "sleep"
          }

          proc := exec.Command(cmd, "1")
          proc.Start()

          // Wait function will
          // wait till the process ends.
          proc.Wait()

          // After the process terminates
          // the *os.ProcessState contains
          // simple information
          // about the process run
          fmt.Printf("PID: %d\n", proc.ProcessState.Pid())
          fmt.Printf("Process took: %dms\n", 
                     proc.ProcessState.SystemTime()/time.Microsecond)
          fmt.Printf("Exited sucessfuly : %t\n",
                     proc.ProcessState.Success())
        }

The os/exec standard library provides the way to execute the process. Using Command, the Cmd structure is returned. The Cmd provides the access to process the representation. When the process is running, you can only find out the PID.

There is only a little information that you can retrieve about the process. But by retrieving the PID of the process, you are able to call the utilities from the OS to get more information.

There are Reading/writing from the child process and Calling an external process recipes in this chapter that are related to process handling.

Verify if the following commands work in the Terminal:

       package main

       import (
         "fmt"
         "os/exec"
         "runtime"
       )

       func main() {

         var cmd string

         if runtime.GOOS == "windows" {
           cmd = "dir"
         } else {
           cmd = "ls"
         }

         proc := exec.Command(cmd)

         // Output will run the process
         // terminates and returns the standard
         // output in a byte slice.
         buff, err := proc.Output()

         if err != nil {
           panic(err)
         }

         // The output of child
         // process in form
         // of byte slice
         // printed as string
         fmt.Println(string(buff))

       }

        package main

        import (
          "bytes"
          "fmt"
          "os/exec"
          "runtime"
        )

        func main() {

          var cmd string

          if runtime.GOOS == "windows" {
            cmd = "dir"
          } else {
            cmd = "ls"
          }

          proc := exec.Command(cmd)

          buf := bytes.NewBuffer([]byte{})

          // The buffer which implements
          // io.Writer interface is assigned to
          // Stdout of the process
          proc.Stdout = buf

          // To avoid race conditions
          // in this example. We wait till
          // the process exit.
          proc.Run()

          // The process writes the output to
          // to buffer and we use the bytes
          // to print the output.
          fmt.Println(string(buf.Bytes()))

        }

        package main

        import (
          "bufio"
          "context"
          "fmt"
          "os/exec"
          "time"
        )

        func main() {
          cmd := "ping"
          timeout := 2 * time.Second

          // The command line tool
          // "ping" is executed for
          // 2 seconds
          ctx, _ := context.WithTimeout(context.TODO(), timeout)
          proc := exec.CommandContext(ctx, cmd, "example.com")

          // The process output is obtained
          // in form of io.ReadCloser. The underlying
          // implementation use the os.Pipe
          stdout, _ := proc.StdoutPipe()
          defer stdout.Close()

          // Start the process
          proc.Start()

          // For more comfortable reading the
          // bufio.Scanner is used.
          // The read call is blocking.
          s := bufio.NewScanner(stdout)
          for s.Scan() {
            fmt.Println(s.Text())
          }
        }

        package main

        import (
          "bufio"
          "fmt"
          "os"
        )

        func main() {
          sc := bufio.NewScanner(os.Stdin)

          for sc.Scan() {
            fmt.Println(sc.Text())
          }
        }

        package main

        import (
          "bufio"
          "fmt"
          "io"
          "os/exec"
          "time"
        )

        func main() {
          cmd := []string{"go", "run", "sample.go"}

          // The command line tool
          // "ping" is executed for
          // 2 seconds
          proc := exec.Command(cmd[0], cmd[1], cmd[2])

          // The process input is obtained
          // in form of io.WriteCloser. The underlying
          // implementation use the os.Pipe
          stdin, _ := proc.StdinPipe()
          defer stdin.Close()

          // For debugging purposes we watch the
          // output of the executed process
          stdout, _ := proc.StdoutPipe()
          defer stdout.Close()

          go func() {
            s := bufio.NewScanner(stdout)
            for s.Scan() {
              fmt.Println("Program says:" + s.Text())
            }
          }()

          // Start the process
          proc.Start()

          // Now the following lines
          // are written to child
          // process standard input
          fmt.Println("Writing input")
          io.WriteString(stdin, "Hello\n")
          io.WriteString(stdin, "Golang\n")
          io.WriteString(stdin, "is awesome\n")

          time.Sleep(time.Second * 2)

          proc.Process.Kill()

        }

The Cmd structure of the os/exec package provides the functions to access the output/input of the process. There are a few approaches to read the output of the process.

One of the simplest ways to read the process output is to use the Output or CombinedOutput method of the Cmd structure (gets Stderr and Stdout). While calling this function, the program synchronously waits till the child process terminates and then returns the output to a byte buffer.

Besides the Output and OutputCombined methods, the Cmd structure provides the Stdout property, where the io.Writer could be assigned. The assigned writer then serves as a destination for the process output. It could be a file, byte buffer or any type implementing the io.Writer interface.

The last approach to read the process output is to get the io.Reader from the Cmd structure by calling the StdoutPipe method. The StdoutPipe method creates the pipe between the Stdout, where the process writes the output, and provides Reader which works as the interface for the program to read the process output. This way the output of the process is piped to the retrieved io.Reader . 

Writing to a process stdin works the same way. Of all the options, the one with io.Writer will be demonstrated.

As could be seen, there are a few ways to read and write from the child process. The use of stderr and stdin is almost the same as described in steps 6-7. Finally, the approach of how to access the input/output could be divided this way:

The IO type is more flexible and could also be used asynchronously.

        package main

        import (
          "fmt"
          "io"
          "log"
          "os"
          "os/signal"
          "syscall"
          "time"
        )

        var writer *os.File

        func main() {

          // The file is opened as
          // a log file to write into.
          // This way we represent the resources
          // allocation.
          var err error
          writer, err = os.OpenFile(fmt.Sprintf("test_%d.log",
                time.Now().Unix()), os.O_RDWR|os.O_CREATE, os.ModePerm)
          if err != nil {
            panic(err)
          }

          // The code is running in a goroutine
          // independently. So in case the program is
          // terminated from outside, we need to
          // let the goroutine know via the closeChan
          closeChan := make(chan bool)
          go func() {
            for {
              time.Sleep(time.Second)
              select {
                case <-closeChan:
                  log.Println("Goroutine closing")
                  return
                default:
                  log.Println("Writing to log")
                  io.WriteString(writer, fmt.Sprintf("Logging access
                                 %s\n", time.Now().String()))
              }  

            }
          }()

          sigChan := make(chan os.Signal, 1)
          signal.Notify(sigChan,
            syscall.SIGTERM,
            syscall.SIGQUIT,
            syscall.SIGINT)

          // This is blocking read from
          // sigChan where the Notify function sends
          // the signal.
          <-sigChan

          // After the signal is received
          // all the code behind the read from channel could be
          // considered as a cleanup.
          // CLEANUP SECTION
          close(closeChan)
          releaseAllResources()
          fmt.Println("The application shut down gracefully")
        }

        func releaseAllResources() {
          io.WriteString(writer, "Application releasing 
                         all resources\n")
          writer.Close()
        }

The reading from a sigChan is blocking so the program keeps running until the Signal is sent through the channel. The sigChan is the channel where the Notify function sends the signals.

The main code of the program runs in a new goroutine. This way, the work continues while the main function is blocked on the sigChan. Once the signal from operation system is sent to process, the sigChan receives the signal and the code below the line where the reading from the sigChan channel is executed. This code section could be considered as the cleanup section.

Note that the step 7 terminal output contains the final log, Application releasing all resources, which is part of the cleanup section.

A detailed description of how the signal catching works is in the recipe Handling operating system signals.

        package main

        import (
          "encoding/json"
          "fmt"
          "os"
        )

        type Client struct {
          consulIP string
          connString string
        }

        func (c *Client) String() string {
          return fmt.Sprintf("ConsulIP: %s , Connection String: %s",
                             c.consulIP, c.connString)
        }

        var defaultClient = Client{
          consulIP: "localhost:9000",
          connString: "postgres://localhost:5432",
        }

        // ConfigFunc works as a type to be used
        // in functional options
        type ConfigFunc func(opt *Client)

        // FromFile func returns the ConfigFunc
        // type. So this way it could read the configuration
        // from the json.
        func FromFile(path string) ConfigFunc {
          return func(opt *Client) {
            f, err := os.Open(path)
            if err != nil {
              panic(err)
            }
            defer f.Close()
            decoder := json.NewDecoder(f)

            fop := struct {
              ConsulIP string `json:"consul_ip"`
            }{}
            err = decoder.Decode(&fop)
            if err != nil {
              panic(err)
            }
            opt.consulIP = fop.ConsulIP
          }
        }

        // FromEnv reads the configuration
        // from the environmental variables
        // and combines them with existing ones.
        func FromEnv() ConfigFunc {
          return func(opt *Client) {
            connStr, exist := os.LookupEnv("CONN_DB")
            if exist {
              opt.connString = connStr
            }
          }
        }

        func NewClient(opts ...ConfigFunc) *Client {
          client := defaultClient
          for _, val := range opts {
            val(&client)
          }
          return &client
        }

        func main() {
          client := NewClient(FromFile("config.json"), FromEnv())
          fmt.Println(client.String())
        }

        {
          "consul_ip":"127.0.0.1"
        }

The core concept of the functional options pattern is that the configuration API contains the functional parameters. In this case, the NewClient function accepts a various number of ConfigFunc arguments, which are then applied one by one on the defaultClient struct. This way, the default configuration is modified with huge flexibility. 

See the FromFile and FromEnv functions, which return the ConfigFunc, that is in fact, accessing the file or environmental variables.

Finally, you can check the output which applied both the configuration options and resulting Client struct that contains the values from the file and environmental variables.