In this article by Mikael Lundin, author of the book Testing with F#, we will see how unit testing is the art of designing our program in such a way that we can easily test each function as isolated units and such verify its correctness. Unit testing is not only a tool for verification of functionality, but also mostly a tool for designing that functionality in a testable way. What you gain is the means of finding problems early, facilitating change, documentation, and design.

In this article, we will dive into how to write good unit tests using F#:

• Testing in isolation
• Finding the abstraction level

(For more resources related to this topic, see here.)

FsUnit

The current state of unit testing in F# is good. You can get all the major test frameworks running with little effort, but there is still something that feels a bit off with the way tests and asserts are expressed:

open NUnit.Framework
Assert.That(result, Is.EqualTo(42))

Using FsUnit, you can achieve much higher expressiveness in writing unit tests by simply reversing the way the assert is written:

open FsUnit
result |> should equal 42

The FsUnit framework is not a test runner in itself, but uses an underlying test framework to execute. The underlying framework can be of MSTest, NUnit, or xUnit. FsUnit can best be explained as having a different structure and syntax while writing tests.

While this is a more dense syntax, the need for structure still exists and AAA is more needed more than ever. Consider the following test example:

[<Measure>] type EUR
[<Measure>] type SEK
type Country = | Sweden | Germany | France
 
let calculateVat country (amount : float<'u>) =
   match country with
   | Sweden -> amount * 0.25
   | Germany -> amount * 0.19
   | France -> amount * 0.2
 
open NUnit.Framework
open FsUnit
 
[<Test>]
let ``Sweden should have 25% VAT`` () =
   let amount = 200.<SEK>
 
   calculateVat Sweden amount |> should equal 50<SEK>

This code will calculate the VAT in Sweden in Swedish currency. What is interesting is that when we break down the test code and see that it actually follows the AAA structure, even it doesn't explicitly tell us this is so:

[<Test>]
let ``Germany should have 19% VAT`` () =
   // arrange
   let amount = 200.<EUR>
   // act
   calculateVat Germany amount
   //assert
   |> should equal 38<EUR>

The only thing I did here was add the annotations for AAA. It gives us the perspective of what we're doing, what frames we're working inside, and the rules for writing good unit tests.

Assertions

We have already seen the equal assertion, which verifies that the test result is equal to the expected value:

result |> should equal 42

You can negate this assertion by using the not' statement, as follows:

result |> should not' (equal 43)

With strings, it's quite common to assert that a string starts or ends with some value, as follows:

"$12" |> should startWith "$"
"$12" |> should endWith "12"

And, you can also negate that, as follows:

"$12" |> should not' (startWith "€")
"$12" |> should not' (endWith "14")

You can verify that a result is within a boundary. This will, in turn, verify that the result is somewhere between the values of 35-45:

result |> should (equalWithin 5) 40

And, you can also negate that, as follows:

result |> should not' ((equalWithin 1) 40)

With the collection types list, array, and sequence, you can check that it contains a specific value:

[1..10] |> should contain 5

And, you can also negate it to verify that a value is missing, as follows:

[1; 1; 2; 3; 5; 8; 13] |> should not' (contain 7)

It is common to test the boundaries of a function and then its exception handling. This means you need to be able to assert exceptions, as follows:

let getPersonById id = failwith "id cannot be less than 0"
(fun () -> getPersonById -1 |> ignore) |> should throw typeof<System.Exception>

There is a be function that can be used in a lot of interesting ways. Even in situations where the equal assertion can replace some of these be structures, we can opt for a more semantic way of expressing our assertions, providing better error messages.

Let us see examples of this, as follows:

// true or false
1 = 1 |> should be True
1 = 2 |> should be False
      
// strings as result
"" |> should be EmptyString
null |> should be NullOrEmptyString
 
// null is nasty in functional programming
[] |> should not' (be Null)
 
// same reference
let person1 = new System.Object()
let person2 = person1
person1 |> should be (sameAs person2)
 
// not same reference, because copy by value
let a = System.DateTime.Now
let b = a
a |> should not' (be (sameAs b))
 
// greater and lesser
result |> should be (greaterThan 0)
result |> should not' (be lessThan 0)
 
// of type
result |> should be ofExactType<int>
 
// list assertions
[] |> should be Empty
[1; 2; 3] |> should not' (be Empty)

With this, you should be able to assert most of the things you're looking for. But there still might be a few edge cases out there that default FsUnit asserts won't catch.

Custom assertions

FsUnit is extensible, which makes it easy to add your own assertions on top of the chosen test runner. This has the possibility of making your tests extremely readable.

The first example will be a custom assert which verifies that a given string matches a regular expression. This will be implemented using NUnit as a framework, as follows:

open FsUnit
open NUnit.Framework.Constraints
open System.Text.RegularExpressions
 
// NUnit: implement a new assert
type MatchConstraint(n) =
   inherit Constraint() with
      override this.WriteDescriptionTo(writer : MessageWriter) : unit =
           writer.WritePredicate("matches")
           writer.WriteExpectedValue(sprintf "%s" n)
       override this.Matches(actual : obj) =
           match actual with
           | :? string as input -> Regex.IsMatch(input, n)
           | _ -> failwith "input must be of string type"
          
let match' n = MatchConstraint(n)
 
open NUnit.Framework
 
[<Test>]
let ``NUnit custom assert`` () =
   "2014-10-11" |> should match' "d{4}-d{2}-d{2}"
   "11/10 2014" |> should not' (match' "d{4}-d{2}-d{2}")

In order to create your own assert, you need to create a type that implements the NUnit.Framework.Constraints.IConstraint interface, and this is easily done by inheriting from the Constraint base class.

You need to override both the WriteDescriptionTo() and Matches() method, where the first one controls the message that will be output from the test, and the second is the actual test. In this implementation, I verify that input is a string; or the test will fail. Then, I use the Regex.IsMatch() static function to verify the match.

Next, we create an alias for the MatchConstraint() function, match', with the extra apostrophe to avoid conflict with the internal F# match expression, and then we can use it as any other assert function in FsUnit.

Doing the same for xUnit requires a completely different implementation. First, we need to add a reference to NHamcrest API. We'll find it by searching for the package in the NuGet Package Manager:



Instead, we make an implementation that uses the NHamcrest API, which is a .NET port of the Java Hamcrest library for building matchers for test expressions, shown as follows:

open System.Text.RegularExpressions
open NHamcrest
open NHamcrest.Core
 
// test assertion for regular expression matching
let match' pattern =
   CustomMatcher<obj>(sprintf "Matches %s" pattern, fun c ->
       match c with
       | :? string as input -> Regex.IsMatch(input, pattern)
       | _ -> false)
 
open Xunit
open FsUnit.Xunit
 
[<Fact>]
let ``Xunit custom assert`` () =
   "2014-10-11" |> should match' "d{4}-d{2}-d{2}"
   "11/10 2014" |> should not' (match' "d{4}-d{2}-d{2}")

The functionality in this implementation is the same as the NUnit version, but the implementation here is much easier. We create a function that receives an argument and return a CustomMatcher<obj> object. This will only take the output message from the test and the function to test the match.

Writing an assertion for FsUnit driven by MSTest works exactly the same way as it would in Xunit, by NHamcrest creating a CustomMatcher<obj> object.

Unquote

There is another F# assertion library that is completely different from FsUnit but with different design philosophies accomplishes the same thing, by making F# unit tests more functional. Just like FsUnit, this library provides the means of writing assertions, but relies on NUnit as a testing framework.

Instead of working with a DSL like FsUnit or API such as with the NUnit framework, the Unquote library assertions are based on F# code quotations.

Code quotations is a quite unknown feature of F# where you can turn any code into an abstract syntax tree. Namely, when the F# compiler finds a code quotation in your source file, it will not compile it, but rather expand it into a syntax tree that represents an F# expression.

The following is an example of a code quotation:

<@ 1 + 1 @>

If we execute this in F# Interactive, we'll get the following output:

val it : Quotations.Expr =
Call (None, op_Addition, [Value (1), Value (1)])

This is truly code as data, and we can use it to write code that operates on code as if it was data, which in this case, it is. It brings us closer to what a compiler does, and gives us lots of power in the metadata programming space.

We can use this to write assertions with Unquote. Start by including the Unquote NuGet package in your test project, as shown in the following screenshot:



And now, we can implement our first test using Unquote, as follows:

open NUnit.Framework
open Swensen.Unquote
 
[<Test>]
let ``Fibonacci sequence should start with 1, 1, 2, 3, 5`` () =     test <@ fibonacci |> Seq.take 5 |> 
 List.ofSeq = [1; 1; 2; 3; 5] @>

This works by Unquote first finding the equals operation, and then reducing each side of the equals sign until they are equal or no longer able to reduce.

Writing a test that fails and watching the output more easily explains this. The following test should fail because 9 is not a prime number:

[<Test>]
let ``prime numbers under 10 are 2, 3, 5, 7, 9`` () =
   test <@ primes 10 = [2; 3; 5; 7; 9] @> // fail

The test will fail with the following message:

Test Name: prime numbers under 10 are 2, 3, 5, 7, 9
Test FullName: chapter04.prime numbers under 10 are 2, 3, 5, 7, 9
Test Outcome: Failed
Test Duration: 0:00:00.077
 
Result Message:
primes 10 = [2; 3; 5; 7; 9]
[2; 3; 5; 7] = [2; 3; 5; 7; 9]
false
 
Result StackTrace:
at Microsoft.FSharp.Core.Operators.Raise[T](Exception exn)
at chapter04.prime numbers under 10 are 2, 3, 5, 7, 9()

In the resulting message, we can see both sides of the equals sign reduced until only false remains. It's a very elegant way of breaking down a complex assertion.

Assertions

The assertions in Unquote are not as specific or extensive as the ones in FsUnit. The idea of having lots of specific assertions for different situations is to get very descriptive error messages when the tests fail. Since Unquote actually outputs the whole reduction of the statements when the test fails, the need for explicit assertions is not that high. You'll get a descript error message anyway. The absolute most common is to check for equality, as shown before. You can also verify that two expressions are not equal:

test <@ 1 + 2 = 4 - 1 @>
test <@ 1 + 2 <> 4 @>

We can check whether a value is greater or smaller than the expected value:

test <@ 42 < 1337 @>
test <@ 1337 > 42 @>

You can check for a specific exception, or just any exception:

raises<System.NullReferenceException> <@ (null : string).Length @>
raises<exn> <@ System.String.Format(null, null) @>

Here, the Unquote syntax excels compared to FsUnit, which uses a unit lambda expression to do the same thing in a quirky way.

The Unquote library also has its reduce functionality in the public API, making it possible for you to reduce and analyze an expression. Using the reduceFully syntax, we can get the reduction in a list, as shown in the following:

> <@ (1+2)/3 @> |> reduceFully |> List.map decompile;;
val it : string list = ["(1 + 2) / 3"; "3 / 3"; "1"]

If we just want the output to console output, we can run the unquote command directly:

> unquote <@ [for i in 1..5 -> i * i] = ([1..5] |> List.map (fun i -> i * i)) @>;;
Seq.toList (seq (Seq.delay (fun () -> Seq.map (fun i -> i * i) {1..5}))) = ([1..5] |> List.map (fun i -> i * i))
Seq.toList (seq seq [1; 4; 9; 16; ...]) = ([1; 2; 3; 4; 5] |> List.map (fun i -> i * i))
Seq.toList seq [1; 4; 9; 16; ...] = [1; 4; 9; 16; 25]
[1; 4; 9; 16; 25] = [1; 4; 9; 16; 25]
true

It is important to know what tools are out there, and Unquote is one of those tools that is fantastic to know about when you run into a testing problem in which you want to reduce both sides of an equals sign. Most often, this belongs to difference computations or algorithms like price calculation.

We have also seen that Unquote provides a great way of expressing tests for exceptions that is unmatched by FsUnit.

Testing in isolation

One of the most important aspects of unit testing is to test in isolation. This does not only mean to fake any external dependency, but also that the test code itself should not be tied up to some other test code.

If you're not testing in isolation, there is a potential risk that your test fails. This is not because of the system under test, but the state that has lingered from a previous test run, or external dependencies.

Writing pure functions without any state is one way of making sure your test runs in isolation. Another way is by making sure that the test creates all the needed state itself.

• Shared state, like connections, between tests is a bad idea.
• Using TestFixtureSetUp/TearDown attributes to set up a state for a set of tests is a bad idea.
• Keeping low performance resources around because they're expensive to set up is a bad idea.

The most common shared states are the following:

• The ASP.NET Model View Controller (MVC) session state
• Dependency injection setup
• Database connection, even though it is no longer strictly a unit test

Here's how one should think about unit testing in isolation, as shown in the following screenshot:



Each test is responsible for setting up the SUT and its database/web service stubs in order to perform the test and assert on the result. It is equally important that the test cleans up after itself, which in the case of unit tests most often can be handed over to the garbage collector, and doesn't need to be explicitly disposed. It is common to think that one should only isolate a test fixture from other test fixtures, but this idea of a test fixture is bad. Instead, one should strive for having each test stand for itself to as large an extent as possible, and not be dependent on outside setups. This does not mean you will have unnecessary long unit tests, provided you write SUT and tests well within that context.

The problem we often run into is that the SUT itself maintains some kind of state that is present between tests. The state can simply be a value that is set in the application domain and is present between different test runs, as follows:

let getCustomerFullNameByID id =

   if cache.ContainsKey(id) then
       (cache.[id] :?> Customer).FullName
   else
       // get from database
       // NOTE: stub code
       let customer = db.getCustomerByID id
       cache.[id] <- customer
       customer.FullName

The problem we see here is that the cache will be present from one test to another, so when the second test is running, it needs to make sure that its running with a clean cache, or the result might not be as expected.

One way to test it properly would be to separate the core logic from the cache and test them each independently. Another would be to treat it as a black box and ignore the cache completely. If the cache makes the test fail, then the functionality fails as a whole.

This depends on if we see the cache as an implementation detail of the function or a functionality by itself. Testing implementation details, or private functions, is dirty because our tests might break even if the functionality hasn't changed. And yet, there might be benefits into taking the implementation detail into account. In this case, we could use the cache functionality to easily stub out the database without the need of any mocking framework.

Vertical slice testing

Most often, we deal with dependencies as something we need to mock away, where as the better option would be to implement a test harness directly into the product. We know what kind of data and what kind of calls we need to make to the database, so right there, we have a public API for the database. This is often called a data access layer in a three-tier architecture (but no one ever does those anymore, right?).

As we have a public data access layer, we could easily implement an in-memory representation that can be used not only by our tests, but in development of the product, as shown in the following image:



When you're running the application in development mode, you configure it toward the in-memory version of the dependency. This provides you with the following benefits:

• You'll get a faster development environment
• Your tests will become simpler
• You have complete control of your dependency

As your development environment is doing everything in memory, it becomes blazing fast. And as you develop your application, you will appreciate adjusting that public API and getting to understand completely what you expect from that dependency. It will lead to a cleaner API, where very few side effects are allowed to seep through.

Your tests will become much simpler, as instead of mocking away the dependency, you can call the in-memory dependency and set whatever state you want.

Here's an example of what a public data access API might look like:

type IDataAccess =
   abstract member GetCustomerByID : int -> Customer
   abstract member FindCustomerByName : string -> Customer option
   abstract member UpdateCustomerName : int -> string -> Customer
   abstract member DeleteCustomerByID : int -> bool

This is surely a very simple API, but it will demonstrate the point. There is a database with a customer inside it, and we want to do some operations on that.

In this case, our in-memory implementation would look like this:

type InMemoryDataAccess() =
   let data = new System.Collections.Generic.Dictionary<int, Customer>()
 
   // expose the add method
   member this.Add customer = data.Add(customer.ID, customer)
 
   interface IDataAccess with
      // throw exception if not found
       member this.GetCustomerByID id =
           data.[id]
       member this.FindCustomerByName fullName =
           data.Values |> Seq.tryFind (fun customer -> customer.FullName = fullName)
 
       member this.UpdateCustomerName id fullName =
           data.[id] <- { data.[id] with FullName = fullName }
           data.[id]
 
       member this.DeleteCustomerByID id =
           data.Remove(id)

This is a simple implementation that provides the same functionality as the database would, but in memory. This makes it possible to run the tests completely in isolation without worrying about mocking away the dependencies. The dependencies are already substituted with in-memory replacements, and as seen with this example, the in-memory replacement doesn't have to be very extensive.

The only extra function except from the interface implementation is the Add() function which lets us set the state prior to the test, as this is something the interface itself doesn't provide for us.

Now, in order to sew it together with the real implementation, we need to create a configuration in order to select what version to use, as shown in the following code:

open System.Configuration
open System.Collections.Specialized
 
// TryGetValue extension method to NameValueCollection
type NameValueCollection with
   member this.TryGetValue (key : string) =
       if this.Get(key) = null then
           None
       else
           Some (this.Get key)
 
let dataAccess : IDataAccess =
   match ConfigurationManager.AppSettings.TryGetValue("DataAccess") with
   | Some "InMemory" -> new InMemoryDataAccess() :> IDataAccess
   | Some _ | None -> new DefaultDataAccess() :> IDataAccess
      
// usage
let fullName = (dataAccess.GetCustomerByID 1).FullName

Again, with only a few lines of code, we manage to select the appropriate IDataAccess instance and execute against it without using dependency injection or taking a penalty in code readability, as we would in C#.

The code is straightforward and easy to read, and we can execute any tests we want without touching the external dependency, or in this case, the database.

Finding the abstraction level

In order to start unit testing, you have to start writing tests; this is what they'll tell you. If you want to get good at it, just start writing tests, any and a lot of them. The rest will solve itself.

I've watched experienced developers sit around staring dumbfounded at an empty screen because they couldn't get into their mind how to get started, what to test.

The question is not unfounded. In fact, it is still debated in the Test Driven Development (TDD) community what should be tested. The ground rule is that the test should bring at least as much value as the cost of writing it, but that is a bad rule for someone new to testing, as all tests are expensive for them to write.

Summary

In this article, we've learned how to write unit tests by using the appropriate tools to our disposal: NUnit, FsUnit, and Unquote. We have also learned about different techniques for handling external dependencies, using interfaces and functional signatures, and executing dependency injection into constructors, properties, and methods.

Resources for Article:

• Learning Option Pricing [article]
• Pentesting Using Python [article]
• Penetration Testing [article]