





















































This article, by Mani Tadayon, author of the book, RSpec Essentials, discusses support code to set tests up and clean up after them. Initialization, configuration, cleanup, and other support code related to RSpec specs are important in real-world RSpec usage. We will learn how to cleanly organize support code in real-world applications by learning about the following topics:
(For more resources related to this topic, see here.)
The RSpec specs that we've seen so far have functioned as standalone units. Specs in the real world, however, almost never work without supporting code to prepare the test environment before tests are run and ensure it is cleaned up afterwards. In fact, the first line of nearly every real-world RSpec spec file loads a file that takes care of initialization, configuration, and cleanup:
require 'spec_helper'
By convention, the entry point for all support code for specs is in a file called spec_helper.rb. Another convention is that specs are located in a folder called spec in the root folder of the project. The spec_helper.rb file is located in the root of this spec folder.
Now that we know where it goes, what do we actually put in spec_helper.rb? Let's start with an example:
# spec/spec_helper.rb
require 'rspec'
RSpec.configure do |config|
config.order = 'random'
config.profile_examples = 3
end
To see what these two options do, let's create a couple of dummy spec files that include our spec_helper.rb. Here's the first spec file:
# spec/first_spec.rb
require 'spec_helper'
describe 'first spec' do
it 'sleeps for 1 second' do
sleep 1
end
it 'sleeps for 2 seconds' do
sleep 2
end
it 'sleeps for 3 seconds' do
sleep 3
end
end
And here's our second spec file:
# spec/second_spec.rb
require 'spec_helper'
describe 'second spec' do
it 'sleeps for 4 second' do
sleep 4
end
it 'sleeps for 5 seconds' do
sleep 5
end
it 'sleeps for 6 seconds' do
sleep 6
end
end
Now let's run our two spec files and see what happens:
We note that we used --format documentation when running RSpec so that we see the order in which the tests were run (the default format just outputs a green dot for each passing test). From the output, we can see that the tests were run in a random order. We can also see the three slowest specs.
Although this was a toy example, I would recommend using both of these configuration options for RSpec. Running examples in a random order is very important, as it is the only reliable way of detecting bad tests which sometimes pass and sometimes fail based on the order the in which overall test suite is run. Also, keeping tests running fast is very important for maintaining a productive development flow, and seeing which tests are slow on every test run is the most effective way of encouraging developers to make the slow tests fast, or remove them from the test run.
We'll return to both test order and test speed later. For now, let us just note that RSpec configuration is very important to keeping our specs reliable and fast.
Real-world applications rely on resources, such as databases, and external services, such as HTTP APIs. These must be initialized and configured for the application to work properly. When writing tests, dealing with these resources and services can be a challenge because of two opposing fundamental interests.
First, we would like the test environment to match as closely as possible the production environment so that tests that interact with resources and services are realistic. For example, we may use a powerful database system in production that runs on many servers to provide the best performance. Should we spend money and effort to create and maintain a second production-grade database environment just for testing purposes?
Second, we would like the test environment to be simple and relatively easy to understand, so that we understand what we are actually testing. We would also like to keep our code modular so that components can be tested in isolation, or in simpler environments that are easier to create, maintain, and understand. If we think of the example of the system that relies on a database cluster in production, we may ask ourselves whether we are better off using a single-server setup for our test database. We could even go so far as to use an entirely different database for our tests, such as the file-based SQLite.
As always, there are no easy answers to such trade-offs. The important thing is to understand the costs and benefits, and adjust where we are on the continuum between production faithfulness and test simplicity as our system evolves, along with the goals it serves. For example, for a small hobbyist application or a project with a limited budget, we may choose to completely favor test simplicity. As the same code grows to become a successful fan site or a big-budget project, we may have a much lower tolerance for failure, and have both the motivation and resources to shift towards production faithfulness for our test environment.
Some rules of thumb to keep in mind:
Let's put these ideas into practice. I haven't changed the application code, except to rename the module OldWeatherQuery. The test code is also slightly changed to require a spec_helper file and to use a subject block to define an alias for the module name, which makes it easier to rename the code without having to change many lines of test code.
So let's look at our three files now. First, here's the application code:
# old_weather_query.rb
require 'net/http'
require 'json'
require 'timeout'
module OldWeatherQuery
extend self
class NetworkError < StandardError
end
def forecast(place, use_cache=true)
add_to_history(place)
if use_cache
cache[place] ||= begin
@api_request_count += 1
JSON.parse( http(place) )
end
else
JSON.parse( http(place) )
end
rescue JSON::ParserError
raise NetworkError.new("Bad response")
end
def api_request_count
@api_request_count ||= 0
end
def history
(@history || []).dup
end
def clear!
@history = []
@cache = {}
@api_request_count = 0
end
private
def add_to_history(s)
@history ||= []
@history << s
end
def cache
@cache ||= {}
end
BASE_URI = 'http://api.openweathermap.org/data/2.5/weather?q='
def http(place)
uri = URI(BASE_URI + place)
Net::HTTP.get(uri)
rescue Timeout::Error
raise NetworkError.new("Request timed out")
rescue URI::InvalidURIError
raise NetworkError.new("Bad place name: #{place}")
rescue SocketError
raise NetworkError.new("Could not reach #{uri.to_s}")
end
end
Next is the spec file:
# spec/old_weather_query_spec.rb
require_relative 'spec_helper'
require_relative '../old_weather_query'
describe OldWeatherQuery do
subject(:weather_query) { described_class }
describe 'caching' do
let(:json_response) do
'{"weather" : { "description" : "Sky is Clear"}}'
end
around(:example) do |example|
actual = weather_query.send(:cache)
expect(actual).to eq({})
example.run
weather_query.clear!
end
it "stores results in local cache" do
weather_query.forecast('Malibu,US')
actual = weather_query.send(:cache)
expect(actual.keys).to eq(['Malibu,US'])
expect(actual['Malibu,US']).to be_a(Hash)
end
it "uses cached result in subsequent queries" do
weather_query.forecast('Malibu,US')
weather_query.forecast('Malibu,US')
weather_query.forecast('Malibu,US')
end
end
describe 'query history' do
before do
expect(weather_query.history).to eq([])
allow(weather_query).to receive(:http).and_return("{}")
end
after do
weather_query.clear!
end
it "stores every place requested" do
places = %w(
Malibu,US
Beijing,CN
Delhi,IN
Malibu,US
Malibu,US
Beijing,CN
)
places.each {|s| weather_query.forecast(s) }
expect(weather_query.history).to eq(places)
end
it "does not allow history to be modified" do
expect {
weather_query.history = ['Malibu,CN']
}.to raise_error
weather_query.history << 'Malibu,CN'
expect(weather_query.history).to eq([])
end
end
describe 'number of API requests' do
before do
expect(weather_query.api_request_count).to eq(0)
allow(weather_query).to receive(:http).and_return("{}")
end
after do
weather_query.clear!
end
it "stores every place requested" do
places = %w(
Malibu,US
Beijing,CN
Delhi,IN
Malibu,US
Malibu,US
Beijing,CN
)
places.each {|s| weather_query.forecast(s) }
expect(weather_query.api_request_count).to eq(3)
end
it "does not allow count to be modified" do
expect {
weather_query.api_request_count = 100
}.to raise_error
expect {
weather_query.api_request_count += 10
}.to raise_error
expect(weather_query.api_request_count).to eq(0)
end
end
end
And last but not least, our spec_helper file, which has also changed only slightly: we only configure RSpec to show one slow spec (to keep test results uncluttered) and use color in the output to distinguish passes and failures more easily:
# spec/spec_helper.rb
require 'rspec'
RSpec.configure do |config|
config.order = 'random'
config.profile_examples = 1
config.color = true
end
When we run these specs, something unexpected happens. Most of the time the specs pass, but sometimes they fail. If we keep running the specs with the same command, we'll see the tests pass and fail apparently at random. These are flaky tests, and we have exposed them because of the random order configuration we chose. If our tests run in a certain order, they fail. The problem could be simply in our tests. For example, we could have forgotten to clear state before or after a test. However, there could also be a problem with our code. In any case, we need to get to the bottom of the situation:
We first notice that at the end of the failing test run, RSpec tells us "Randomized with seed 318". We can use this information to run the tests in the order that caused the failure and start to debug and diagnose the problem. We do this by passing the --seed parameter with the value 318, as follows:
$ rspec spec/old_weather_query_spec.rb --seed 318
The problem has to do with the way that we increment @api_request_count without ensuring it has been initialized. Looking at our code, we notice that the only place we initialize @api_request_count is in OldWeatherQuery.api_request_count and OldWeatherQuery.clear!. If we don't call either of these methods first, then OldWeatherQuery.forecast, the main method in this module, will always fail. Our tests sometimes pass because our setup code calls one of these methods first when tests are run in a certain order, but that is not at all how our code would likely be used in production. So basically, our code is completely broken, but our specs pass (sometimes). Based on this, we can create a simple spec that will always fail:
describe 'api_request is not initialized' do
it "does not raise an error" do
weather_query.forecast('Malibu,US')
end
end
At least now our tests fail deterministically. But this is not the end of our troubles with these specs. If we run our tests many times with the seed value of 318, we will start seeing a second failing test case that is even more random than the first. This is an OldWeatherQuery::NetworkError and it indicates that our tests are actually making HTTP requests to the Internet! Let's do an experiment to confirm this. We'll turn off our Wi-Fi access, unplug our Ethernet cables, and run our specs. When we run our tests without any Internet access, we will see three errors in total. One of them is the error with the uninitialized @api_request_count instance variable, and two of them are instances of OldWeatherQuery::NetworkError, which confirms that we are indeed making real HTTP requests in our code.
What's so bad about making requests to the Internet? After all, the test failures are indeed very random and we had to purposely shut off our Internet access to replicate the errors. Flaky tests are actually the least of our problems. First, we could be performing destructive actions that affect real systems, accounts, and people! Imagine if we were testing an e-commerce application that charged customer credit cards by using a third-party payment API via HTTP. If our tests actually hit our payment provider's API endpoint over HTTP, we would get a lot of declined transactions (assuming we are not storing and using real credit card numbers), which could lead to our account being suspended due to suspicions of fraud, putting our e-commerce application out of service. Also, if we were running a continuous integration (CI) server such as Jenkins, which did not have access to the public Internet, we would get failures in our CI builds due to failing tests that attempted to access the Internet.
There are a few approaches to solving this problem. In our tests, we attempted to mock our HTTP requests, but obviously failed to do so effectively. A second approach is to allow actual HTTP requests but to configure a special server for testing purposes. Let's focus on figuring out why our HTTP mocks were not successful. In a small set of tests like in this example, it is not hard to hunt down the places where we are sending actual HTTP requests. In larger code bases with a lot of test support code, it may be harder. Also, it would be nice to prevent access to the Internet altogether so we notice these issues as soon as we run the offending tests.
Fortunately, Ruby has many excellent tools for testing, and there is one that addresses our needs exactly: WebMock (https://github.com/bblimke/webmock). We simply install the gem and add a couple of lines to our spec helper file to disable all network connections in our tests:
require 'rspec'
# require the webmock gem
require 'webmock/rspec'
RSpec.configure do |config|
# this is done by default, but let's make it clear
WebMock.disable_net_connect!
Config.order = 'random'
config.profile_examples = 1
config.color = true
end
When we run our tests again, we'll see one or more instances of WebMock::NetConnectNotAllowedError, along with a backtrace to lead us to the point in our tests where the HTTP request was made:
If we examine our test code, we'll notice that we mock the OldWeatherQuery.http method in a few places. However, we forgot to set up the mock in the first describe block for caching where we defined a json_response object, but never mocked the OldWeatherQuery.http method to return json_response. We can solve the problem by mocking OldWeatherQuery.http throughout the entire test file. We'll also take this opportunity to clean up the initialization of @api_request_count in our code. Here's what we have now:
# new_weather_query.rb
require 'net/http'
require 'json'
require 'timeout'
module NewWeatherQuery
extend self
class NetworkError < StandardError
end
def forecast(place, use_cache=true)
add_to_history(place)
if use_cache
cache[place] ||= begin
increment_api_request_count
JSON.parse( http(place) )
end
else
JSON.parse( http(place) )
end
rescue JSON::ParserError => e
raise NetworkError.new("Bad response: #{e.inspect}")
end
def increment_api_request_count
@api_request_count ||= 0
@api_request_count += 1
end
def api_request_count
@api_request_count ||= 0
end
def history
(@history || []).dup
end
def clear!
@history = []
@cache = {}
@api_request_count = 0
end
private
def add_to_history(s)
@history ||= []
@history << s
end
def cache
@cache ||= {}
end
BASE_URI = 'http://api.openweathermap.org/data/2.5/weather?q='
def http(place)
uri = URI(BASE_URI + place)
Net::HTTP.get(uri)
rescue Timeout::Error
raise NetworkError.new("Request timed out")
rescue URI::InvalidURIError
raise NetworkError.new("Bad place name: #{place}")
rescue SocketError
raise NetworkError.new("Could not reach #{uri.to_s}")
end
end
And here is the spec file to go with it:
# spec/new_weather_query_spec.rb
require_relative 'spec_helper'
require_relative '../new_weather_query'
describe NewWeatherQuery do
subject(:weather_query) { described_class }
after { weather_query.clear! }
let(:json_response) { '{}' }
before do
allow(weather_query).to receive(:http).and_return(json_response)
end
describe 'api_request is initialized' do
it "does not raise an error" do
weather_query.forecast('Malibu,US')
end
end
describe 'caching' do
let(:json_response) do
'{"weather" : { "description" : "Sky is Clear"}}'
end
around(:example) do |example|
actual = weather_query.send(:cache)
expect(actual).to eq({})
example.run
end
it "stores results in local cache" do
weather_query.forecast('Malibu,US')
actual = weather_query.send(:cache)
expect(actual.keys).to eq(['Malibu,US'])
expect(actual['Malibu,US']).to be_a(Hash)
end
it "uses cached result in subsequent queries" do
weather_query.forecast('Malibu,US')
weather_query.forecast('Malibu,US')
weather_query.forecast('Malibu,US')
end
end
describe 'query history' do
before do
expect(weather_query.history).to eq([])
end
it "stores every place requested" do
places = %w(
Malibu,US
Beijing,CN
Delhi,IN
Malibu,US
Malibu,US
Beijing,CN
)
places.each {|s| weather_query.forecast(s) }
expect(weather_query.history).to eq(places)
end
it "does not allow history to be modified" do
expect {
weather_query.history = ['Malibu,CN']
}.to raise_error
weather_query.history << 'Malibu,CN'
expect(weather_query.history).to eq([])
end
end
describe 'number of API requests' do
before do
expect(weather_query.api_request_count).to eq(0)
end
it "stores every place requested" do
places = %w(
Malibu,US
Beijing,CN
Delhi,IN
Malibu,US
Malibu,US
Beijing,CN
)
places.each {|s| weather_query.forecast(s) }
expect(weather_query.api_request_count).to eq(3)
end
it "does not allow count to be modified" do
expect {
weather_query.api_request_count = 100
}.to raise_error
expect {
weather_query.api_request_count += 10
}.to raise_error
expect(weather_query.api_request_count).to eq(0)
end
end
end
Now we've fixed a major bug with our code that slipped through our specs and used to pass randomly. We've made it so that our tests always pass, regardless of the order in which they are run, and without needing to access the Internet. Our test code and application code has also become clearer as we've reduced duplication in a few places.
We're not done with our WeatherQuery example just yet. Let's take a look at how we would add a simple database to store our cached values. There are some serious limitations to the way we are caching with instance variables, which persist only within the scope of a single Ruby process. As soon as we stop or restart our app, the entire cache will be lost. In a production app, we would likely have many processes running the same code in order to serve traffic effectively. With our current approach, each process would have a separate cache, which would be very inefficient. We could easily save many HTTP requests if we were able to share the cache between processes and across restarts. Economizing on these requests is not simply a matter of improved response time. We also need to consider that we cannot make unlimited requests to external services. For commercial services, we would pay for the number of requests we make. For free services, we are likely to get throttled if we exceed some threshold. Therefore, an effective caching scheme that reduces the number of HTTP requests we make to our external services is of vital importance to the function of a real-world app. Finally, our cache is very simplistic and has no expiration mechanism short of clearing all entries. For a cache to be effective, we need to be able to store entries for individual locations for some period of time within which we don't expect the weather forecast to change much. This will keep the cache small and up to date.
We'll use Redis (http://redis.io) as our database since it is very fast, simple, and easy to set up. You can find instructions on the Redis website on how to install it, which is an easy process on any platform. Once you have Redis installed, you simply need to start the server locally, which you can do with the redis-server command. We'll also need to install the Redis Ruby client as a gem (https://github.com/redis/redis-rb).
Let's start with a separate configuration file to set up our Redis client for our tests:
# spec/config/redis.rb
require 'rspec'
require 'redis'
ENV['WQ_REDIS_URL'] ||= 'redis://localhost:6379/15'
RSpec.configure do |config|
if ! ENV['WQ_REDIS_URL'].is_a?(String)
raise "WQ_REDIS_URL environment variable not set"
end
::REDIS_CLIENT = Redis.new( :url => ENV['WQ_REDIS_URL'] )
config.after(:example) do
::REDIS_CLIENT.flushdb
end
end
Note that we place this file in a new config folder under our main spec folder. The idea is to configure each resource separately in its own file to keep everything isolated and easy to understand. This will make maintenance easy and prevent problems with configuration management down the road.
We don't do much in this file, but we do establish some important conventions. There is a single environment variable, which takes care of the Redis connection URL. By using an environment variable, we make it easy to change configuration and also allow flexibility in how these configurations are stored. Our code doesn't care if the Redis connection URL is stored in a simple .env file with key-value pairs or loaded from a configuration database. We can also easily override this value manually simply by setting it when we run RSpec, like so:
$ WQ_REDIS_URL=redis://1.2.3.4:4321/0 rspec spec
Note that we also set a sensible default value, which is to run on the default Redis port of 6379 on our local machine, on database number 15, which is less likely to be used for local development. This prevents our tests from relying on our development database, or from polluting or destroying it. It is also worth mentioning that we prefix our environment variable with WQ (short for weather query). Small details like this are very important for keeping our code easy to understand and to prevent dangerous clashes. We could imagine the kinds of confusion and clashes that could be caused if we relied on REDIS_URL and we had multiple apps running on the same server, all relying on Redis. It would be very easy to break many applications if we changed the value of REDIS_URL for a single app to point to a different instance of Redis.
We set a global constant, ::REDIS_CLIENT, to point to a Redis client. We will use this in our code to connect to Redis. Note that in real-world code, we would likely have a global namespace for the entire app and we would define globals such as REDIS_CLIENT under that namespace rather than in the global Ruby namespace.
Finally, we configure RSpec to call the flushdb command after every example tagged with :redis to empty the database and keep state clean across tests. In our code, all tests interact with Redis, so this tag seems pointless. However, it is very likely that we would add code that had nothing to do with Redis, and using tags helps us to constrain the scope of our configuration hooks only to where they are needed. This will also prevent confusion about multiple hooks running for the same example. In general, we want to prevent global hooks where possible and make configuration hooks explicitly triggered where possible.
So what does our spec look like now? Actually, it is almost exactly the same. Only a few lines have changed to work with the new Redis cache. See if you can spot them!
# spec/redis_weather_query_spec.rb
require_relative 'spec_helper'
require_relative '../redis_weather_query'
describe RedisWeatherQuery, redis: true do
subject(:weather_query) { described_class }
after { weather_query.clear! }
let(:json_response) { '{}' }
before do
allow(weather_query).to receive(:http).and_return(json_response)
end
describe 'api_request is initialized' do
it "does not raise an error" do
weather_query.forecast('Malibu,US')
end
end
describe 'caching' do
let(:json_response) do
'{"weather" : { "description" : "Sky is Clear"}}'
end
around(:example) do |example|
actual = weather_query.send(:cache).all
expect(actual).to eq({})
example.run
end
it "stores results in local cache" do
weather_query.forecast('Malibu,US')
actual = weather_query.send(:cache).all
expect(actual.keys).to eq(['Malibu,US'])
expect(actual['Malibu,US']).to be_a(Hash)
end
it "uses cached result in subsequent queries" do
weather_query.forecast('Malibu,US')
weather_query.forecast('Malibu,US')
weather_query.forecast('Malibu,US')
end
end
describe 'query history' do
before do
expect(weather_query.history).to eq([])
end
it "stores every place requested" do
places = %w(
Malibu,US
Beijing,CN
Delhi,IN
Malibu,US
Malibu,US
Beijing,CN
)
places.each {|s| weather_query.forecast(s) }
expect(weather_query.history).to eq(places)
end
it "does not allow history to be modified" do
expect {
weather_query.history = ['Malibu,CN']
}.to raise_error
weather_query.history << 'Malibu,CN'
expect(weather_query.history).to eq([])
end
end
describe 'number of API requests' do
before do
expect(weather_query.api_request_count).to eq(0)
end
it "stores every place requested" do
places = %w(
Malibu,US
Beijing,CN
Delhi,IN
Malibu,US
Malibu,US
Beijing,CN
)
places.each {|s| weather_query.forecast(s) }
expect(weather_query.api_request_count).to eq(3)
end
it "does not allow count to be modified" do
expect {
weather_query.api_request_count = 100
}.to raise_error
expect {
weather_query.api_request_count += 10
}.to raise_error
expect(weather_query.api_request_count).to eq(0)
end
end
end
So what about the actual WeatherQuery code? It changes very little as well:
# redis_weather_query.rb
require 'net/http'
require 'json'
require 'timeout'
# require the new cache module
require_relative 'redis_weather_cache'
module RedisWeatherQuery
extend self
class NetworkError < StandardError
end
# ... same as before ...
def clear!
@history = []
@api_request_count = 0
# no more clearing of cache here
end
private
# ... same as before ...
# the new cache module has a Hash-like interface
def cache
RedisWeatherCache
end
# ... same as before ...
end
We can see that we've preserved pretty much the same code and specs as before. Almost all of the new functionality is accomplished in a new module that caches with Redis. Here is what it looks like:
# redis_weather_cache.rb
require 'redis'
module RedisWeatherCache
extend self
CACHE_KEY = 'weather_query:cache'
EXPIRY_ZSET_KEY = 'weather_query:expiry_tracker'
EXPIRE_FORECAST_AFTER = 300 # 5 minutes
def redis_client
if ! defined?(::REDIS_CLIENT)
raise("No REDIS_CLIENT defined!")
end
::REDIS_CLIENT
end
def []=(location, forecast)
redis_client.hset(CACHE_KEY, location, JSON.generate(forecast))
redis_client.zadd(EXPIRY_ZSET_KEY, Time.now.to_i, location)
end
def [](location)
remove_expired_entries
raw_value = redis_client.hget(CACHE_KEY, location)
if raw_value
JSON.parse(raw_value)
else
nil
end
end
def all
redis_client.hgetall(CACHE_KEY).inject({}) do |memo, (location, forecast_json)|
memo[location] = JSON.parse(forecast_json)
memo
end
end
def clear!
redis_client.del(CACHE_KEY)
end
def remove_expired_entries
# expired locations have a score, i.e. creation timestamp, less than a certain threshold
expired_locations = redis_client.zrangebyscore(EXPIRY_ZSET_KEY, 0, Time.now.to_i - EXPIRE_FORECAST_AFTER)
if ! expired_locations.empty?
# remove the cache entry
redis_client.hdel(CACHE_KEY, expired_locations)
# also clear the expiry entry
redis_client.zrem(EXPIRY_ZSET_KEY, expired_locations)
end
end
end
We'll avoid a detailed explanation of this code. We simply note that we accomplish all of the design goals we discussed at the beginning of the section: a persistent cache with expiration of individual values. We've accomplished this using some simple Redis functionality along with ZSET or sorted set functionality, which is a bit more complex, and which we needed because Redis does not allow individual entries in a Hash to be deleted. We can see that by using method names such as RedisWeatherCache.[] and RedisWeatherCache.[]=, we've maintained a Hash-like interface, which made it easy to use this cache instead of the simple in-memory Ruby Hash we had in our previous iteration. Our tests all pass and are still pretty simple, thanks to the modularity of this new cache code, the modular configuration file, and the previous fixes we made to our specs to remove Internet and run-order dependencies.
In this article, we delved into setting up and cleaning up state for real-world specs that interact with external services and local resources by extending our WeatherQuery example to address a big bug, isolate our specs from the Internet, and cleanly configure a Redis database to serve as a better cache.
Further resources on this subject: