Continuous integration for benchmarks
Once we know how to benchmark (and I will use the nightly way from now on), we can set up our continuous integration environment so that we can get alerts when a performance regression occurs. There are multiple ways of achieving something like this, but I will be using the Travis-CI infrastructure, some Bash, and a Rust library to do it.
Travis-CI integration
Let's first start by thanking the great work of Lloyd Chan and Sunjay Varma, who were the first to suggest this approach. You can find the code we will be using in Sunjay's blog (http://sunjay.ca/2017/04/27/rust-benchmark-comparison-travis). Nevertheless, it makes sense to check it, understand it, and see how it works.
The idea is simple: on Travis-CI builds, you can build against multiple Rust channels. When a pull request is received when building against the nightly channel, let's run all the benchmarks and then compare them to benchmarks we will run on the pull request target branch. Finally,...
