IPython Interactive Computing and Visualization Cookbook

This is a special chapter about good practices in interactive computing. If the rest of the book is about the content, then this chapter is about the form. It describes how to work efficiently and properly with the tools this book is about. We will cover the essentials of the version control system Git before tackling reproducible computing experiments (notably with the IPython notebook).

We will also cover more general topics in software development, such as code quality, debugging, and testing. Attention to these subjects can greatly improve the quality of our end products (for example, software, research, and publications). We will only scratch the surface here, but you will find many references to learn more about these important topics.

In this first recipe, we will briefly cover a transverse and kind of a prosaic subject: Python 2 or Python 3?

Python 3 has been available since 2008, but many Python users are still stuck with Python 2. By improving many aspects of Python 2, Python 3 has broken compatibility with the previous branch. Migrating to Python 3 may therefore require a significant investment.

Even if there aren't that many compatibility-breaking changes, a program that works perfectly fine in Python 2 may not work at all in Python 3. For example, your very first Hello World Python 2 program doesn't work anymore in Python 3; print "Hello World!" raises a SyntaxError in Python 3. Indeed, print is now a function rather than a statement. You should write print("Hello World!"), which also works fine in Python 2.

Whether you start a new project or need to maintain an old Python library, the question of choosing between Python 2 and Python 3 arises. Here, we give some arguments...

There are multiple ways of using IPython for interactive computing. Some of them are better in terms of flexibility, modularity, reusability, and reproducibility. We will review and discuss them in this recipe.

Any interactive computing workflow is based on the following cycle:

This fundamental loop (also known as Read-Eval-Print Loop or REPL) is particularly useful when doing exploratory research on data or model simulations, or when building a complex algorithm step by step. A more classical workflow (the edit-compile-run-debug loop) would consist of writing a full-blown program, and then performing a complete analysis. This is generally more tedious. It is more common to build an algorithmic solution iteratively, by doing small-scale experiments and tweaking the parameters, and this is precisely what interactive computing is about.

Integrated Development Environments (IDEs), providing...

Using a distributed version control system is so natural nowadays that if you are reading this book, you are probably already using one. However, if you aren't, read this recipe carefully. You should always use a version control system for your code.

A distributed version control system such as Git is designed for complex and nonlinear workflows typical in interactive computing and exploratory research. A central concept is branching, which we will discuss in this recipe.

In this recipe, we present ten tips that can help you conduct efficient and reproducible interactive computing experiments. These are more guidelines than absolute rules.

First, we will show how you can improve your productivity by minimizing the time spent doing repetitive tasks and maximizing the time spent thinking about your core work.

Second, we will demonstrate how you can achieve more reproducibility in your computing work. Notably, academic research requires experiments to be reproducible so that any result or conclusion can be verified independently by other researchers. It is not uncommon for errors or manipulations in methods to result in erroneous conclusions that can have damaging consequences. For example, in the 2010 research paper in economics Growth in a Time of Debt, by Carmen Reinhart and Kenneth Rogoff, computational errors were partly responsible for a flawed study with global ramifications for policy...

Writing code is easy. Writing high-quality code is much harder. Quality is to be understood both in terms of actual code (variable names, comments, docstrings, and so on) and architecture (functions, modules, and classes). In general, coming up with a well-designed code architecture is much more challenging than the implementation itself.

In this recipe, we will give a few tips about how to write high-quality code. This is a particularly important topic in academia, as more and more scientists without prior experience in software development need to program.

The references given at the end of this recipe contain much more details than what we could mention here.

Manual testing is essential to ensuring that our software works as expected and does not contain critical bugs. However, manual testing is severely limited because bugs may be introduced every time a change is made in the code. We can't possibly expect to manually test our entire program at every commit.

Nowadays, automated testing is a standard practice in software engineering. In this recipe, we will briefly cover important aspects of automated testing: unit tests, test-driven development, test coverage, and continuous integration. Following these practices is absolutely necessary in order to produce high-quality software.

Debugging is an integral part of software development and interactive computing. A widespread debugging technique consists of placing print statements in various places in the code. Who hasn't done this? It is probably the simplest solution, but it is certainly not the most efficient (it's the poor man's debugger).

IPython is perfectly adapted for debugging, and the integrated debugger is quite easy to use (actually, IPython merely offers a nice interface to the native Python debugger pdb). In particular, tab completion works in the IPython debugger. This recipe describes how to debug code with IPython.