Distributing Python code across multiple cores with IPython

Despite CPython's GIL, it is possible to execute several tasks in parallel on multi-core computers using multiple processes instead of multiple threads. Python offers a native multiprocessing module. IPython's parallel extension, called ipyparallel, offers an even simpler interface that brings powerful parallel computing features in an interactive environment. We will describe this tool here.

Getting started

You need to install ipyparallel with conda install ipyparallel.

Then, you need to activate the ipyparallel Jupyter extension with ipcluster nbextension enable --user.

How to do it...

1. First, we launch four IPython engines in separate processes. We have basically two options to do this:

   • Executing ipcluster start -n 4 in a system shell

   • Using the web interface provided in Jupyter Notebook's main page by clicking on the IPython Clusters tab and launching four engines

2. Then, we create a client that will act as a proxy to the IPython engines...