Waiting for content to be available in Selenium

A common problem with dynamic web pages is that even after the whole page has loaded, and hence the get() method in Selenium has returned, there still may be content that we need to access later as there are outstanding Ajax requests from the page that are still pending completion. An example of this is needing to click a button, but the button not being enabled until all data has been loaded asyncronously to the page after loading.

Take the following page as an example: http://the-internet.herokuapp.com/dynamic_loading/2. This page finishes loading very quickly and presents us with a Start button:

The Start button presented on screen

When pressing the button, we are presented with a progress bar for five seconds:

The status bar while waiting

And when this is completed, we are presented with Hello World!

After the page is completely rendered

Now suppose we want to scrape this page to get the content that is exposed only after the button is pressed...