Installing Scrapy

Scrapy can be installed with the pip command, as follows:

pip install scrapy

Scrapy relies on some external libraries, so if you have trouble installing it there is additional information available on the official website at: http://doc.scrapy.org/en/latest/intro/install.html.

If Scrapy is installed correctly, a scrapy command will now be available in the terminal:

$ scrapy
    Scrapy 1.3.3 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
        bench    Run quick benchmark test
        commands 
        fetch    Fetch a URL using the Scrapy downloader
...

We will use the following commands in this chapter:

• startproject: Creates a new project
• genspider: Generates a new spider from a template
• crawl: Runs a spider
• shell: Starts the interactive scraping console

Note

For detailed information about these and other commands available, refer to http://doc.scrapy.org/en/latest/topics/commands.html