Scraping legality and scraping politely
There's no real code in this recipe. It's simply an exposition of some of the concepts related to the legal issues involved in scraping. I'm not a lawyer, so don't take anything I write here as legal advice. I'll just point out a few things you need to be concerned with when using a scraper.
Getting ready
The legality of scraping breaks down into two issues:
- Ownership of content
- Denial of service
Fundamentally, anything posted on the web is open for reading. Every time you load a page, any page, your browser downloads that content from the web server and visually presents it to you. So in a sense, you and your browser are already scraping anything you look at on the web. And by the nature of the web, because someone is posting content publicly on the web, they are inherently asking you to take that information, but often only for specific purposes.
The big issue comes with creating automated tools that directly look for and make copies of things on...
