Core functionality

Let's start by creating a directory for your module. Inside of the project's root directory, create a subdirectory called weatherterm. The subdirectory weatherterm is where our module will live. The module directory needs two subdirectories - core and parsers. The project's directory structure should look like this:

weatherterm
├── phantomjs
└── weatherterm
    ├── core
    ├── parsers   

Loading parsers dynamically

This application is intended to be flexible and allow developers to create different parsers for different weather websites. We are going to create a parser loader that will dynamically discover files inside of the parsers directory, load them, and make them available to be used by the application without requiring changes to any other parts of the code. Here are the rules that our loader will require when implementing new parsers:

• Create a file with a class implementing the methods for fetching the current weather forecast as well as five-day, ten-day, and weekend weather forecasts
• The file name has to end with parser, for example, weather_com_parser.py
• The file name can't start with double underscores

With that said, let's go ahead and create the parser loader. Create a file namedparser_loader.py inside of the weatherterm/core directory and add the following content:

import os
import re
import inspect


def _get_parser_list(dirname):
    files = [f.replace('.py', '')
             for f in os.listdir(dirname)
             if not f.startswith('__')]

    return files


def _import_parsers(parserfiles):

    m = re.compile('.+parser$', re.I)

    _modules = __import__('weatherterm.parsers',
                          globals(),
                          locals(),
                          parserfiles,
                          0)

    _parsers = [(k, v) for k, v in inspect.getmembers(_modules)
                if inspect.ismodule(v) and m.match(k)]

    _classes = dict()

    for k, v in _parsers:
        _classes.update({k: v for k, v in inspect.getmembers(v)
                         if inspect.isclass(v) and m.match(k)})

    return _classes


def load(dirname):
    parserfiles = _get_parser_list(dirname)
    return _import_parsers(parserfiles)

First, the _get_parser_list function is executed and returns a list of all files located in weatherterm/parsers; it will filter the files based on the rules of the parser described previously. After returning a list of files, it is time to import the module. This is done by the _import_parsers function, which first imports the weatherterm.parsers module and makes use of the inspect package in the standard library to find the parser classes within the module.

The inspect.getmembers function returns a list of tuples where the first item is a key representing a property in the module, and the second item is the value, which can be of any type. In our scenario, we are interested in a property with a key ending with parser and with the value of type class.

Assuming that we already have a parser in place in the weatherterm/parsers  directory, the value returned by the inspect.getmembers(_modules) will look something like this:

[('WeatherComParser',
  <class 'weatherterm.parsers.weather_com_parser.WeatherComParser'>),
  ...]

Note

inspect.getmembers(_module) returns many more items, but they have been omitted since it is not relevant to show all of them at this point.

Lastly, we loop through the items in the module and extract the parser classes, returning a dictionary containing the name of the class and the class object that will be later used to create instances of the parser.

Creating the application's model

Let's start creating the model that will represent all the information that our application will scrape from the weather website. The first item we are going to add is an enumeration to represent each option of the weather forecast we will provide to the users of our application. Create a file named forecast_type.py in the directory weatherterm/core with the following contents:

from enum import Enum, unique


@unique
class ForecastType(Enum):
    TODAY = 'today'
    FIVEDAYS = '5day'
    TENDAYS = '10day'
    WEEKEND = 'weekend'

Enumerations have been in Python's standard library since version 3.4 and they can be created using the syntax for creating classes. Just create a class inheriting from enum.Enum containing a set of unique properties set to constant values. Here, we have values for the four types of forecast that the application will provide, and where values such as ForecastType.TODAY, ForecastType.WEEKEND, and so on can be accessed.

Note that we are assigning constant values that are different from the property item of the enumeration, the reason being that later these values will be used to build the URL to make requests to the weather website.

The application needs one more enumeration to represent the temperature units that the user will be able to choose from in the command line. This enumeration will contain Celsius and Fahrenheit items. 

First, let's include a base enumeration. Create a file called base_enum.py in the weatherterm/core directory with the following contents:

from enum import Enum


class BaseEnum(Enum):
    def _generate_next_value_(name, start, count, last_value):
        return name

 BaseEnum is a very simple class inheriting from Enum . The only thing we want to do here is override the method _generate_next_value_ so that every enumeration that inherits from BaseEnum and has properties with the value set to auto()  will automatically get the same value as the property name.

Now, we can create an enumeration for the temperature units. Create a file called unit.py in the weatherterm/core directory with the following content:

from enum import auto, unique

from .base_enum import BaseEnum


@unique
class Unit(BaseEnum):
    CELSIUS = auto()
    FAHRENHEIT = auto()

This class inherits from the BaseEnum that we just created, and every property is set to auto(), meaning the value for every item in the enumeration will be set automatically for us. Since the Unit class inherits from BaseEnum, every time the auto() is called, the _generate_next_value_ method on BaseEnum will be invoked and will return the name of the property itself.

Before we try this out, let's create a file called __init__.py in the weatherterm/core directory and import the enumeration that we just created, like so:

from .unit import Unit

If we load this class in the Python REPL and check the values, the following will occur:

Python 3.6.2 (default, Sep 11 2017, 22:31:28) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from weatherterm.core import Unit
>>> [value for key, value in Unit.__members__.items()]
[<Unit.CELSIUS: 'CELSIUS'>, <Unit.FAHRENHEIT: 'FAHRENHEIT'>]

Another item that we also want to add to the core module of our application is a class to represent the weather forecast data that the parser returns. Let's go ahead and create a file named forecast.py in the weatherterm/core directory with the following contents:

from datetime import date

from .forecast_type import ForecastType


class Forecast:
    def __init__(
            self,
            current_temp,
            humidity,
            wind,
            high_temp=None,
            low_temp=None,
            description='',
            forecast_date=None,
            forecast_type=ForecastType.TODAY):
        self._current_temp = current_temp
        self._high_temp = high_temp
        self._low_temp = low_temp
        self._humidity = humidity
        self._wind = wind
        self._description = description
        self._forecast_type = forecast_type

        if forecast_date is None:
            self.forecast_date = date.today()
        else:
            self._forecast_date = forecast_date

    @property
    def forecast_date(self):
        return self._forecast_date

    @forecast_date.setter
    def forecast_date(self, forecast_date):
        self._forecast_date = forecast_date.strftime("%a %b %d")

    @property
    def current_temp(self):
        return self._current_temp

    @property
    def humidity(self):
        return self._humidity

    @property
    def wind(self):
        return self._wind

    @property
    def description(self):
        return self._description

    def __str__(self):
        temperature = None
        offset = ' ' * 4

        if self._forecast_type == ForecastType.TODAY:
            temperature = (f'{offset}{self._current_temp}\xb0\n'
                           f'{offset}High {self._high_temp}\xb0 / '
                           f'Low {self._low_temp}\xb0 ')
        else:
            temperature = (f'{offset}High {self._high_temp}\xb0 / '
                           f'Low {self._low_temp}\xb0 ')

        return(f'>> {self.forecast_date}\n'
               f'{temperature}'
               f'({self._description})\n'
               f'{offset}Wind: '
               f'{self._wind} / Humidity: {self._humidity}\n')

In the Forecast class, we will define properties for all the data we are going to parse: 

current_temp	Represents the current temperature. It will only be available when getting today's weather forecast.
humidity	The humidity percentage for the day.
wind	Information about today's current wind levels.
high_temp	The highest temperature for the day.
low_temp	The lowest temperature for the day.
description	A description of the weather conditions, for example, Partly Cloudy.
forecast_date	Forecast date; if not supplied, it will be set to the current date.
forecast_type	Any value in the enumeration ForecastType (today, fivedays, tendays, or weekend).

 

We can also implement two methods called forecast_date  with the decorators @property  and @forecast_date.setter . The @property  decorator will turn the method into a getter for the _forecast_date property of the Forecast class, and the @forecast_date.setter will turn the method into a setter.  The setter was defined here because, every time we need to set the date in an instance of Forecast, we need to make sure that it will be formatted accordingly. In the setter, we call the strftime method, passing the format codes %a (weekday abbreviated name), %b (monthly abbreviated name), and %d (day of the month).

Note

The format codes %a and %b will use the locale configured in the machine that the code is running on.

Lastly, we override the __str__ method to allow us to format the output the way we would like when using the print, format, and str functions.

By default, the temperature unit used by weather.com is Fahrenheit, and we want to give the users of our application the option to use Celsius instead. So, let's go ahead and create one more file in the weatherterm/core directory called unit_converter.py with the following content:

from .unit import Unit


class UnitConverter:
    def __init__(self, parser_default_unit, dest_unit=None):
        self._parser_default_unit = parser_default_unit
        self.dest_unit = dest_unit

        self._convert_functions = {
            Unit.CELSIUS: self._to_celsius,
            Unit.FAHRENHEIT: self._to_fahrenheit,
        }

    @property
def dest_unit(self):
        return self._dest_unit

    @dest_unit.setter
    def dest_unit(self, dest_unit):
        self._dest_unit = dest_unit

    def convert(self, temp):

        try:
            temperature = float(temp)
        except ValueError:
            return 0

        if (self.dest_unit == self._parser_default_unit or
                self.dest_unit is None):
            return self._format_results(temperature)

        func = self._convert_functions[self.dest_unit]
        result = func(temperature)

        return self._format_results(result)

    def _format_results(self, value):
        return int(value) if value.is_integer() else f'{value:.1f}'

    def _to_celsius(self, fahrenheit_temp):
        result = (fahrenheit_temp - 32) * 5/9
        return result

    def _to_fahrenheit(self, celsius_temp):
        result = (celsius_temp * 9/5) + 32
        return result

This is the class that is going to make the temperature conversions from Celsius to Fahrenheit and vice versa. The initializer of this class gets two arguments; the default unit used by the parser and the destination unit. In the initializer, we will define a dictionary containing the functions that will be used for temperature unit conversion.

The convert method only gets one argument, the temperature. Here, the temperature is a string, so the first thing we need to do is try converting it to a float value; if it fails, it will return a zero value right away.

You can also verify whether the destination unit is the same as the parser's default unit or not. In that case, we don't need to continue and perform any conversion; we simply format the value and return it.

If we need to perform a conversion, we can look up the _convert_functions  dictionary to find the conversion function that we need to run. If we find the function we are looking for, we invoke it and return the formatted value.

The code snippet below shows the _format_results method, which is a utility method that will format the temperature value for us:

return int(value) if value.is_integer() else f'{value:.1f}'

The _format_results method checks if the number is an integer; the value.is_integer() will return True if the number is, for example, 10.0. If True, we will use the int function to convert the value to 10; otherwise, the value is returned as a fixed-point number with a precision of 1. The default precision in Python is 6. Lastly, there are two utility methods that perform the temperature conversions, _to_celsius and _to_fahrenheit.

Now, we only need to edit the __init__.py file in the weatherterm/core directory and include the following import statements:

from .base_enum import BaseEnum
from .unit_converter import UnitConverter
from .forecast_type import ForecastType
from .forecast import Forecast

Fetching data from the weather website

We are going to add a class named Request that will be responsible for getting the data from the weather website. Let's add a file named request.py in the weatherterm/core directory with the following content:

import os
from selenium import webdriver


class Request:
    def __init__(self, base_url):
        self._phantomjs_path = os.path.join(os.curdir,
                                          'phantomjs/bin/phantomjs')
        self._base_url = base_url
        self._driver = webdriver.PhantomJS(self._phantomjs_path)

    def fetch_data(self, forecast, area):
        url = self._base_url.format(forecast=forecast, area=area)
        self._driver.get(url)

        if self._driver.title == '404 Not Found':
            error_message = ('Could not find the area that you '
                             'searching for')
            raise Exception(error_message)

        return self._driver.page_source

This class is very simple; the initializer defines the base URL and creates a PhantomJS driver, using the path where PhantomJS is installed. The fetch_data method formats the URL, adding the forecast option and the area. After that, the webdriver performs a request and returns the page source. If the title of the markup returned is 404 Not Found, it will raise an exception. Unfortunately, Selenium doesn't provide a proper way of getting the HTTP Status code; this would have been much better than comparing strings.

Note

You may notice that I prefix some of the class properties with an underscore sign. I usually do that to show that the underlying property is private and shouldn't be set outside the class. In Python, there is no need to do that because there's no way to set private or public properties; however, I like it because I can clearly show my intent.

Now, we can import it in the __init__.py  file in the weatherterm/core directory:

from .request import Request

Now we have a parser loader to load any parser that we drop into the directory weatherterm/parsers,  we have a class representing the forecast model, and an enumeration ForecastType so we can specify which type of forecast we are parsing. The enumeration represents temperature units and utility functions to convert temperatures from Fahrenheit to Celsius and Celsius to Fahrenheit. So now, we should be ready to create the application's entry point to receive all the arguments passed by the user, run the parser, and present the data on the terminal.