Getting Twitter data

In this article by Paulo A Pereira, the author of Elixir Cookbook, we will build an application that will query the Twitter timeline for a given word and will display any new tweet with that keyword in real time. We will be using an Elixir twitter client extwitter as well as an Erlang application to deal with OAuth. We will wrap all in a phoenix web application.

(For more resources related to this topic, see here.)

Getting ready

Before getting started, we need to register a new application with Twitter to get the API keys that will allow the authentication and use of Twitter's API. To do this, we will go to https://apps.twitter.com and click on the Create New App button. After following the steps, we will have access to four items that we need: consumer_key, consumer_secret, access_token, and access_token_secret.

These values can be used directly in the application or setup as environment variables in an initialization file for bash or zsh (if using Unix).

After getting the keys, we are ready to start building the application.

How to do it…

To begin with building the application, we need to follow these steps:

Create a new Phoenix application:

> mix phoenix.new phoenix_twitter_stream code/phoenix_twitter_stream

Add the dependencies in the mix.exs file:

defp deps do
  [
    {:phoenix, "~> 0.8.0"},
    {:cowboy, "~> 1.0"},
    {:oauth, github: "tim/erlang-oauth"},
    {:extwitter, "~> 0.1"}
  ]
end

Get the dependencies and compile them:
```
> mix deps.get && mix deps.compile
```

Configure the application to use the Twitter API keys by adding the configuration block with the keys we got from Twitter in the Getting ready section of this article. Edit lib/phoenix_twitter_stream.ex so that it looks like this:

defmodule PhoenixTweeterStream do
  use Application
  def start(_type, _args) do
    import Supervisor.Spec, warn: false
    ExTwitter.configure(
      consumer_key: System.get_env("SMM_TWITTER_CONSUMER_KEY"),
      consumer_secret: System.get_env("SMM_TWITTER_CONSUMER_SECRET"),
      access_token: System.get_env("SMM_TWITTER_ACCESS_TOKEN"),
      access_token_secret: System.get_env("SMM_TWITTER_ACCESS_TOKEN_SECRET")
    )
    children = [
      # Start the endpoint when the application starts
      worker(PhoenixTweeterStream.Endpoint, []),
      # Here you could define other workers and supervisors as children
      # worker(PhoenixTweeterStream.Worker, [arg1, arg2, arg3]),
    ]
    opts = [strategy: :one_for_one, name: PhoenixTweeterStream.Supervisor]
    Supervisor.start_link(children, opts)
  end
  def config_change(changed, _new, removed) do
    PhoenixTweeterStream.Endpoint.config_change(changed, removed)
    :ok
  end
end

In this case, the keys are stored as environment variables, so we use the System.get_env function:
System.get_env("SMM_TWITTER_CONSUMER_KEY") (…)
If you don't want to set the keys as environment variables, the keys can be directly declared as strings this way:
consumer_key: "this-is-an-example-key" (…)

Define a module that will handle the query for new tweets in the lib/phoenix_twitter_stream/tweet_streamer.ex file, and add the following code:

defmodule PhoenixTwitterStream.TweetStreamer do
  def start(socket, query) do
    stream = ExTwitter.stream_filter(track: query)
    for tweet <- stream do
      Phoenix.Channel.reply(socket, "tweet:stream", tweet)
    end
  end
end

Create the channel that will handle the tweets in the web/channels/tweets.ex file:

defmodule PhoenixTwitterStream.Channels.Tweets do
  use Phoenix.Channel
  alias PhoenixTwitterStream.TweetStreamer
  def join("tweets", %{"track" => query}, socket) do
    spawn(fn() -> TweetStreamer.start(socket, query) end)
    {:ok, socket}
  end 
end

Edit the application router (/web/router.ex) to register the websocket handler and the tweets channel. The file will look like this:

defmodule PhoenixTwitterStream.Router do
  use Phoenix.Router
  pipeline :browser do
    plug :accepts, ~w(html)
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
  end
  pipeline :api do
    plug :accepts, ~w(json)
  end
  socket "/ws" do
    channel "tweets", PhoenixTwitterStream.Channels.Tweets
  end
  scope "/", PhoenixTwitterStream do
    pipe_through :browser # Use the default browser stack
    get "/", PageController, :index
  end
end

Replace the index template (web/templates/page/index.html.eex) content with this:

<div class="row">
  <div class="col-lg-12">
    <ul id="tweets"></ul>
  </div>
  <script src="/js/phoenix.js" type="text/javascript"></script>
  <script src="https://code.jquery.com/jquery-2.1.1.js" type="text/javascript"></script>
  <script type="text/javascript">
    var my_track = "programming";
    var socket = new Phoenix.Socket("ws://" + location.host + "/ws");
    socket.join("tweets", {track: my_track}, function(chan){
      chan.on("tweet:stream", function(message){
        console.log(message);
        $('#tweets').prepend($('<li>').text(message.text));
        });
    });
  </script>
</div>

Start the application:
```
> mix phoenix.server
```
Go to http://localhost:4000/ and after a few seconds, tweets should start arriving and the page will be updated to display every new tweet at the top.

How it works…

We start by creating a Phoenix application. We could have created a simple application to output the tweets in the console. However, Phoenix is a great choice for our purposes, displaying a web page with tweets getting updated in real time via websockets!

In step 2, we add the dependencies needed to work with the Twitter API. We use parroty's extwitter Elixir application (https://hex.pm/packages/extwitter) and Tim's erlang-oauth application (https://github.com/tim/erlang-oauth/). After getting the dependencies and compiling them, we add the Twitter API keys to our application (step 4). These keys will be used to authenticate against Twitter where we previously registered our application.

In step 5, we define a function that, when started, will query Twitter for any tweets containing a specific query.

The stream = ExTwitter.stream_filter(track: query) line defines a stream that is returned by the ExTwitter application and is the result of filtering Twitter's timeline, extracting only the entries (tracks) that contain the defined query.

The next line, which is for tweet <- stream do Phoenix.Channel.reply(socket, "tweet:stream", tweet), is a stream comprehension. For every new entry in the stream defined previously, send the entry through a Phoenix channel.

Step 6 is where we define the channel. This channel is like a websocket handler. Actually, we define a join function:

 def join(socket, "stream", %{"track" => query}) do
   reply socket, "join", %{status: "connected"}
   spawn(fn() -> TweetStreamer.start(socket, query) end)
   {:ok, socket}
 end

It is here, when the websocket connection is performed, that we initialize the module defined in step 5 in the spawn call. This function receives a query string defined in the frontend code as track and passes that string to ExTwitter, which will use it as the filter.

In step 7, we register and mount the websocket handler in the router using use Phoenix.Router.Socket, mount: "/ws", and we define the channel and its handler module using channel "tweets", PhoenixTwitterStream.Channels.Tweets.

The channel definition must occur outside any scope definition!

If we tried to define it, say, right before get "/", PageController, :index, the compiler would issue an error message and the application wouldn't even start.

The last code we need to add is related to the frontend. In step 8, we mix HTML and JavaScript on the same file that will be responsible for displaying the root page and establishing the websocket connection with the server. We use a phoenix.js library helper (<script src="/js/phoenix.js" type="text/javascript"></script>), providing some functions to deal with Phoenix websockets and channels.

We will take a closer look at some of the code in the frontend:

// initializes the query … in this case filter the timeline for
// all tweets containing "programming" 
var my_track = "programming";
// initialize the websocket connection. The endpoint is /ws.  //(we already have registered with the phoenix router on step 7)
var socket = new Phoenix.Socket("ws://" + location.host + "/ws");
// in here we join the channel 'tweets'
// this code triggers the join function we saw on step 6
// when a new tweet arrives from the server via websocket
// connection it is prepended to the existing tweets in the page
socket.join("tweets", "stream", {track: my_track}, function(chan){
      chan.on("tweet:stream", function(message){
        $('#tweets').prepend($('<li>').text(message.text));
        });
    });