Other computational frameworks

We've frequently discussed the myriad possibilities brought to the Hadoop platform by YARN. We went into details of two new models, Samza and Spark. Additionally, other more established frameworks such as Pig are also being ported to the framework.

To give a view of the much bigger picture in this section, we will illustrate the breadth of processing possible using YARN by presenting a set of computational models that are currently being ported to Hadoop on top of YARN.

Apache Storm

Storm (http://storm.apache.org) is a distributed computation framework written (mainly) in the Clojure programming language. It uses custom-created spouts and bolts to define information sources and manipulations to allow distributed processing of streaming data. A Storm application is designed as a topology of interfaces that creates a stream of transformations. It provides similar functionality to a MapReduce job with the exception that the topology will theoretically run indefinitely...