Chapter 9. Social Data Analytics at Scale – Spark and Amazon Web Services
In the age of big data we have to handle new problems for data handling that did not exist before in terms of the three Vs (volume, variety, and velocity). When we handle very large amounts of data, we have to change our approach entirely. For example, algorithms can no longer use exhaustive brute force, because this approach might just take years to complete. Instead, we would use intelligent filtering to reduce the search space. Another example is when we have very high dimensions; for example, in text analysis, where every word or combination of words in the vocabulary constitute a dimension we need to change algorithms to adapt to such scenarios.
Advances in cluster computing have given us a new tool to handle the challenge of big data. No longer do we think of performing an analysis on a single node (your computer), we have progressed to thinking in terms of clusters of resources. Of course, cluster systems existed...
