Summary
RDDs are a fundamental unit of data in Spark and Spark programming revolves around creating and performing operations on RDDs such as transformations and actions. Apache Spark programs can be interactively executed in a shell or by submitting applications. Parallelism is defined by the number of partitions in an RDD. The number of partitions is decided by the number of blocks in the HDFS file, or type of resource manager and configuration properties used for non-HDFS files.
Caching RDDs in memory is useful for performing multiple actions on the same RDD as it provides higher performance. When an RDD is cached with the MEMORY_ONLY option, partitions that do not fit in memory will be re-computed as and when needed. If re-compute is expensive, it is better to choose MEMORY_AND_DISK as the persistence level.
Spark's application can be submitted in client or cluster mode. While client mode is used for development and testing, cluster mode is used for production deployment. Spark has three...
