Summary

Spark Streaming is based on a micro-batching model that is suitable for applications with throughput and high latency (> 0.5 seconds). Spark Streaming's DStream API provides transformations and actions for working with DStreams, including conventional transformations, window operations, output actions, and stateful operations such as updateStateByKey. Spark Streaming supports a variety of input sources and output sources used in the Big Data ecosystem. Spark Streaming supports the direct approach with Kafka, which really provides great benefits such as exactly once processing and avoiding WAL replication.

There are two types of failures in a Spark Streaming application; executor failure and driver failure. Executor failures are automatically taken care of by the Spark Streaming framework, but for handling driver failures, checkpointing and WAL must be enabled with high availability options for the driver such as --supervise.

Structured Streaming is a new paradigm shift in streaming...