Chapter 10. Introducing Lambda Architecture

Enterprises have come a long way where there's a demand for a unified system that meets both batch and real-time data processing needs. To extend it further, the need is for a distributed, scalable, highly-available, and fault-tolerant Big Data enterprise system, which is capable of presenting unified views/insights from the batch as well as real-time data systems. Though architects/developers have been working on discrete systems where batch and real-time use cases were developed and deployed separately, presenting this as a single view to the users was a real challenge. Perceiving the objective of a unified view has its own challenges. In some places where it was realized using leveraged traditional architectural patterns, it made the whole system too complex and, in some cases, made the whole system almost unmanageable.

In order to meet the needs and challenges of enterprises for presenting a unified view, which is combination of batch and real...