Packt+ | Advance your knowledge in tech

You're reading from Data Lake for Enterprises Lambda Architecture for building enterprise data systems

Product type Paperback

Published in May 2017

Publisher Packt

ISBN-13 9781787281349

Length 596 pages

Edition 1st Edition

Languages

Java

Tools

Hadoop

Concepts

Data Processing

Authors (3):

Mishra

John

Pankaj Misra

View More author details

Table of Contents (23) Chapters

Title Page

Credits

Foreword

About the Authors

About the Reviewers

www.PacktPub.com

Customer Feedback

Preface

Part 1 - Overview

Part 2 - Technical Building blocks of Data Lake

Part 3 - Bringing It All Together

1. Introduction to Data FREE CHAPTER

2. Comprehensive Concepts of a Data Lake

3. Lambda Architecture as a Pattern for Data Lake

4. Applied Lambda for Data Lake

5. Data Acquisition of Batch Data using Apache Sqoop

6. Data Acquisition of Stream Data using Apache Flume

7. Messaging Layer using Apache Kafka

8. Data Processing using Apache Flink

9. Data Store Using Apache Hadoop

10. Indexed Data Store using Elasticsearch

11. Data Lake Components Working Together

12. Data Lake Use Case Suggestions

Context for Data Lake - Data Storage and lambda Batch layer

In our Data Lake implementation, we have a dedicated layer where the data permanently resides and this is the Data Storage Layer. The data gathered from various sources is persisted in various stores capable of handling different types and forms of data. In this chapter, we are storing non-indexed raw data in our Data Lake.

We have chosen Apache Hadoop as our technology for this data storage capability. I am sure there was not much debate when we chose this technology in this layer, obviously because of the fantastic features this technology. Also, the level of maturity and support this technology possesses is quite astonishing over the short span of its existence.

The following sections of this chapter aim at covering Hadoop in detail so that you get a clear picture of this technology as well as get to know the data storage layer in detail.