Open Source Summit Japan & Automotive Linux Summit 2020

Around 2006, Apache Hadoop realized the open source based “Data Lake” architecture for enterprises to utilize large amounts of data, "Big Data". However, there are also growing expectations against "real-time analysis" that delivers analyzed results to end-users in seconds to minutes by immediately processing a large amount of “stream data”. In this talk, we present the history of open source software related to Data Lake, the overview of current software, and the potential tradeoffs.We also talk about how recent storage technologies, such as Apache Iceberg, Apache Hudi, Delta Lake, try to provide features to leverage both of historical and stream data on Data Lake in a different way from Lambda Architecture. Finally, we summarize these products based on the comparison of internal architectures. Attendances will learn about the overview of current storage software, and similarities and differencesof architectures. This helps you to design the system architecture build on Data Lake technologies to realize both batch and real-time based analysis. This post reflects some software upgrades from previous domestic presentation.