Open Source Summit Japan & Automotive Linux Summit 2020

Around 2006, Apache Hadoop realized the open source based “Data Lake” architecture for enterprises to utilize large amounts of data, "Big Data". However, there are also growing expectations against "real-time analysis" that delivers analyzed results to end-users in seconds to minutes by immediately processing a large amount of “stream data”. In this talk, we present the history of open source software related to Data Lake, the overview of current software, and the potential tradeoffs.We also talk about how recent storage technologies, such as Apache Iceberg, Apache Hudi, Delta Lake, try to provide features to leverage both of historical and stream data on Data Lake in a different way from Lambda Architecture. Finally, we summarize these products based on the comparison of internal architectures. Attendances will learn about the overview of current storage software, and similarities and differencesof architectures. This helps you to design the system architecture build on Data Lake technologies to realize both batch and real-time based analysis. This post reflects some software upgrades from previous domestic presentation.

Giant Hogweed is a highly toxic plant originating in the Western Caucasus. It has spread across Central and Western Europe and there are sightings of Giant Hogweed reported from North America, too. Landowners are obliged to eradicate it, due to its toxicity and invasive nature in Europe. However, it is difficult for landowners to find and remove Giant Hogweed across large areas of land since it is a very cumbersome manual process. To automate the process of detecting the Giant Hogweed by exploiting technologies like drones and image recognition/detection using Machine Learning is an effective way to address this problem. In this presentation, we show you how we designed the architecture towards the Petabyte scale, how we took advantage of both of Big Data and Machine / Deep Learning technologies and lessons learned through this project. For example, we integrated a drone, Apache Hadoop, Apache Spark and TensorFlow to achieve the usability, flexibility and scalability for both of data engineers and data analysts. We talk about why this integration was needed for us, technical challenges from the viewpoint of enterprises and tips to leverage the above open source software.