Zpracování velkých dat logistiky v automotive
Loading...
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Tato práce řeší problém zpracování, transformace a přenosu dat z~platformy Splunk do data lake Cloudera Hadoop a následně do Power BI. Cílem práce je navrhnout a implementovat univerzální a přenositelnou aplikaci v~jazyce Python, která bude tento problém řešit. Na základě analýz možností komunikace výše zmíněných systémů je vytvořena univerzální aplikace, která se skládá z~několika Python skriptů. Univerzálnost a přenositelnost je zajištěna tím, že se pro jiný zdroj dat ze Splunku bude měnit pouze jeden konfigurační skript a ostatní zůstanou beze změny. Navržená aplikace byla nasazena do produkce a úspěšně řeší první use case pro sklad logistiky, který je v~této práci popsán.
The issue discussed in this work concerns processing, transforming and transferring of data from Splunk platform to Cloudera Hadoop data lake and then to Power BI. The main goal of this work is to design and implement a universal and transferable application in Python language which is supposed to solve this issue. The universal application consisting of several Python scripts is based on analyses of communication capabilities between the systems mentioned above. For any other source of Splunk type data, there is only one configuration script that needs to be changed, hence the needs of universality and transferability are met. The application was put into production and is now solving first use case in a logistic warehouse which is described in this work.
The issue discussed in this work concerns processing, transforming and transferring of data from Splunk platform to Cloudera Hadoop data lake and then to Power BI. The main goal of this work is to design and implement a universal and transferable application in Python language which is supposed to solve this issue. The universal application consisting of several Python scripts is based on analyses of communication capabilities between the systems mentioned above. For any other source of Splunk type data, there is only one configuration script that needs to be changed, hence the needs of universality and transferability are met. The application was put into production and is now solving first use case in a logistic warehouse which is described in this work.
Description
Subject(s)
zpracování, transformace a přenos velkých dat, big data, Splunk, Python, Cloudera Hadoop, Power BI, processing, transforming and transferring of big data, big data, Splunk, Python, Cloudera Hadoop, Power BI