Keynote | Technical

NRT data enrichment and ingestion with NiFi and Spark

Thursday 16th | 15:55 - 16:25 | Theatre 18

---

Keywords defining the session:

- NiFi

- Data enrichment

- Near real-time

In order to fulfill increasing needs of business to deliver rich and near real time data for various analysis of Web users behaviour, the main and complex Apache Spark process was replaced in most parts by more modular and reusable solution built on top of Apache NiFi. Due to this transformation a significant boost and stability of entire process was achieved along with decrease of resources consumption. Also a few approaches were made to ingest the data to Hive in small batches.