Big Data Spain

17th ~ 18th NOV 2016 MADRID, SPAIN #BDS16

Solving the Industry 4.0 challenges on the logistics domain using Apache Mesos

Thursday 17th

from 16:30 to 17:10

Theatre 19

-

Keynote

ULMA Handling Systems (UHS) develops an extensive range of integrated logistics solutions including automated handling systems for warehouses worldwide in most sectors of the market (such as Distribution, Food and beverage, Industrial, Textile and Storage). UHS has got a huge knowledge in the world of the intra-logistics with a broad experience (27 years in this sector) and offers an integrated service from design and planning of logistics solutions to post-sales service and reengineering, providing all-round logistics systems consisting of design, development, assembly and maintenance of automatic logistics systems.

Each handling facility such as cranes, conveyors, sorting systems, picking systems, rolling tables, lifts, and intermediate storage forms a complex physical unit. All together are deployed to one handling system, e.g. automatic storage. A handling system always includes a Supervisor System (SS) that interacts with the diverse types of physical units, network equipment and other warehouse management systems in order to request information about operational data and malfunctions produced within these complex environments. Malfunctions typically appear in the interaction of the control for different elements and these can lead to significant downtimes and reduction of availability and overall performance of the storage facility.

In their bid to improve the efficiency, performance and to reduce maintenance downtime to their final clients, a new Handling System Cloud Supervisor (HSCS) been developed. For each handling system deployed in storage warehouses worldwide, there is a SS that gathers its operational data and sends them to a digital platform running on the cloud. Furthermore, there is a live clone HSCS running on the developed cloud platform for each SS. The HSCS clones should maintain the state of the local SSs and should be available all the time. The availability of the HSCS is assured by using Apache Mesos.

Apache Mesos, an open source cluster manager is used as platform kernel as it will assure that our clone supervisors are running using a framework called Marathon. Each of the real SSs communicates with their clones HSCSs using a reverse proxy called Træfɪk, a modern reverse proxy with native integration with Mesos. When a clone supervisor generates a report or receives an alert of the real supervisor, a message is sent to a distributed message broker called Kafka. Then, those messages are ingested using Spark Streaming and saved to an HDFS using Parquet as file format. Furthermore, Spark Streaming is also employed to retain recent data cached in memory for performance reasons. The platform uses Consul as dynamic DNS provider and for monitoring its services.

The developed platform can be deployed to different cloud providers using the automation tools Terraform and Ansible. These tools provide great automation capabilities on provisioning tasks.

The ingestion rate performance of the platform has been tested using a small cluster with an ingestion rate of 100,000 messages per second. This approach, allows to UHS to query Spark SQL over the ingested data using standard BI tools, and in the future be able to offer new services based on data analytics.

As conclusion, we can see that an integrated platform like Mesos enables UHS to confront the Industry 4.0 challenges with reduced costs. Moreover, Mesos and Marathon assure the availability of whole the system. As future work, we would like to migrate to Cassandra to avoid the HDFS compaction problem. Moreover, the migration to the open source version of DCOS is planned.

Angel Conde foto

Angel Conde

IK4-IKERLAN DevOps & Data Engineer