IoT/Serverless/Event Driven Architecture for manufacturing industry: Case Study

The case study reflects a business problem a fictious manufacturing customer is facing to solve by enabling cloud technology.

Business Problem:

• EarthBuild is a fictional heavy equipment manufacturing company operates mainly in two industries — construction (80%) and building automation(20%). They are also planning on launching new industry sub-verticals in precision tools, home and building security.
• Currently they have 1000 dealers in 50 countries. Supports 30 million EarthBuild equipment and devices.
• Approximately 60,000 (2%) vehicles are connected to cellular network to collect data directly through maintenance port and upload to FTP server with 100 fields of data for 18 hours a day. Total 15 TB of data collected a day.
• Because data capture, staging, and processing taking long time, resulting in the aggregated report generation is based on 3 weeks old data.
• Due to stale data, EarthBuild preemptively stock vehicle parts to reduce downtime by 50%. However, some customers are without their vehicles for up to 5 weeks. This would also lead to inventory cost due to parts procurement in advance.
• Currently they have single data center in the U.S. west coast is a concern in terms of disaster recovery and availability requirements for their business continuity to meets RPO and RTOs.

Current State and Challenges:

• ETL & COTS applications are running on Windows Servers
• PostgreSQL is running on Redhat Linux Server, RAID 0, HDD
• Single Data center (SPOF)
• Data from vehicles, uploaded in compressed csv files to FTP server (Possible network latency from cell towers to data center)
• Python application running on ETL server extracts the compressed gzip csv files , read them and load them into data warehouse.
• Not all vehicles data is available in order to determine which vehicle needs service. EarthBuild’s inability to collect and process data timely is leading to vehicle down time.
• Current vehicle connectivity (2% of total vehicles) is not enough to serve dealers and their customers on time without data.
• Building automation business is fast growing, EarthBuild’s ability to scale compute resources for data processing is limited. Also they need to do capacity planning for future business growth in precision tools, home and security fields.

Current Process:

• Vehicles data is uploaded to FTP server via maintenance port
• ETL (Python) process extracts, read csv data and load them to data warehouse (PostgreSQL)
• COTS application connects to data warehouse used for analysis and report generation.

Solution:

• The core problem with stale data is two folds
• Connectivity to 30 million vehicles
• Process the data as it arrives for real time streaming and analytics
• Ability to process late arriving data because of slow network connectivity
• Increase the network connectivity to 30 mil. vehicles
• Use IoT Core GCP service for device management, registration, authentication
• Use cloud pub/sub for messaging
• Use cloud data flow for real-time stream processing
• Use cloud bigquery for analysis, aggregation, data life cycle management via partitioning.
• Use cloud storage for pre-processed aggregate reports, long-term storage
• Use signed URLs to send vehicle data to dealers.
• Increase the availability of cloud resources to multi-regions for HA and low latency.

Future State Architecture: