The Importance of Integration and Data Management for Big Data

Transcript

1. Classification: Public The Importance of Integration and Data Management for Big Data September 2022 Erik Assink, Managing Director Yenlo North America
2. Big Data Today and Tomorrow • Big Data provides immense opportunities that businesses tap into this via analytics and data mining. (Value) • The world created 1.2 zettabytes (1021 bytes) of new data in 2010. By 2025, we can expect 175 zettabytes or more (Volume). • As data sources proliferate, and real-time BigData becomes more prevalent (Variety, Velocity), the need for modern integration & data management will increase. Business first. Technology doesn't matter. 2 Every company has big data in its future and every company will eventually be in the data business. Thomas H. Davenport
3. There is Value of Big Data across all Sectors And across the Value Chain • Product Development > Anticipate Demand, Success • Predictive Maintenance > Predict Failure, Root Cause • Customer Experience > Personalized Experience • Fraud and Compliance > Detect Fraud Patterns • Operational Efficiency > Better Decisions • Machine Learning > Teach Machines • Innovation > New Insights, Interdependencies Business first. Technology doesn't matter. 3 How to adopt use cases efficiently?
4. Big Data 3 V’s drive need for Integration & Data Management Gartner defined the original “three V’s” in 2001. Additional “V’s” such as “Value”, were added later. Business first. Technology doesn't matter. 4
5. Accelerating Increase of IoT Devices Business first. Technology doesn't matter. 5 IoT and non-IoT Devices worldwide How to adopt use cases efficiently?
6. Use Case example Transit Sector Solution • Infra: Devices track position, speed and condition of equipment. IoT sensors detects passengers in transit vehicles. • Integration & Data: API based integrations, connect IoT, systems, and other sources, to make data available for consumption in real time. Big- Data & Analytics consumes data and provides meaningful insights to improve customer experience and operational efficiency. • Services: Mobile Route Planning, and Equipment Scheduling. Business first. Technology doesn't matter. 6 Objectives • Optimize transit schedules and equipment to improve customer experience, equipment utilization, and maintenance.
7. Use Case examples in the Utilities Sector • Maximize the use of Renewable Power Sources in a ”Smart Grid” • Optimize Short Term Energy Trades • Use residual heat generated by gas plants, and decide when to shut down, or fire up. • Improve Customer Insights by Collecting and Aggregating Consumer Data • Detect malfunctioning equipment within households based on usage patterns. Business first. Technology doesn't matter. 7 How to adopt use cases efficiently?
8. Simple Reference Model Business first. Technology doesn't matter. 8 Effort / Time Performance / Value The Data layer is a prerequisite for the Services layer Data is produced Data is managed Data is consumed & Devices Integration &Data
9. Integration & Big Data Combined Example: Google Maps Business first. Technology doesn't matter. 9 Traditional/Analog • Rigid and slower • Manual coordination Digitally Enabled • Agile • Scalable • Lower Cost Integration & Data & Devices
10. Google Maps Architecture Business first. Technology doesn't matter. 10
11. From: Point to Point Integration
12. To: API Driven Integration Business first. Technology doesn't matter. 12
13. Integration Platform Enable RAPID implementation of ANY IoT and BIG DATA use case in a re reusable way: • Agile .. Allow to adapt to ever changing business conditions, customer needs, business, to in the end quickly enable new Value with smarter, safer, and better services, and better insights. • Open & Real Time .. Connect and collect data from a large Variety of ‘things’ and ‘data sources’ using open standards and APIs, process and analyze large Volumes of data with high Velocity in real-time. Now lets combine this into a uniform Integration & BigData architecture.
14. Data Storage & Sources Data Integration (Mapping, Transformation, Routing, and Security) Streaming Analytics ERP DW Big Data Message Queue Integration Engine Identity & Access Management OAUTH, SAML REST, JSON, SFTP MQTT, JMS, gRPC, Avro Cassandra HBase MongoDB SAP Oracle Integration & Big Data Architecture SaaS Kafka ActiveMQ RabbitMQ Redshift Synapse Teradata Databricks API Mgmt / Gateway Data Analytics & Visualization REST, JSON Data Ingestion Open Data Message Bus Data Sources (Edge, IoT, Scraping) Video, GPS, RFID, Sensors, Beacons, Machines, Weblogs, Social Media Web-Sockets, AMQP, MQTT, DDS, REST, SOAP, JMS, HTTP REST, JSON, ETL, ELT, CDC REST, JSON, GraphQL Engine Connectors NetSuite WorkDay SalesForce MapReduce Spark Storm Hive Sqoop Data Processing MapReduce Spark Hive Sqoop MapReduce Spark Storm Hive Sqoop Scala R MLib Grafana Tableau Light Weight APIs
15. Integration Platform Infra Requirements Business first. Technology doesn't matter. 15 Integration Management Platform High Availability We are in business 24/7, of course it must be high- available. Scalability I am not sure when to receive peak loads, the system should be able to deal with it instantly. 24/7 Support When something goes wrong, I need it resolved fast. Security Data and processes is crucial to us. We need the highest level of security matters possible. Short Time-to-Market Go-live in a matter of weeks instead of months. Flexibility (or agility) I know one thing for sure: tomorrow the whole world changed again. Guaranteed Quality The quality of the solution should be consistent, we always need to rely on it. Affordability I don’t have a huge budget; the investment must be low and predictable.
16. Classification: Public Questions Contact: Erik Assink Erik.Assink@yenlo.com