are hidden in Historical Data on Big Data Platforms Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models Ø Event Processing uses these Models (without Redevelopment) to take Action in Real Time
+ Product Recommendation Picture Detection (Friends, Locations, Products) Machine Learning is already present in daily life… Now, every enterprise is beginning to leverage it! The Next Disruption: Google Beats Go Champion
Random Forest, Gradient Boosting Machine (GBM) • Method – Average many simple trees • Sample the data: fit a simple tree • Re-sample the data; up-weighting the observations that weren’t fitted well in previous model • Continue adding trees until fit is good • Save all the trees and average them • Better fit + prediction than single trees
Long-Term Competitive Advantage Value to the Organization A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Self-service Dashboards Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Visual Analytics Event Processing Analytics
Long-Term Competitive Advantage Value to the Organization Visual Analytics Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Analytics
Long-Term Competitive Advantage Value to the Organization Self-service Dashboards Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Visual Analytics Event Processing Analytics
to Proactive Actions Model Present Data Wrangling Signals Dashboards SAP Historian Production Well Filter Enrich Merge Shape Explore Clean Assemble Data Business Case Increase Productivity Grow Revenue Completions Visualize GeoLocation Production Value Theses Reduce Risk G&G Equipment Decision, Action Prediction Action Develop Model Pressure Temperature Production Interrupt Drill Bit Movement Equipment Failure
Long-Term Competitive Advantage Value to the Organization A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Self-service Dashboards Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity Visual Analytics Event Processing Analytics
Enterprises Custom GUI-driven data access via SDK Siebel eBusiness Local data sources Access Excel STDF Drag-and-drop MySQL SQL Server Oracle Information Services (join, transform, reusable, parameterized, dynamic query for in-memory use) Databases JDBC/ODBC Hadoop SFDC PostgreSQL Teradata Netezza Etc. XML RDBMS Flat Files Spread- sheets Web Services Oracle E-Business RDBMS RDBMS RDBMS SAP BW SAP R/3 D A T A F A B R I C Salesforce ODBC OLE DB SqlClient Direct connection Oracle TeradataAster MS SSAS Teradata Direct Query (dynamically query and retrieve data for visualization and analysis) Databases MySQL Etc. OBIEE Netezza Hadoop
Long-Term Competitive Advantage Value to the Organization Visual Analytics Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Analytics
discovery capability that automatically finds and explains insights from advanced analytics to business users or citizen data scientists” Smart Data Discovery (for the Business User) Leverage Machine Learning without the help of a Data Scientist
Software Inc. Open Source GPL License (including its restrictions) http://www.revolutionanalytics.com/webinars/introducing-revolution-r-open-enhanced-open-source-r-distribution- revolution-analytics
ML is Spark’s machine learning library. Its goal is to make practical machine learning scalable and easy. It consists of common learning algorithms and utilities, including classification, regression, clustering and collaborative filtering. General Data-processing Framework à However, focus is especially on Analytics (at least these days) x
Source Platform for Analytics • Best of Breed Open Source Technology • Easy-to-use Web UI and Familiar Interfaces • Data Agnostic Support for all Common Database and File Types • Massively Scalable Big Data Analysis • Real-time Data Scoring (“Nanofast Scoring Engine”) http://www.h2o.ai/
TIBCO Software Inc. Let the business user leverage Analytic Models (created by the Data Scientist) to find insights! Example: Customer Churn with Random Forest Algorithm • ‘refresh model’ button lives a ‘random forest algorithm’ • requires no a priori assumptions at all, it just always works • The business user doesn’t need to know what random forest is to be empowered by it Select variables for the model
Managed SaaS service for building ML models and generating predictions • Integrated into the corresponding cloud ecosystem • Easy to use, but limited feature set and potential latency issues if combined with external data or applications http://docs.aws.amazon.com/machine-learning/latest/dg/tutoria
Language ) • XML-based de facto standard to represent predictive analytic models • Developed by the Data Mining Group (DMG) • Easily share models between PMML compliant applications (e.g. between model creation and deployment for operations)
Long-Term Competitive Advantage Value to the Organization Self-service Dashboards Event Processing Advanced Analytics Measure Diagnose Predict Optimize Alert Automate Analytics Maturity A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases Visual Analytics Event Processing Analytics
A “Stream”? Clickstream Sensors Usage Data Logs • Consists of pieces of data typically generated due to a change of state. • One or more identifiers • Timestamp & payload • Immutable • Typically unbounded; there is no end to the data. • Batch dataset: “bounded”. • Can be raw or derived.
Interaction Actions by Operations Human decisions in real time informed by up to date information 69 Automated action based on models of history combined with live context and business rules Machine-to-Machine Automation
Frameworks and Products Slide Deck from JavaOne 2015: http://www.kai-waehner.de/blog/2015/10/25/ comparison-of-stream-processing-frameworks-and-products/ Updated slide deck coming in November 2016 (Big Data Spain, Madrid)
Scrap parts as early as possible automatically to reduce costs in a manufacturing process. Question: When to scrap a part in Station 1 instead of doing re-work or sending it to Station 2? Station 1 Station 2 Cost Before 9€ 7€ 13€ Total Cost 29€ (or more) Scrap? Scrap?
UI CSV Batch JSON Real Time XML Real Time Streaming Analytics Action Aggregate Rules Analytics Correlate Live Datamart Continuous query processing Alerts Manual action, escalation HISTORICAL ANALYSIS Data Scientists Flume HDFS Spotfire R / TERR HDFS Hadoop (Cloudera) StreamBase TIBCO Fast Data Platform H2O Oracle RDBMS Avro Parquet … PMML Internal Data
Spark 1. Fast Data Preparation for IoT Dozens of enterprise and IoT data preparation adapters: MQTT, Databases; inbound creation of HDFS, Parquet, Hbase, Avro… 2. Spotfire Model Discovery Template Use Spotfire to explore Spark data lake, create predictive model, train in H20, and deploy to Streaming Analytics. 3. Operationalize Predictive Models Zookeeper deployment to StreamBase nodes living in Spark cluster via H20, PMML, TERR models 4. Streaming Analytics for Automation Automate action based on predictive models – make offers to customers, stop fraudulent transactions, alert. 5. Monitor & Retrain Model Monitor behavior of model, retrain when necessary. 6. Drag & Drop for Business Solution Developers Code-free development environment for work with H20, HDFS, Avro, TERR The TIBCO Accelerator for Spark is a TIBCO engineered, light-weight open-source fast- start for systems to stream data into Spark, discover patterns in Spark with Spotfire, and operationalize the insights on Big Data. FUNCTIONAL COMPONENTS
Cloud Native Microservices Codebase One codebase tracked in revision control, many deploys. Dependencies Explicitly declare and isolate dependencies. Config Store config in the environment. Backing Services Treat backing services as attached resources. Build, Release, Run Strictly separate build and run stages. Processes Execute the app as one or more stateless processes. Port Binding Export services via port binding. Concurrency Scale out via the process model. Disposability Maximize robustness with fast startup and graceful shutdown. Dev / Prod Parity Keep dev, staging, and prod as similar as possible. Logs Treat logs as event streams. Admin Processes Run admin/mgmt tasks as one-off processes. https://12factor.net/
time 1 2 3 4 5 6 7 8 9 Event Streams Apply your intelligent (micro)service to any event. Microservice event. Application event. Legacy event. IoT event. You name it.
are hidden in Historical Data on Big Data Platforms Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models Ø Event Processing uses these Models (without Redevelopment) to take Action in Real Time
teamlab
gionn
takanorip
raspython3
codmoninc
endonanana
daidaidai
greenmidori
pfn
kentarofujii
maimyyym
camiha
pedronauck
garrettdimon
addyosmani
palkan
takai
cassininazir
aki_iinuma
brad_frost
brettharned
bryan
tamaranovitovic
axbom
![Kai Wähner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.kai-waehner.de Big Data](https://files.speakerdeck.com/presentations/e90beb2579234f15b1417dcb07d5cd11/slide_1.jpg){kind=link}
