How to Collect, Transform and Visualize Your Data on the Cloud

Data collection, Data processing and Data visualisation and Reporting are becoming more and more challenging, as data volumes grow and data formats change.

Combined with the increase use of public cloud infrastructure, developers and data professionals need to find new strategies and adopt new technologies in order to design and maintain scalable and cost effective data architectures.

In this talk, Xplenty and WalkMe presents how to build such an architecture, using Amazon CloudFront, Xplenty, Amazon Redshift and Looker. We examine how WalkMe, the guidance and engagement platform empowers product teams, partners and clients by utilising such an architecture to build innovative products and services delivered based on big amounts of online user behaviour data.

References:
WalkMe's Presentation: http://goo.gl/EAiHSf

Xplenty - https://www.xplenty.com
WalkMe - http://www.walkme.com
Looker - http://www.looker.com

Xplenty Intro Video - https://www.youtube.com/watch?v=GU3vtB0WJsc

Transcript

1. HOW TO COLLECT TRANSFORM VISUALIZE YOUR DATA ON THE CLOUD Moty Michaely - Xplenty David Ilievsky - WalkMe
2. Moty Michaely VP R&D @xplenty I tweet from @motymichaely
4. XPLENTY IS A BIG DATA PROCESSING PLATFORM POWERED BY HADOOP & SPARK
5. ANALYTICS SERVICES LOGGING SERVICES DATA STORES CLOUD STORAGE XPLENTY INTEGRATES WITH
6. XPLENTY IS CLOUD BASED ON AMAZON WEB SERVICES GOOGLE CLOUD IBM SOFTLAYER RACKSPACE CLOUD
7. LET’S TALK ABOUT DATA COLLECTION DATA TRANSFORMATION DATA VISUALIZATION IN THE CLOUD
8. CLOUD DATA ARCHITECTURE
9. WHY BUILD YOUR OWN DATA ARCHITECTURE: 3REASONS
10. WHY BUILD YOUR OWN DATA ARCHITECTURE: 1. INTEGRATION Make different systems work together.
11. WHY BUILD YOUR OWN DATA ARCHITECTURE: 2. CONTROL Any data type. From anywhere.
12. WHY BUILD YOUR OWN DATA ARCHITECTURE: 3. OWNERSHIP Own your data. It’s yours.
13. BUILDING YOUR OWN DATA ARCHITECTURE INTEGRATION CONTROL OWNERSHIP = RECAP
14. DATA ARCHITECTURE BUILDING BLOCKS
15. STORE TRACK COLLECT CloudFront, Kinesis, Mongo, EC2, S3 JS, Arduino, Sensors, AWS Lambda
16. PROCESS STORE TRACK COLLECT TRANSFORM Xplenty, Hadoop, Talend, Kinesis CloudFront, Kinesis, Mongo, EC2, S3 JS, Arduino, Sensors, AWS Lambda
17. PROCESS STORE TRACK ANALYTIC STORE COLLECT TRANSFORM VISUALIZE Xplenty, Hadoop, Talend, Kinesis CloudFront, Kinesis, Mongo, EC2, S3 JS, Arduino, Sensors, AWS Lambda Redshift, BigQuery, SAP Hana, RDS EXPLORE Looker, Tableau, Bime, Excel, d3.js
18. CLOUD DATA ARCHITECTURE
19. Looker, Tableau, Bime, Excel, d3.js JS, Arduino, Sensors, AWS Lambda REDSHIFT DATA PIPELINE EMR ELB EC2 DYNAMO PROCESS STORE TRACK ANALYTIC STORE COLLECT TRANSFORM EXPLORE VISUALIZE
20. SIMPLE CLOUD DATA ARCHITECTURE
21. SCALABLE CLOUD DATA ARCHITECTURE
22. RELIABLE CLOUD DATA ARCHITECTURE
23. ROBUST CLOUD DATA ARCHITECTURE
24. Looker, Tableau, Bime, Excel, d3.js JS, Arduino, Sensors, AWS Lambda REDSHIFT DATA PIPELINE EMR ELB EC2 DYNAMO PROCESS STORE TRACK ANALYTIC STORE COLLECT TRANSFORM EXPLORE VISUALIZE
25. CASE STUDY
26. JS, Arduino, Sensors, AWS Lambda REDSHIFT CLOUDFRONT S3 PROCESS STORE TRACK ANALYTIC STORE COLLECT TRANSFORM VISUALIZE EXPLORE
27. CLOUD DATA ARCHITECTURE SIMPLE SCALABILITY RELIABILITY ROBUSTNESS RECAP =
28. DATA COLLECTION
29. DATA COLLECTION IS ESSENTIAL
30. DATA COLLECTION DRIVES BUSINESS
31. DATA COLLECTION IS CHALLENGING
32. TRACK WITH A TRACKER
33. JAVASCRIPT TRACKER
34. JAVASCRIPT TRACKER $.track_event=function(action, meta_data) { var url = 'https://my.tracker.com/' + action + '.png'; return $.get(url, meta_data); };
35. JAVASCRIPT TRACKER track_event('click', { partner: 'Amazon' trackable_type: 'Product', trackable_id: 512, user_id: 412451 });
36. JAVASCRIPT TRACKER track_event('click', { partner: 'Amazon' trackable_type: 'Product', trackable_id: 512, user_id: 412451 }); EVENT TYPE
37. JAVASCRIPT TRACKER track_event('click', { partner: 'Amazon' trackable_type: 'Product', trackable_id: 512, user_id: 412451 }); EVENT META DATA
38. JAVASCRIPT TRACKER track_event('click', { partner: 'Amazon' trackable_type: 'Product', trackable_id: 512, user_id: 412451 });
39. STORE WITH A COLLECTOR
40. AWS CLOUDFRONT COLLECTOR
41. AWS CLOUDFRONT COLLECTOR IS FAST (it’s a CDN) SIMPLE ROBUST RELIABLE SCALABLE
42. AWS CLOUDFRONT COLLECTOR IS FAST SIMPLE (just upload a file) ROBUST RELIABLE SCALABLE
43. AWS CLOUDFRONT COLLECTOR IS FAST SIMPLE ROBUST (based on AWS logging and S3) RELIABLE SCALABLE
44. AWS CLOUDFRONT COLLECTOR IS FAST SIMPLE ROBUST RELIABLE (high durability) SCALABLE
45. AWS CLOUDFRONT COLLECTOR IS FAST SIMPLE ROBUST RELIABLE SCALABLE (built for HUGE scale)
46. DATA TRANSFORMATION
47. DATA TRANSFORMATION IS ESSENTIAL
48. DATA TRANSFORMATION KEEPS CONSISTENCY
49. DATA TRANSFORMATION KEEPS DATA INTEGRITY
50. DATA TRANSFORMATION IS CHALLENGING
51. TRANSFORM WITH DATA PROCESSING PLATFORM
52. XPLENTY DATA PROCESSING PLATFORM
53. XPLENTY DATA PROCESSING PLATFORM IS SIMPLE (code free, no deployment) SCALABLE INTEGRATED FLEXIBLE ROBUST
54. XPLENTY DATA PROCESSING PLATFORM IS SIMPLE SCALABLE (Hadoop and Spark) INTEGRATED FLEXIBLE ROBUST
55. XPLENTY DATA PROCESSING PLATFORM IS SIMPLE SCALABLE INTEGRATED (connectors) FLEXIBLE ROBUST
56. XPLENTY DATA PROCESSING PLATFORM IS SIMPLE SCALABLE INTEROPERABLE FLEXIBLE (transformations) ROBUST
57. XPLENTY DATA PROCESSING PLATFORM IS SIMPLE SCALABLE INTEROPERABLE FLEXIBLE ROBUST (schema on load)
58. DEMO
59. DATA VISUALIZATION
60. STORE INTO DATA WAREHOUSE
61. AMAZON REDSHIFT ANALYTIC STORE
62. AMAZON REDSHIFT IS FAST (columnar) FULLY MANAGED SCALABLE SQL COMPLIANT
63. AMAZON REDSHIFT IS FAST FULLY MANAGED (AWS service) SCALABLE SQL COMPLIANT
64. AMAZON REDSHIFT IS FAST FULLY MANAGED SCALABLE (scale out options) SQL COMPLIANT
65. AMAZON REDSHIFT IS FAST FULLY MANAGED SCALABLE SQL COMPLIANT (postgres)
66. EXPLORE WITH DATA EXPLORATION TOOL
67. LOOKER DATA EXPLORATION
68. LOOKER DATA EXPLORATION SIMPLE (no SQL required) DATA MODELING COLLABORATION DASHBOARDS
69. LOOKER DATA EXPLORATION SIMPLE DATA MODELING (abstraction LookML) COLLABORATION DASHBOARDS
70. LOOKER DATA EXPLORATION SIMPLE DATA MODELING COLLABORATION (share, git) DASHBOARDS
71. LOOKER DATA EXPLORATION SIMPLE DATA MODELING COLLABORATION DASHBOARDS (for business users)
72. DEMO
73. THANK YOU moty@xplenty.com