GCPUG Austin - Data Infrastructure

Overview of Google Cloud Platform data infrastructure

Transcript

1. Google Cloud Data Infrastructure Sandeep Parikh Solutions Architect @crcsmnky
2. Stuff I Want To Talk About Me Google and Big Data Google Cloud Platform and Big Data Open Source Questions
3. 3 Google Cloud Platform Hey That’s Me! I spend my time working on repeatable architectural patterns and guidance for people interested in using Google Cloud Platform in the form of papers, code, architectures. Typically spend time talking about Big Data and Containers. Before Google, I was at MongoDB, Ravel, 21CT, Affinegy, Apple. I’ve been in Austin for ~12 years so I get to complain about everything. Find me on Twitter @crcsmnky
4. Google and Data
5. 5 Google Cloud Platform Organize the world’s information and make it universally accessible and useful.
6. Google Cloud Platform 6 2012 2013 2002 2004 2006 2008 2010 GFS MapReduce BigTable Colossus Dremel Flume Megastore Spanner Millwheel PubSub F1 Google Research in Data Technologies Google Research Publications referenced are available here: http://research.google.com/pubs/papers.html The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2009 http://research.google.com/pubs/pub35290.html
7. Google Cloud Platform 7 2012 2013 2002 2004 2006 2008 2010 GFS MapReduce BigTable Colossus Dremel Flume Megastore Spanner Millwheel PubSub F1 Open Source Ecosystem Google Research Publications referenced are available here: http://research.google.com/pubs/papers.html The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2009 http://research.google.com/pubs/pub35290.html
8. Google Cloud Platform 8 Google Research Publications referenced are available here: http://research.google.com/pubs/papers.html The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2009 http://research.google.com/pubs/pub35290.html 2012 2013 2002 2004 2006 2008 2010 Cloud Storage Dataproc Bigtable Cloud Storage BigQuery Dataflow Datastore Spanner Dataflow PubSub F1 Cloud Platform Data Infrastructure
9. Google Cloud Platform and Data
10. 10 Google Cloud Platform Management Mobile Services Compute Data Networking Storage Developer Tools
11. 11 Google Cloud Platform Capabilities Collaboration and exploration Data processing Intelligence Cloud Databases Analytics Storage Services Messaging Services Data exploration in the Cloud Cloud Datalab Fast & economical data warehouse for large-scale data analytics Google BigQuery Mainstream Cloud based artificial intelligence and machine learning Cloud ML, Vision API, Prediction API, Translate API Flexible, scalable and reliable data processing Streaming/batch processing, Hadoop/Spark - Cloud Dataflow, Cloud Dataproc Cloud Databases for all kinds of applications Relational, key-value, NoSQL - Cloud Bigtable, Cloud SQL, Cloud Datastore Proven storage platform GCS Standard, GCS DRA, GCS Nearline Reliable, large scale messaging Cloud Pub/Sub
12. 12 Google Cloud Platform Data Lifecycle Capture Process Store Analyze Mobile Phones In-Game Analytics Clickstream Object Store HDFS Hadoop Spark Storm SQL Pig Hive
13. 13 Google Cloud Platform Data Lifecycle Capture Process Store Analyze App Engine Cloud Pub/Sub Cloud Storage Cloud SQL Cloud Bigtable Cloud Datastore Cloud Dataflow Cloud Dataproc BigQuery Cloud Datalab
14. 14 Google Cloud Platform Globally redundant Low latency (sub sec.) Batched read/write Custom labels Push & Pull Auto expiration Cloud Pub/Sub Pub A Pub B Pub C Topic C Sub A Sub B Sub C1 Sub C2 Cloud Pub/Sub Subscriber X Subscriber Y Subscriber Z Message 1 Message 2 Topic A Topic B Message 3 Message 1 Message 2 Message 3 Message 3
15. 15 Google Cloud Platform Cloud Dataflow Autoscaling mid-job Fully managed - No-Ops Intuitive Data Processing Framework Batch and Stream Processing in one Dynamic rebalancing mid-job 1 2 3 4 5
16. 16 Google Cloud Platform Cloud Dataflow Pipeline p = Pipeline.create(); p.begin() .apply(TextIO.Read.from(“gs://…”)) .apply(ParDo.of(new ExtractTags()) .apply(Count.create()) .apply(ParDo.of(new ExpandPrefixes()) .apply(Top.largestPerKey(3)) .apply(TextIO.Write.to(“gs://…”)); p.run(); Pipeline p = Pipeline.create(); p.begin() .apply(TextIO.Read.from(“gs://…”)) .apply(ParDo.of(new ExtractTags()) .apply(Count.create()) .apply(ParDo.of(new ExpandPrefixes()) .apply(Top.largestPerKey(3)) .apply(TextIO.Write.to(“gs://…”)); p.run(); .apply(PubsubIO.Read.from(“input_topic”)) .apply(Window.<Integer>by(FixedWindows.of(5, MINUTES)) .apply(PubsubIO.Write.to(“output_topic”)); Batch to Streaming
17. 17 Google Cloud Platform Cloud Dataproc (managed Hadoop & Spark) Preemptible VMs are 70% cheaper Spin up clusters of any size in 90 seconds Separation of storage and compute Run clusters segregated by job or function Per-minute billing 1 2 3 4 5
18. 18 Google Cloud Platform Hive for IT Business Reporting Spark MLlib Hive for Analysts MapReduce ETL Google Cloud Storage Cloud Dataproc
19. 19 Google Cloud Platform Cloud Bigtable Easily scale to hundreds of PBs Low-latency and high-throughput Massively scalable NoSQL database SSDs or HDDs depending on need Easy to integrate with Dataflow, Dataproc 1 2 3 4 5
20. 20 Google Cloud Platform Cloud Bigtable Put, Increment, Append Bigtable Replication Full Scan, Map Reduce Gets, Short Scan Bulk Import Low Latency High Throughput Bigtable Replication
21. 21 Google Cloud Platform Cloud Bigtable in Context NoSQL Key Value Blob SQL Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Good for: Structured and unstructured binary or object data Such as: Images, large media files, backups Good for: Getting started, App Engine, serve use cases Such as: User profiles, product catalog Good for: Web frameworks, existing applications Such as: User credentials, customer orders Good for: Heavy read + write, events, and analytical data Such as: AdTech, Financial and IoT data Memcache Good for: Web/mobile applications, gaming Such as: Game state, user sessions
22. 22 Google Cloud Platform Petabyte scale data warehouse Supports SQL Fast and scales automatically No setup or administration Stream up to 100,000 rows/sec Integrates with third-party software like Tableau Google BigQuery
23. 23 Google Cloud Platform Cloud Datalab Launch notebooks to explore, transform and process data on Google Cloud Platform or locally. Built on IPython/Jupyter which already has a thriving ecosystem of modules and a huge knowledge base. Write code in multiple languages: Python, SQL and JavaScript. Fully Integrated Built on Jupyter Choose your language Notebooks It leverages the power of Cloud Storage, BigQuery, Cloud DataStore and Cloud SQL for analyses.
24. 24 Google Cloud Platform Cloud Datalab Increased productivity through interactive tools and availability of third party libraries. Explore and analyze data with ad hoc queries and visualizations. Explore, transform and process data collaboratively or publish data as reports, dashboards or APIs. Collaboration Reach Increase Productivity Simplicity Makes Google’s Big Data capabilities easier to use and therefore more accessible across the company.
25. 25 Google Cloud Platform Cloud Datalab
26. Open Source
27. 27 Google Cloud Platform Open Source Cloud Dataflow → Apache Beam Incubating at Apache Open source SDKs Open source runners for Spark and Flink Cloud Datalab built on Jupyter Cloud Bigtable supports HBase 1.0 API Cloud Dataproc open source Hadoop and Spark Kubernetes completely open source
28. Solutions on Google Cloud Platform
29. 29 Google Cloud Platform • Reverse Geocoding using Cloud and Maps • Scalable Geolocation Telemetry using Cloud • Machine Learning with Financial Time Series • Cloud Bigtable Schema Design for Time Series Data • Analyzing Financial Time Series using BigQuery • Processing Logs at Scale using Cloud Dataflow A Few Solutions • Reliable Task Scheduling on Google Compute Engine • Real-Time Inventory using Google Cloud • Distributed Load Testing Using Kubernetes • Deploying Microservices on Google App Engine • Automated Image Builds with Jenkins, Packer, and Kubernetes • Real-time data analysis with Kubernetes, Redis, and BigQuery
30. 30 Google Cloud Platform Processing Logs using Cloud Dataflow
31. 31 Google Cloud Platform Complex Event Processing
32. 32 Google Cloud Platform Real-Time Inventory
33. Thanks!
34. 34 Google Cloud Platform Questions & Comments Sandeep Parikh parikhs@google.com @crcsmnky