GCPUG Austin - Data Infrastructure

GCPUG Austin - Data Infrastructure

Overview of Google Cloud Platform data infrastructure

A1af6e45dfb6e6cb9a64834484adf788?s=128

Sandeep Parikh

June 22, 2016
Tweet

Transcript

  1. Google Cloud Data Infrastructure Sandeep Parikh Solutions Architect @crcsmnky

  2. Stuff I Want To Talk About Me Google and Big

    Data Google Cloud Platform and Big Data Open Source Questions
  3. 3 Google Cloud Platform Hey That’s Me! I spend my

    time working on repeatable architectural patterns and guidance for people interested in using Google Cloud Platform in the form of papers, code, architectures. Typically spend time talking about Big Data and Containers. Before Google, I was at MongoDB, Ravel, 21CT, Affinegy, Apple. I’ve been in Austin for ~12 years so I get to complain about everything. Find me on Twitter @crcsmnky
  4. Google and Data

  5. 5 Google Cloud Platform Organize the world’s information and make

    it universally accessible and useful.
  6. Google Cloud Platform 6 2012 2013 2002 2004 2006 2008

    2010 GFS MapReduce BigTable Colossus Dremel Flume Megastore Spanner Millwheel PubSub F1 Google Research in Data Technologies Google Research Publications referenced are available here: http://research.google.com/pubs/papers.html The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2009 http://research.google.com/pubs/pub35290.html
  7. Google Cloud Platform 7 2012 2013 2002 2004 2006 2008

    2010 GFS MapReduce BigTable Colossus Dremel Flume Megastore Spanner Millwheel PubSub F1 Open Source Ecosystem Google Research Publications referenced are available here: http://research.google.com/pubs/papers.html The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2009 http://research.google.com/pubs/pub35290.html
  8. Google Cloud Platform 8 Google Research Publications referenced are available

    here: http://research.google.com/pubs/papers.html The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2009 http://research.google.com/pubs/pub35290.html 2012 2013 2002 2004 2006 2008 2010 Cloud Storage Dataproc Bigtable Cloud Storage BigQuery Dataflow Datastore Spanner Dataflow PubSub F1 Cloud Platform Data Infrastructure
  9. Google Cloud Platform and Data

  10. 10 Google Cloud Platform Management Mobile Services Compute Data Networking

    Storage Developer Tools
  11. 11 Google Cloud Platform Capabilities Collaboration and exploration Data processing

    Intelligence Cloud Databases Analytics Storage Services Messaging Services Data exploration in the Cloud Cloud Datalab Fast & economical data warehouse for large-scale data analytics Google BigQuery Mainstream Cloud based artificial intelligence and machine learning Cloud ML, Vision API, Prediction API, Translate API Flexible, scalable and reliable data processing Streaming/batch processing, Hadoop/Spark - Cloud Dataflow, Cloud Dataproc Cloud Databases for all kinds of applications Relational, key-value, NoSQL - Cloud Bigtable, Cloud SQL, Cloud Datastore Proven storage platform GCS Standard, GCS DRA, GCS Nearline Reliable, large scale messaging Cloud Pub/Sub
  12. 12 Google Cloud Platform Data Lifecycle Capture Process Store Analyze

    Mobile Phones In-Game Analytics Clickstream Object Store HDFS Hadoop Spark Storm SQL Pig Hive
  13. 13 Google Cloud Platform Data Lifecycle Capture Process Store Analyze

    App Engine Cloud Pub/Sub Cloud Storage Cloud SQL Cloud Bigtable Cloud Datastore Cloud Dataflow Cloud Dataproc BigQuery Cloud Datalab
  14. 14 Google Cloud Platform Globally redundant Low latency (sub sec.)

    Batched read/write Custom labels Push & Pull Auto expiration Cloud Pub/Sub Pub A Pub B Pub C Topic C Sub A Sub B Sub C1 Sub C2 Cloud Pub/Sub Subscriber X Subscriber Y Subscriber Z Message 1 Message 2 Topic A Topic B Message 3 Message 1 Message 2 Message 3 Message 3
  15. 15 Google Cloud Platform Cloud Dataflow Autoscaling mid-job Fully managed

    - No-Ops Intuitive Data Processing Framework Batch and Stream Processing in one Dynamic rebalancing mid-job 1 2 3 4 5
  16. 16 Google Cloud Platform Cloud Dataflow Pipeline p = Pipeline.create();

    p.begin() .apply(TextIO.Read.from(“gs://…”)) .apply(ParDo.of(new ExtractTags()) .apply(Count.create()) .apply(ParDo.of(new ExpandPrefixes()) .apply(Top.largestPerKey(3)) .apply(TextIO.Write.to(“gs://…”)); p.run(); Pipeline p = Pipeline.create(); p.begin() .apply(TextIO.Read.from(“gs://…”)) .apply(ParDo.of(new ExtractTags()) .apply(Count.create()) .apply(ParDo.of(new ExpandPrefixes()) .apply(Top.largestPerKey(3)) .apply(TextIO.Write.to(“gs://…”)); p.run(); .apply(PubsubIO.Read.from(“input_topic”)) .apply(Window.<Integer>by(FixedWindows.of(5, MINUTES)) .apply(PubsubIO.Write.to(“output_topic”)); Batch to Streaming
  17. 17 Google Cloud Platform Cloud Dataproc (managed Hadoop & Spark)

    Preemptible VMs are 70% cheaper Spin up clusters of any size in 90 seconds Separation of storage and compute Run clusters segregated by job or function Per-minute billing 1 2 3 4 5
  18. 18 Google Cloud Platform Hive for IT Business Reporting Spark

    MLlib Hive for Analysts MapReduce ETL Google Cloud Storage Cloud Dataproc
  19. 19 Google Cloud Platform Cloud Bigtable Easily scale to hundreds

    of PBs Low-latency and high-throughput Massively scalable NoSQL database SSDs or HDDs depending on need Easy to integrate with Dataflow, Dataproc 1 2 3 4 5
  20. 20 Google Cloud Platform Cloud Bigtable Put, Increment, Append Bigtable

    Replication Full Scan, Map Reduce Gets, Short Scan Bulk Import Low Latency High Throughput Bigtable Replication
  21. 21 Google Cloud Platform Cloud Bigtable in Context NoSQL Key

    Value Blob SQL Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Good for: Structured and unstructured binary or object data Such as: Images, large media files, backups Good for: Getting started, App Engine, serve use cases Such as: User profiles, product catalog Good for: Web frameworks, existing applications Such as: User credentials, customer orders Good for: Heavy read + write, events, and analytical data Such as: AdTech, Financial and IoT data Memcache Good for: Web/mobile applications, gaming Such as: Game state, user sessions
  22. 22 Google Cloud Platform Petabyte scale data warehouse Supports SQL

    Fast and scales automatically No setup or administration Stream up to 100,000 rows/sec Integrates with third-party software like Tableau Google BigQuery
  23. 23 Google Cloud Platform Cloud Datalab Launch notebooks to explore,

    transform and process data on Google Cloud Platform or locally. Built on IPython/Jupyter which already has a thriving ecosystem of modules and a huge knowledge base. Write code in multiple languages: Python, SQL and JavaScript. Fully Integrated Built on Jupyter Choose your language Notebooks It leverages the power of Cloud Storage, BigQuery, Cloud DataStore and Cloud SQL for analyses.
  24. 24 Google Cloud Platform Cloud Datalab Increased productivity through interactive

    tools and availability of third party libraries. Explore and analyze data with ad hoc queries and visualizations. Explore, transform and process data collaboratively or publish data as reports, dashboards or APIs. Collaboration Reach Increase Productivity Simplicity Makes Google’s Big Data capabilities easier to use and therefore more accessible across the company.
  25. 25 Google Cloud Platform Cloud Datalab

  26. Open Source

  27. 27 Google Cloud Platform Open Source Cloud Dataflow → Apache

    Beam Incubating at Apache Open source SDKs Open source runners for Spark and Flink Cloud Datalab built on Jupyter Cloud Bigtable supports HBase 1.0 API Cloud Dataproc open source Hadoop and Spark Kubernetes completely open source
  28. Solutions on Google Cloud Platform

  29. 29 Google Cloud Platform • Reverse Geocoding using Cloud and

    Maps • Scalable Geolocation Telemetry using Cloud • Machine Learning with Financial Time Series • Cloud Bigtable Schema Design for Time Series Data • Analyzing Financial Time Series using BigQuery • Processing Logs at Scale using Cloud Dataflow A Few Solutions • Reliable Task Scheduling on Google Compute Engine • Real-Time Inventory using Google Cloud • Distributed Load Testing Using Kubernetes • Deploying Microservices on Google App Engine • Automated Image Builds with Jenkins, Packer, and Kubernetes • Real-time data analysis with Kubernetes, Redis, and BigQuery
  30. 30 Google Cloud Platform Processing Logs using Cloud Dataflow

  31. 31 Google Cloud Platform Complex Event Processing

  32. 32 Google Cloud Platform Real-Time Inventory

  33. Thanks!

  34. 34 Google Cloud Platform Questions & Comments Sandeep Parikh parikhs@google.com

    @crcsmnky