Introduction to Google Cloud Patform

Introduction to Google Cloud Patform

An introduction to Google Cloud Platform. The slides for the talk I've done to +GDG Jeddah titled: Introduction to Google Cloud Platform. Also covered some of the new features introduced in GCP Next 2017

81f81af7cc5d1e767303b83bd76a1e46?s=128

Omer Dawelbeit

April 22, 2017
Tweet

Transcript

  1. Introduction to Google Cloud Platform Dr. Omer Dawelbeit | @omerio

    | +OmerDawelbeit Wadi Makkah Corp and GDG Jeddah Event, Saturday 22nd April 2017
  2. About me ‘omerio’ • Co-founder GDG Reading, UK • Google

    Developer Expert (Cloud) Google • 16+ years as Tech Consultant (UK, Middle East & South East Asia) • PhD Computer Science, University of Reading (Cloud + Big Data) • Blog at omerio.com, Tweets @omerio
  3. Content • Google Cloud Platform ◦ Compute ◦ Storage &

    Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  4. Google Cloud Platform Family • Google Cloud Platform ◦ Compute

    ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  5. On-Prem, IaaS, PaaS, SaaS! Source: https://www.linkedin.com/pulse/20140730172610-9679881-pizza-as-a-service

  6. Compute Compute Engine App Engine Container Engine Container Registry Cloud

    Functions Networking Cloud Virtual Network Cloud Load Balancing Cloud CDN Cloud Interconnect Cloud DNS Big Data BigQuery Cloud Dataflow Cloud Dataproc Cloud Datalab Cloud Pub/Sub Genomics Storage and Databases Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Cloud Spanner Identity & Security Cloud IAM Cloud Resource Manager Cloud Security Scanner Key Management Service Data Loss Prevention Identity-Aware Proxy Security Key Enforcement Persistent Disk Machine Learning Cloud Machine Learning Cloud Vision API Cloud Speech API Cloud Natural Language API Cloud Translation API Cloud Jobs API Networking BETA BETA BETA BETA
  7. Google Grade Security Data Loss Prevention Security starting from the

    hardware (Titan Chip)
  8. Management Tools Stackdriver Monitoring Logging Error Reporting Trace Debugger Cloud

    Deployment Manager Cloud Endpoints Cloud Console Developer Tools Cloud SDK Cloud Deployment Manager Cloud Source Repositories Cloud Tools for Android Studio Cloud Tools for IntelliJ Cloud Tools for PowerShell Cloud Tools for Visual Studio Google Plug-in for Eclipse Cloud Test Lab Cloud Shell
  9. Compute • Google Cloud Platform ◦ Compute ◦ Storage &

    Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  10. Compute Engine IaaS Run VMs on Google Infrastructure Predefined &

    Custom machine types Compute, Storage & Network Sub hour billing Up to 64 cores and 416 GB memory
  11. App Engine PaaS You, your app and Google Standard: Java,

    Python, PHP & Go Flexible: .Net, Node.js or your own Free quota Auto scaling Massively scalable Lego-like services
  12. Powered by App Engine • Khan Academy ◦ Non for

    profit educational organisation ◦ 10 million unique users / month, 300+ million lessons, 5000+ videos • Snapchat ◦ Photo and video messaging app ◦ 100 million active users / day, 8,796 photos shared / second, 7 billion video views / day • Rovio ◦ Mobile / Web games ◦ 140+ million downloads, 13+ million users / month
  13. Container Engine Infrastructure as Code Docker Support Fully Managed Kubernetes

    Clusters Highly Scalable Private Container Registry
  14. Cloud Function Serverless Model Integrated with: PubSub, Datastore, etc. Mobile

    Ready: Execute from Firebase Support for Node.js Pay only for function execution BETA Use Cases • Mobile Backend • APIs, Microservices • IoT • HTTP Hooks • ETL
  15. Storage & Databases • Google Cloud Platform ◦ Compute ◦

    Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  16. Cloud Storage IaaS Multi-petabytes Object storage Globally replicated Highly available

    and reliable Flexible security models
  17. Cloud SQL Relational MySQL database Fully managed Highly available Auto

    backup MySQL + PostgresQL
  18. Cloud Datastore NoSQL like solution Fully managed Highly available and

    scalable Flexible access Autoscaled
  19. BigQuery Columnar Database Big Data Warehouse SQL like queries Analyse

    terabytes of data Streaming data Batch processing
  20. Cloud Spanner Relational Database Horizontally scalable High performance & availability

    Automatic replication Fully managed BETA
  21. Cloud Bigtable Massively scalable NoSQL Low latency, High throughput Good

    for both operational + analytical Seamless cluster resizing Fully managed
  22. None
  23. None
  24. Building your own solutions, is like … LEGO

  25. Big Data Processing • Google Cloud Platform ◦ Compute ◦

    Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  26. Big Data & Challenges • Big data is data that

    is so large and complex that it becomes difficult to process using traditional tools. • Widely characterised in terms of three dimensions (3Vs): ◦ Volume: massive datasets (terabytes, petabytes,...) ◦ Variety: different kinds of data (structured, semi-structured and unstructured). ◦ Velocity: the rate at which the data is generated from batch to real time. • Traditional OLTP and data warehouse (ETL, data marts and star schemas) only deals with structured data • Web logs, social media, device logs, Cisco netflow collector, etc. Nothing done all data is archived or deleted • Schema on Write is not suitable for such data • Opportunities lost (Insights, etc.)
  27. I’m confused!

  28. Big Data Workflow • Speed of Data ◦ Batch ◦

    Streaming ▪ Near Real Time ▪ Real Time • Ingest/Capture/Acquire • Process/Organise/Format • Store • Analyze (Machine Learning) • Decide/Use (Machine Learning) Big Data Machine Learning Statistics Data Mining Data Science Data Analytics
  29. Big Data on GCP • Google Cloud Platform ◦ Compute

    ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  30. Big Data on GCP Cloud Pub/Sub (robust publish/subscribe messaging system)

    - Injest Cloud Dataflow (realtime data pipeline, batch + stream) - Process BigQuery/Bigtable (HBase compatible) - Store/Analyze Cloud Datalab - Analyze Cloud Dataproc. Managed Spark Hadoop clusters Zero-ops, serverless Big Data architecture
  31. Reference Architecture - 1

  32. Reference Architecture - 2

  33. Example: Retail > Fraud Detection Third-Party Data External Sources Fraud

    Modeling Processed Events Cloud Bigtable Events Time-Series Streaming Cloud Pub/Sub Transaction Streams Processing Cloud Dataflow Transaction Streams Fraud Analysis BigQuery Spark MLlib Cloud Dataproc Regression Prediction API Applications App Engine Rules Cache Cloud Cache Architecture: Retail > Fraud Detection
  34. Kubernetes & Container Engine • Google Cloud Platform ◦ Compute

    ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  35. What is a container? • Lightweight environment • Static application

    environment • No external dependencies • Application isolation • Reusable, runnable artifacts • Loosely coupled • Portable
  36. Containers vs. VMs

  37. Kubernetes (k8s) /koo-ber-nay'-tace/ From κυβερνήτης: Greek for "helmsman" or "pilot"

    • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  38. What is Kubernetes? • Containers orchestrator (Docker, Rocket) • Open

    source • Run on dev machine, own hardware or cloud • Inspired by Google’s experience • Written in Go • Manages applications, not machines
  39. Containers Orchestration

  40. What can Kubernetes do? • co-locating helper processes, • mounting

    storage systems (volumes), • application health checking, • replicating application instances, • horizontal auto-scaling, • load balancing, • rolling updates, • and resource monitoring.
  41. Cluster Overview • Pods • Containers • Labels • Replication

    Controllers • Services • Nodes • Kubernetes Master http://omerio.com/2015/12/18/learn-the-kubernetes-key-concepts-in-10-minutes/
  42. Pods • Smallest deployable unit • Scheduled to nodes •

    Hosts containers and volumes • Ephemeral • Each has its own routable IP (no NAT) • Containers in a pod ◦ Tightly coupled ◦ Share IP, port and IPC namespaces ◦ Talk to each other through localhost http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html
  43. Pods Questions • How to persist data across container restarts?

    • How to create multiple copies of a Pod? • How can containers reference each other reliably?
  44. Labels • Key/value pairs (tier=frontend, app=myapp) • Metadata with semantic

    meaning • Membership identifier (services & rc) • Queryable by Selectors (grouping)
  45. Replication Controllers • Keeps Pods running • Ensures N copies

    of Pods are running • Provides direct control of Pods • Uses a Pod template • Determines constituent Pods using Label Selector
  46. Replication Controllers

  47. Services • An abstraction • Logical grouping of Pods •

    Determines constituent Pods using Label Selector • Load balancing between constituent Pods • Gets a stable virtual IP, port & DNS name
  48. Services

  49. A few more... • Volumes • Secrets • Namespaces •

    Resource based scheduling ... spec: containers: - name: locust image: gcr.io/rabbit-skateboard/guestbook:gdg-rtv resources: requests: memory: "300Mi" cpu: "300m" limits: memory: "300Mi" cpu: "300m"
  50. Google Container Engine • Google Cloud Platform ◦ Compute ◦

    Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  51. Hosted Kubernetes Environment • Managed Kubernetes • Manages Kubernetes master

    • Manages updates • Cluster resize via Managed Instance Groups • Cluster node autoscaling (Cloud Monitoring) • Centralised logging • Google Cloud VPN support
  52. Google Container Engine Overview

  53. Getting Started • Google Cloud Platform ◦ Compute ◦ Storage

    & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  54. Useful tips • Start here: https://cloud.google.com/getting-started/ • Utilise Cloud Shell

    • Start with Hello World Examples • Progress to to other specific examples • Cloud-first design, avoid life-and-shift • Utilise available service (Don’t reinvent the wheel) • Utilise available boilerplates • Always check the docs • Reference architecture: https://cloud.google.com/docs/tutorials#architecture
  55. https://cloud.google.com/free/ • Free Trial: $300 and (3) 12 months •

    Free Tier extends beyond 12 months too (Always Free)
  56. https://cloud.google.com/getting-started/

  57. https://codelabs.developers.google.com/?cat=Cloud

  58. Demo Time • Google Cloud Platform ◦ Compute ◦ Storage

    & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  59. Wrap Up • Google Cloud Platform ◦ Compute ◦ Storage

    & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  60. What we have covered • Google Cloud Platform ◦ Compute

    ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo
  61. Thank you Stay in touch Google+: +OmerDawelbeit Twitter: @omerio LinkedIn:

    OmerDawelbeit Blog: omerio.com Slides: https://goo.gl/mDmycg