Introduction to Google Cloud Patform

Introduction to Google Cloud Patform

An introduction to Google Cloud Platform. The slides for the talk I've done to +GDG Jeddah titled: Introduction to Google Cloud Platform. Also covered some of the new features introduced in GCP Next 2017

81f81af7cc5d1e767303b83bd76a1e46?s=128

Omer Dawelbeit

April 22, 2017
Tweet

Transcript

  1. 1.

    Introduction to Google Cloud Platform Dr. Omer Dawelbeit | @omerio

    | +OmerDawelbeit Wadi Makkah Corp and GDG Jeddah Event, Saturday 22nd April 2017
  2. 2.

    About me ‘omerio’ • Co-founder GDG Reading, UK • Google

    Developer Expert (Cloud) Google • 16+ years as Tech Consultant (UK, Middle East & South East Asia) • PhD Computer Science, University of Reading (Cloud + Big Data) • Blog at omerio.com, Tweets @omerio
  3. 3.

    Content • Google Cloud Platform ◦ Compute ◦ Storage &

    Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  4. 4.

    Google Cloud Platform Family • Google Cloud Platform ◦ Compute

    ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  5. 6.

    Compute Compute Engine App Engine Container Engine Container Registry Cloud

    Functions Networking Cloud Virtual Network Cloud Load Balancing Cloud CDN Cloud Interconnect Cloud DNS Big Data BigQuery Cloud Dataflow Cloud Dataproc Cloud Datalab Cloud Pub/Sub Genomics Storage and Databases Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Cloud Spanner Identity & Security Cloud IAM Cloud Resource Manager Cloud Security Scanner Key Management Service Data Loss Prevention Identity-Aware Proxy Security Key Enforcement Persistent Disk Machine Learning Cloud Machine Learning Cloud Vision API Cloud Speech API Cloud Natural Language API Cloud Translation API Cloud Jobs API Networking BETA BETA BETA BETA
  6. 8.

    Management Tools Stackdriver Monitoring Logging Error Reporting Trace Debugger Cloud

    Deployment Manager Cloud Endpoints Cloud Console Developer Tools Cloud SDK Cloud Deployment Manager Cloud Source Repositories Cloud Tools for Android Studio Cloud Tools for IntelliJ Cloud Tools for PowerShell Cloud Tools for Visual Studio Google Plug-in for Eclipse Cloud Test Lab Cloud Shell
  7. 9.

    Compute • Google Cloud Platform ◦ Compute ◦ Storage &

    Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  8. 10.

    Compute Engine IaaS Run VMs on Google Infrastructure Predefined &

    Custom machine types Compute, Storage & Network Sub hour billing Up to 64 cores and 416 GB memory
  9. 11.

    App Engine PaaS You, your app and Google Standard: Java,

    Python, PHP & Go Flexible: .Net, Node.js or your own Free quota Auto scaling Massively scalable Lego-like services
  10. 12.

    Powered by App Engine • Khan Academy ◦ Non for

    profit educational organisation ◦ 10 million unique users / month, 300+ million lessons, 5000+ videos • Snapchat ◦ Photo and video messaging app ◦ 100 million active users / day, 8,796 photos shared / second, 7 billion video views / day • Rovio ◦ Mobile / Web games ◦ 140+ million downloads, 13+ million users / month
  11. 13.

    Container Engine Infrastructure as Code Docker Support Fully Managed Kubernetes

    Clusters Highly Scalable Private Container Registry
  12. 14.

    Cloud Function Serverless Model Integrated with: PubSub, Datastore, etc. Mobile

    Ready: Execute from Firebase Support for Node.js Pay only for function execution BETA Use Cases • Mobile Backend • APIs, Microservices • IoT • HTTP Hooks • ETL
  13. 15.

    Storage & Databases • Google Cloud Platform ◦ Compute ◦

    Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  14. 19.

    BigQuery Columnar Database Big Data Warehouse SQL like queries Analyse

    terabytes of data Streaming data Batch processing
  15. 21.

    Cloud Bigtable Massively scalable NoSQL Low latency, High throughput Good

    for both operational + analytical Seamless cluster resizing Fully managed
  16. 22.
  17. 23.
  18. 25.

    Big Data Processing • Google Cloud Platform ◦ Compute ◦

    Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  19. 26.

    Big Data & Challenges • Big data is data that

    is so large and complex that it becomes difficult to process using traditional tools. • Widely characterised in terms of three dimensions (3Vs): ◦ Volume: massive datasets (terabytes, petabytes,...) ◦ Variety: different kinds of data (structured, semi-structured and unstructured). ◦ Velocity: the rate at which the data is generated from batch to real time. • Traditional OLTP and data warehouse (ETL, data marts and star schemas) only deals with structured data • Web logs, social media, device logs, Cisco netflow collector, etc. Nothing done all data is archived or deleted • Schema on Write is not suitable for such data • Opportunities lost (Insights, etc.)
  20. 28.

    Big Data Workflow • Speed of Data ◦ Batch ◦

    Streaming ▪ Near Real Time ▪ Real Time • Ingest/Capture/Acquire • Process/Organise/Format • Store • Analyze (Machine Learning) • Decide/Use (Machine Learning) Big Data Machine Learning Statistics Data Mining Data Science Data Analytics
  21. 29.

    Big Data on GCP • Google Cloud Platform ◦ Compute

    ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  22. 30.

    Big Data on GCP Cloud Pub/Sub (robust publish/subscribe messaging system)

    - Injest Cloud Dataflow (realtime data pipeline, batch + stream) - Process BigQuery/Bigtable (HBase compatible) - Store/Analyze Cloud Datalab - Analyze Cloud Dataproc. Managed Spark Hadoop clusters Zero-ops, serverless Big Data architecture
  23. 33.

    Example: Retail > Fraud Detection Third-Party Data External Sources Fraud

    Modeling Processed Events Cloud Bigtable Events Time-Series Streaming Cloud Pub/Sub Transaction Streams Processing Cloud Dataflow Transaction Streams Fraud Analysis BigQuery Spark MLlib Cloud Dataproc Regression Prediction API Applications App Engine Rules Cache Cloud Cache Architecture: Retail > Fraud Detection
  24. 34.

    Kubernetes & Container Engine • Google Cloud Platform ◦ Compute

    ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  25. 35.

    What is a container? • Lightweight environment • Static application

    environment • No external dependencies • Application isolation • Reusable, runnable artifacts • Loosely coupled • Portable
  26. 37.

    Kubernetes (k8s) /koo-ber-nay'-tace/ From κυβερνήτης: Greek for "helmsman" or "pilot"

    • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  27. 38.

    What is Kubernetes? • Containers orchestrator (Docker, Rocket) • Open

    source • Run on dev machine, own hardware or cloud • Inspired by Google’s experience • Written in Go • Manages applications, not machines
  28. 40.

    What can Kubernetes do? • co-locating helper processes, • mounting

    storage systems (volumes), • application health checking, • replicating application instances, • horizontal auto-scaling, • load balancing, • rolling updates, • and resource monitoring.
  29. 41.

    Cluster Overview • Pods • Containers • Labels • Replication

    Controllers • Services • Nodes • Kubernetes Master http://omerio.com/2015/12/18/learn-the-kubernetes-key-concepts-in-10-minutes/
  30. 42.

    Pods • Smallest deployable unit • Scheduled to nodes •

    Hosts containers and volumes • Ephemeral • Each has its own routable IP (no NAT) • Containers in a pod ◦ Tightly coupled ◦ Share IP, port and IPC namespaces ◦ Talk to each other through localhost http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html
  31. 43.

    Pods Questions • How to persist data across container restarts?

    • How to create multiple copies of a Pod? • How can containers reference each other reliably?
  32. 44.

    Labels • Key/value pairs (tier=frontend, app=myapp) • Metadata with semantic

    meaning • Membership identifier (services & rc) • Queryable by Selectors (grouping)
  33. 45.

    Replication Controllers • Keeps Pods running • Ensures N copies

    of Pods are running • Provides direct control of Pods • Uses a Pod template • Determines constituent Pods using Label Selector
  34. 47.

    Services • An abstraction • Logical grouping of Pods •

    Determines constituent Pods using Label Selector • Load balancing between constituent Pods • Gets a stable virtual IP, port & DNS name
  35. 48.
  36. 49.

    A few more... • Volumes • Secrets • Namespaces •

    Resource based scheduling ... spec: containers: - name: locust image: gcr.io/rabbit-skateboard/guestbook:gdg-rtv resources: requests: memory: "300Mi" cpu: "300m" limits: memory: "300Mi" cpu: "300m"
  37. 50.

    Google Container Engine • Google Cloud Platform ◦ Compute ◦

    Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  38. 51.

    Hosted Kubernetes Environment • Managed Kubernetes • Manages Kubernetes master

    • Manages updates • Cluster resize via Managed Instance Groups • Cluster node autoscaling (Cloud Monitoring) • Centralised logging • Google Cloud VPN support
  39. 53.

    Getting Started • Google Cloud Platform ◦ Compute ◦ Storage

    & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  40. 54.

    Useful tips • Start here: https://cloud.google.com/getting-started/ • Utilise Cloud Shell

    • Start with Hello World Examples • Progress to to other specific examples • Cloud-first design, avoid life-and-shift • Utilise available service (Don’t reinvent the wheel) • Utilise available boilerplates • Always check the docs • Reference architecture: https://cloud.google.com/docs/tutorials#architecture
  41. 55.

    https://cloud.google.com/free/ • Free Trial: $300 and (3) 12 months •

    Free Tier extends beyond 12 months too (Always Free)
  42. 58.

    Demo Time • Google Cloud Platform ◦ Compute ◦ Storage

    & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  43. 59.

    Wrap Up • Google Cloud Platform ◦ Compute ◦ Storage

    & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
  44. 60.

    What we have covered • Google Cloud Platform ◦ Compute

    ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo
  45. 61.

    Thank you Stay in touch Google+: +OmerDawelbeit Twitter: @omerio LinkedIn:

    OmerDawelbeit Blog: omerio.com Slides: https://goo.gl/mDmycg