Introduction to Google Cloud Patform

An introduction to Google Cloud Platform. The slides for the talk I've done to +GDG Jeddah titled: Introduction to Google Cloud Platform. Also covered some of the new features introduced in GCP Next 2017

Transcript

1. Introduction to Google Cloud Platform Dr. Omer Dawelbeit | @omerio | +OmerDawelbeit Wadi Makkah Corp and GDG Jeddah Event, Saturday 22nd April 2017
2. About me ‘omerio’ • Co-founder GDG Reading, UK • Google Developer Expert (Cloud) Google • 16+ years as Tech Consultant (UK, Middle East & South East Asia) • PhD Computer Science, University of Reading (Cloud + Big Data) • Blog at omerio.com, Tweets @omerio
3. Content • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
4. Google Cloud Platform Family • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
5. On-Prem, IaaS, PaaS, SaaS! Source: https://www.linkedin.com/pulse/20140730172610-9679881-pizza-as-a-service
6. Compute Compute Engine App Engine Container Engine Container Registry Cloud Functions Networking Cloud Virtual Network Cloud Load Balancing Cloud CDN Cloud Interconnect Cloud DNS Big Data BigQuery Cloud Dataflow Cloud Dataproc Cloud Datalab Cloud Pub/Sub Genomics Storage and Databases Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Cloud Spanner Identity & Security Cloud IAM Cloud Resource Manager Cloud Security Scanner Key Management Service Data Loss Prevention Identity-Aware Proxy Security Key Enforcement Persistent Disk Machine Learning Cloud Machine Learning Cloud Vision API Cloud Speech API Cloud Natural Language API Cloud Translation API Cloud Jobs API Networking BETA BETA BETA BETA
7. Google Grade Security Data Loss Prevention Security starting from the hardware (Titan Chip)
8. Management Tools Stackdriver Monitoring Logging Error Reporting Trace Debugger Cloud Deployment Manager Cloud Endpoints Cloud Console Developer Tools Cloud SDK Cloud Deployment Manager Cloud Source Repositories Cloud Tools for Android Studio Cloud Tools for IntelliJ Cloud Tools for PowerShell Cloud Tools for Visual Studio Google Plug-in for Eclipse Cloud Test Lab Cloud Shell
9. Compute • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
10. Compute Engine IaaS Run VMs on Google Infrastructure Predefined & Custom machine types Compute, Storage & Network Sub hour billing Up to 64 cores and 416 GB memory
11. App Engine PaaS You, your app and Google Standard: Java, Python, PHP & Go Flexible: .Net, Node.js or your own Free quota Auto scaling Massively scalable Lego-like services
12. Powered by App Engine • Khan Academy ◦ Non for profit educational organisation ◦ 10 million unique users / month, 300+ million lessons, 5000+ videos • Snapchat ◦ Photo and video messaging app ◦ 100 million active users / day, 8,796 photos shared / second, 7 billion video views / day • Rovio ◦ Mobile / Web games ◦ 140+ million downloads, 13+ million users / month
13. Container Engine Infrastructure as Code Docker Support Fully Managed Kubernetes Clusters Highly Scalable Private Container Registry
14. Cloud Function Serverless Model Integrated with: PubSub, Datastore, etc. Mobile Ready: Execute from Firebase Support for Node.js Pay only for function execution BETA Use Cases • Mobile Backend • APIs, Microservices • IoT • HTTP Hooks • ETL
15. Storage & Databases • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
16. Cloud Storage IaaS Multi-petabytes Object storage Globally replicated Highly available and reliable Flexible security models
17. Cloud SQL Relational MySQL database Fully managed Highly available Auto backup MySQL + PostgresQL
18. Cloud Datastore NoSQL like solution Fully managed Highly available and scalable Flexible access Autoscaled
19. BigQuery Columnar Database Big Data Warehouse SQL like queries Analyse terabytes of data Streaming data Batch processing
20. Cloud Spanner Relational Database Horizontally scalable High performance & availability Automatic replication Fully managed BETA
21. Cloud Bigtable Massively scalable NoSQL Low latency, High throughput Good for both operational + analytical Seamless cluster resizing Fully managed
24. Building your own solutions, is like … LEGO
25. Big Data Processing • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
26. Big Data & Challenges • Big data is data that is so large and complex that it becomes difficult to process using traditional tools. • Widely characterised in terms of three dimensions (3Vs): ◦ Volume: massive datasets (terabytes, petabytes,...) ◦ Variety: different kinds of data (structured, semi-structured and unstructured). ◦ Velocity: the rate at which the data is generated from batch to real time. • Traditional OLTP and data warehouse (ETL, data marts and star schemas) only deals with structured data • Web logs, social media, device logs, Cisco netflow collector, etc. Nothing done all data is archived or deleted • Schema on Write is not suitable for such data • Opportunities lost (Insights, etc.)
27. I’m confused!
28. Big Data Workflow • Speed of Data ◦ Batch ◦ Streaming ▪ Near Real Time ▪ Real Time • Ingest/Capture/Acquire • Process/Organise/Format • Store • Analyze (Machine Learning) • Decide/Use (Machine Learning) Big Data Machine Learning Statistics Data Mining Data Science Data Analytics
29. Big Data on GCP • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
30. Big Data on GCP Cloud Pub/Sub (robust publish/subscribe messaging system) - Injest Cloud Dataflow (realtime data pipeline, batch + stream) - Process BigQuery/Bigtable (HBase compatible) - Store/Analyze Cloud Datalab - Analyze Cloud Dataproc. Managed Spark Hadoop clusters Zero-ops, serverless Big Data architecture
31. Reference Architecture - 1
32. Reference Architecture - 2
33. Example: Retail > Fraud Detection Third-Party Data External Sources Fraud Modeling Processed Events Cloud Bigtable Events Time-Series Streaming Cloud Pub/Sub Transaction Streams Processing Cloud Dataflow Transaction Streams Fraud Analysis BigQuery Spark MLlib Cloud Dataproc Regression Prediction API Applications App Engine Rules Cache Cloud Cache Architecture: Retail > Fraud Detection
34. Kubernetes & Container Engine • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
35. What is a container? • Lightweight environment • Static application environment • No external dependencies • Application isolation • Reusable, runnable artifacts • Loosely coupled • Portable
36. Containers vs. VMs
37. Kubernetes (k8s) /koo-ber-nay'-tace/ From κυβερνήτης: Greek for "helmsman" or "pilot" • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
38. What is Kubernetes? • Containers orchestrator (Docker, Rocket) • Open source • Run on dev machine, own hardware or cloud • Inspired by Google’s experience • Written in Go • Manages applications, not machines
39. Containers Orchestration
40. What can Kubernetes do? • co-locating helper processes, • mounting storage systems (volumes), • application health checking, • replicating application instances, • horizontal auto-scaling, • load balancing, • rolling updates, • and resource monitoring.
41. Cluster Overview • Pods • Containers • Labels • Replication Controllers • Services • Nodes • Kubernetes Master http://omerio.com/2015/12/18/learn-the-kubernetes-key-concepts-in-10-minutes/
42. Pods • Smallest deployable unit • Scheduled to nodes • Hosts containers and volumes • Ephemeral • Each has its own routable IP (no NAT) • Containers in a pod ◦ Tightly coupled ◦ Share IP, port and IPC namespaces ◦ Talk to each other through localhost http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html
43. Pods Questions • How to persist data across container restarts? • How to create multiple copies of a Pod? • How can containers reference each other reliably?
44. Labels • Key/value pairs (tier=frontend, app=myapp) • Metadata with semantic meaning • Membership identifier (services & rc) • Queryable by Selectors (grouping)
45. Replication Controllers • Keeps Pods running • Ensures N copies of Pods are running • Provides direct control of Pods • Uses a Pod template • Determines constituent Pods using Label Selector
46. Replication Controllers
47. Services • An abstraction • Logical grouping of Pods • Determines constituent Pods using Label Selector • Load balancing between constituent Pods • Gets a stable virtual IP, port & DNS name
48. Services
49. A few more... • Volumes • Secrets • Namespaces • Resource based scheduling ... spec: containers: - name: locust image: gcr.io/rabbit-skateboard/guestbook:gdg-rtv resources: requests: memory: "300Mi" cpu: "300m" limits: memory: "300Mi" cpu: "300m"
50. Google Container Engine • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
51. Hosted Kubernetes Environment • Managed Kubernetes • Manages Kubernetes master • Manages updates • Cluster resize via Managed Instance Groups • Cluster node autoscaling (Cloud Monitoring) • Centralised logging • Google Cloud VPN support
52. Google Container Engine Overview
53. Getting Started • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
54. Useful tips • Start here: https://cloud.google.com/getting-started/ • Utilise Cloud Shell • Start with Hello World Examples • Progress to to other specific examples • Cloud-first design, avoid life-and-shift • Utilise available service (Don’t reinvent the wheel) • Utilise available boilerplates • Always check the docs • Reference architecture: https://cloud.google.com/docs/tutorials#architecture
55. https://cloud.google.com/free/ • Free Trial: $300 and (3) 12 months • Free Tier extends beyond 12 months too (Always Free)
56. https://cloud.google.com/getting-started/
57. https://codelabs.developers.google.com/?cat=Cloud
58. Demo Time • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
59. Wrap Up • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo Time • Wrap Up
60. What we have covered • Google Cloud Platform ◦ Compute ◦ Storage & Databases • Big Data Processing ◦ Big Data on GCP • Kubernetes & Container Engine ◦ Kubernetes (K8s) ◦ Google Container Engine • Getting Started • Demo
61. Thank you Stay in touch Google+: +OmerDawelbeit Twitter: @omerio LinkedIn: OmerDawelbeit Blog: omerio.com Slides: https://goo.gl/mDmycg