Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Platform end to end: How we built Cloud Spin

Cloud Platform end to end: How we built Cloud Spin

Version of the GCP Next Google Cloud Platform overview talk given at DevFest Ukraine 2015

42ff40d0d729d86739df5ede5591d4f1?s=128

GoogleCloudPlatform

October 24, 2015
Tweet

Transcript

  1. Cloud Platform End to End or “How we built Cloud

    Spin”
  2. Mandy Waite Developer Advocate +MandyWaite @tekgrrl

  3. Cloud Spin How we got all Matrix-y in the Cloud

  4. #dfua, @tekgrrl, @googlecloud Resulting Image https://twitter.com/francesc/status/649405037303631873

  5. #dfua, @tekgrrl, @googlecloud backdrop (16 ft x 14 ft) 2

    ft 8 ft 8 ft 8 ft 19 x Nexus 6 Engineer’s View 10 Degrees Large Display
  6. None
  7. #dfua, @tekgrrl, @googlecloud Video of Cloud Spin https://www.youtube.com/watch?v=6oGWFiSrPBc

  8. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q

    Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Backend Architecture
  9. Video Uploads and Notifications

  10. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q

    Notification Processor Extractor Stitcher Video Uploads Notification Output Bucket Coordinator Video Uploads and notifications
  11. #dfua, @tekgrrl, @googlecloud • Storage for large amounts of unstructured

    data • Immediate access and strong consistency on reads • Notifications fired for all newly arrived objects Requirements
  12. #dfua, @tekgrrl, @googlecloud Object Store Change Notifications Store any amount

    of data, up to TBs per Object. Data encrypted at rest. Has full versioning, change notifications and resumable uploads and downloads 99.9% SLA through highly available, geo-redundant data-replication system. Strong read-after-write consistency for each object Reliable and Consistent Google Cloud Storage
  13. #dfua, @tekgrrl, @googlecloud Standard Storage Durable Reduced Availability Storage speed

    availability durability cost Nearline Universal cloud storage suitable for any workload speed availability durability cost speed availability durability cost For use cases that don’t require high availability and high performance For long term storage of infrequently accessed content Cloud Storage Options
  14. #dfua, @tekgrrl, @googlecloud Cloud Storage Bucket [myapp-input-bucket] $ gsutil notification

    watchbucket \ https://myapp.appspot.com gs://myapp-input-bucket Notification processing endpoint https://myapp.appspot.com Object Change Notifications
  15. #dfua, @tekgrrl, @googlecloud Cloud Storage Bucket [myapp-input-bucket] Client Notification processing

    endpoint Add/update/remove object Object Change Notifications
  16. #dfua, @tekgrrl, @googlecloud Cloud Storage Bucket [myapp-input-bucket] Notification processing endpoint

    Object Change Notification POST /ApplicationUrlPath Content-Length: 1097 Content-Type: application/json; charset="utf-8" Host: ApplicationUrlHost X-Goog-Resource-Uri: https://www.googleapis.com/storage/v1/b/BucketName/o? alt=json { "kind": "storage#object", "id": "BucketName/ObjectName", "selfLink": "https://www.googleapis.com/storage/v1/b/BucketName/o/ObjectName", "name": "ObjectName", "bucket": "BucketName", "generation": "1367014943964000", "metageneration": "1", "contentType": "binary/octet-stream", "updated": "2013-04-26T22:22:23.832Z", "size": "10", "md5Hash": "xHZY0QLVuYng2gnOQD90Yw==", } Object Change Notifications
  17. #dfua, @tekgrrl, @googlecloud Edge Caching

  18. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q

    Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Summary
  19. #dfua, @tekgrrl, @googlecloud Videos Uploaded https://www.youtube.com/watch?v=EtSc9lZ7g74

  20. Notification processing

  21. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q

    Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Notification Processing
  22. #dfua, @tekgrrl, @googlecloud • Handle HTTPS POST Requests • Simple

    Authentication • Process requests and push tasks on to a queue Requirements
  23. #dfua, @tekgrrl, @googlecloud Choice of Runtimes - Java, Python, Go,

    PHP - NodeJS (Beta) - Custom (Containers) Easy to develop - Build and test locally - Focus on App Code - Versioning - Traffic Splitting Trivial to manage - Fully managed - No patches/updates - 24x7 operation by Google SREs - Autoscale Google App Engine
  24. #dfua, @tekgrrl, @googlecloud Sandbox webserver user code sandbox runtime VM

    Container webserver user code optional packages standard/custom runtime Instances App Engine Hosting Environment VM-based Hosting Environment App Engine Hosting Environments
  25. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q

    Go Application on Google App Engine Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator + Summary
  26. Messaging and Pipelines

  27. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q

    Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Building Pipelines
  28. #dfua, @tekgrrl, @googlecloud • Asynchronous Messaging • Pull Subscriptions Requirements

  29. #dfua, @tekgrrl, @googlecloud Reliable and real-time messaging Designed for Fast

    Data Provides reliable, real-time, many-to-many, asynchronous messaging between applications Send data into processing pipelines and out to other apps, devices and Google Cloud Services Global by design and highly available. ‘fire-and-forget’ with minimal latency. Redundant forwarding paths Designed for Google scale Cloud Pub/Sub
  30. #dfua, @tekgrrl, @googlecloud Pub/Sub Topic B Subscriber X Publisher B

    Topic A Publisher A Topic C Publisher C message 1 message 1 message 3 message 2 message 2 Subscriber Y message 3 Subscriber Z message 3 Subscription XA Subscription XB Subscription YC Subscription ZC Publishers and Subscribers
  31. #dfua, @tekgrrl, @googlecloud Pub/Sub Topics Pull Subscriptions Push Subscriptions Google

    Cloud Platform apps 3rd Party network apps Desktop / command line apps Mobile apps Javascript / html5 clients Publisher and Subscriber Endpoints
  32. #dfua, @tekgrrl, @googlecloud Extract Topic Pub/Sub Extractor Extractor Extractor Extractor

    Shared Subscription App Engine Extraction Tasks Use Case: Work Queue (Frame Extraction)
  33. #dfua, @tekgrrl, @googlecloud Frames Extracted

  34. #dfua, @tekgrrl, @googlecloud Stitch Topic Pub/Sub Stitcher Use Case: Work

    Queue (Stitching) Extractor Extractor Extractor Extractor Stitching Messages
  35. #dfua, @tekgrrl, @googlecloud Frames Stitched

  36. Pipeline Processing

  37. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q

    Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Pipeline Processing
  38. #dfua, @tekgrrl, @googlecloud • Long running service with on-demand scaling

    of workers based on queue depth • Able to act as a secure pub/sub subscriber • Simple to develop, deploy and manage Requirements
  39. #dfua, @tekgrrl, @googlecloud Compute Resources - US, Europe and Asia

    zones - Fast SDN-Based virtual networking Innovative - Live Migration - Preemptible VMs - SSD + Local SSD Cost Effective - Sub-Hour Billing - No IOPS charges PD - Sustained Usage Discounts (with no upfront charges) Google Compute Engine
  40. #dfua, @tekgrrl, @googlecloud 1 2 4 8 (Shared) 16 32

    Compute Engine VMs: • Debian, CentOS, SUSE, RHEL, Ubuntu, Windows • shared core or 1 - 32 cores • Up to 208 GB of RAM VM Instances to suit diverse workloads
  41. #dfua, @tekgrrl, @googlecloud create/ destroy VMs Instance Group Manager Instance

    Template Instance Template Managed Instance Group VM VM VM Instance Groups • Grouping of identical VM instances • Provisioned and monitored by Instance Group Manager • Instance Templates separate configuration from provisioning Managed Instance Group
  42. #dfua, @tekgrrl, @googlecloud Managed Instance Group create/ destroy VMs Instance

    Group Manager actuator monitor Autoscaler Cloud Monitoring utilisation VM VM VM Pub/Sub Topic Compute Engine Autoscaler
  43. #dfua, @tekgrrl, @googlecloud Extractor Workers create/ destroy VMs Instance Group

    Manager actuator monitor Autoscaler Cloud Monitoring utilisation VM VM Container VM Extract Topic Containerization
  44. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q

    Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Summary
  45. Coordination

  46. #dfua, @tekgrrl, @googlecloud Extract Q Temp Store Extractor Stitch Q

    Extractor Extractors READY Stitcher Online DB All READY? Retrieve Images Coordination
  47. #dfua, @tekgrrl, @googlecloud • Cloud based Data Storage • No

    Server Side Coding • Updates pushed to Clients in Real Time Requirements
  48. #dfua, @tekgrrl, @googlecloud • NoSQL, JSON database • Pushes updates

    in milliseconds when things change • Security model allows direct access from client devices • Maps each piece of data to a URL The Firebase Realtime Database
  49. Demo Time

  50. #dfua, @tekgrrl, @googlecloud Uses Kubernetes to deploy, run and manage

    Docker containers Spin up clusters of VMs purpose-built for containers Any Docker file can be installed across the cluster in one command Deploy a container as easily as you can install a mobile app Future: Container Engine
  51. Wrap-up

  52. #dfua, @tekgrrl, @googlecloud Summary and Hex Test

  53. #dfua, @tekgrrl, @googlecloud • Bret McGowen • Francesc Campoy •

    Kathy Kam • Ray Tsang Demo Builders
  54. #dfua, @tekgrrl, @googlecloud If You Only Remember Three Things… Full

    Platform Complete set of services for building, hosting, and managing apps and services Leading Infrastructure Industry-leading hardware and services unavailable on-prem or other clouds Innovation Platform Be faster to market, respond rapidly to change, and focus on what matters to you.
  55. Thanks! @googlecloudspin

  56. #dfua, @tekgrrl, @googlecloud Subscriber Subscriber Pub/Sub Pub/Sub Push Subscription Pull

    Subscription msg ack msg ack RPC Send RPC Return