[Mandy Waite] Cloud Platform end to end- How we built Cloud Spin

Presentation from GDG DevFest Ukraine 2015 - the biggest Google related event in the country. October 23-24, Lviv. Learn more at http://devfest.gdg.org.ua/

Transcript

1. Cloud Platform End to End or “How we built Cloud Spin”
2. Mandy Waite Developer Advocate +MandyWaite @tekgrrl
3. Cloud Spin How we got all Matrix-y in the Cloud
4. #dfua, @tekgrrl, @googlecloud Resulting Image
5. #dfua, @tekgrrl, @googlecloud backdrop (16 ft x 14 ft) 2 ft 8 ft 8 ft 8 ft 19 x Nexus 6 Engineer’s View 10 Degrees Large Display
7. Fred steps into the center of the Cloud Spin area and someone starts the process from the tablet app
8. The tablet notifies the phones and each phone starts to record a short video, countdown begins
9. Fred is told to jump, another button is pressed on the tablet and an audio tone is sent to the phones
10. #dfua, @tekgrrl, @googlecloud Video of Cloud Spin
11. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Backend Architecture
12. Video Uploads and Notifications
13. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q Notification Processor Extractor Stitcher Video Uploads Notification Output Bucket Coordinator Video Uploads and notifications
14. #dfua, @tekgrrl, @googlecloud • Storage for large amounts of unstructured data • Immediate access and strong consistency on reads • Notifications fired for all newly arrived objects Requirements
15. #dfua, @tekgrrl, @googlecloud Object Store Change Notifications Store any amount of data, up to TBs per Object. Data encrypted at rest. Has full versioning, change notifications and resumable uploads and downloads 99.9% SLA through highly available, geo-redundant data-replication system. Strong read-after-write consistency for each object Reliable and Consistent Google Cloud Storage
16. #dfua, @tekgrrl, @googlecloud Standard Storage Durable Reduced Availability Storage speed availability durability cost Nearline Universal cloud storage suitable for any workload speed availability durability cost speed availability durability cost For use cases that don’t require high availability and high performance For long term storage of infrequently accessed content Cloud Storage Options
17. #dfua, @tekgrrl, @googlecloud Cloud Storage Bucket [myapp-input-bucket] $ gsutil notification watchbucket \ https://myapp.appspot.com gs://myapp-input-bucket Notification processing endpoint https://myapp.appspot.com Object Change Notifications
18. #dfua, @tekgrrl, @googlecloud Cloud Storage Bucket [myapp-input-bucket] Client Notification processing endpoint Add/update/remove object Object Change Notifications
19. #dfua, @tekgrrl, @googlecloud Cloud Storage Bucket [myapp-input-bucket] Notification processing endpoint Object Change Notification POST /ApplicationUrlPath Content-Length: 1097 Content-Type: application/json; charset="utf-8" Host: ApplicationUrlHost X-Goog-Resource-Uri: https://www.googleapis.com/storage/v1/b/BucketName/o? alt=json { "kind": "storage#object", "id": "BucketName/ObjectName", "selfLink": "https://www.googleapis.com/storage/v1/b/BucketName/o/ObjectName", "name": "ObjectName", "bucket": "BucketName", "generation": "1367014943964000", "metageneration": "1", "contentType": "binary/octet-stream", "updated": "2013-04-26T22:22:23.832Z", "size": "10", "md5Hash": "xHZY0QLVuYng2gnOQD90Yw==", } Object Change Notifications
20. #dfua, @tekgrrl, @googlecloud Edge Caching
21. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Summary
22. #dfua, @tekgrrl, @googlecloud Videos Uploaded
23. Notification processing
24. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Notification Processing
25. #dfua, @tekgrrl, @googlecloud • Handle HTTPS POST Requests • Simple Authentication • Process requests and push tasks on to a queue Requirements
26. #dfua, @tekgrrl, @googlecloud Choice of Runtimes - Java, Python, Go, PHP - NodeJS (Beta) - Custom (Containers) Easy to develop - Build and test locally - Focus on App Code - Versioning - Traffic Splitting Trivial to manage - Fully managed - No patches/updates - 24x7 operation by Google SREs - Autoscale Google App Engine
27. #dfua, @tekgrrl, @googlecloud Sandbox webserver user code sandbox runtime VM Container webserver user code optional packages standard/custom runtime Instances App Engine Hosting Environment VM-based Hosting Environment App Engine Hosting Environments
28. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q Go Application on Google App Engine Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator + Summary
29. Messaging and Pipelines
30. #dfua, @tekgrrl, @googlecloud Throughput Availability • Buffer new requests during outages • Prevent overloads that cause outages • Redirect requests to recover from outages • Smooth out spikes in new request rate • Balance load across multiple workers • Balance arrival rate with service rate Messaging is a shock-absorber
31. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Building Pipelines
32. #dfua, @tekgrrl, @googlecloud • Asynchronous Messaging • Pull Subscriptions Requirements
33. #dfua, @tekgrrl, @googlecloud Reliable and real-time messaging Designed for Fast Data Provides reliable, real-time, many-to-many, asynchronous messaging between applications Send data into processing pipelines and out to other apps, devices and Google Cloud Services Global by design and highly available. ‘fire-and-forget’ with minimal latency. Redundant forwarding paths Designed for Google scale Cloud Pub/Sub
34. #dfua, @tekgrrl, @googlecloud Pub/Sub Topic B Subscriber X Publisher B Topic A Publisher A Topic C Publisher C message 1 message 1 message 3 message 2 message 2 Subscriber Y message 3 Subscriber Z message 3 Subscription XA Subscription XB Subscription YC Subscription ZC Publishers and Subscribers
35. #dfua, @tekgrrl, @googlecloud Pub/Sub Topics Pull Subscriptions Push Subscriptions Google Cloud Platform apps 3rd Party network apps Desktop / command line apps Mobile apps Javascript / html5 clients Publisher and Subscriber Endpoints
36. #dfua, @tekgrrl, @googlecloud Extract Topic Pub/Sub Extractor Extractor Extractor Extractor Shared Subscription App Engine Extraction Tasks Use Case: Work Queue (Frame Extraction)
37. #dfua, @tekgrrl, @googlecloud Frames Extracted
38. #dfua, @tekgrrl, @googlecloud Stitch Topic Pub/Sub Stitcher Use Case: Work Queue (Stitching) Extractor Extractor Extractor Extractor Stitching Messages
39. #dfua, @tekgrrl, @googlecloud Frames Stitched
40. Pipeline Processing
41. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Pipeline Processing
42. #dfua, @tekgrrl, @googlecloud • Long running service with on-demand scaling of workers based on queue depth • Able to act as a secure pub/sub subscriber • Simple to develop, deploy and manage Requirements
43. #dfua, @tekgrrl, @googlecloud Compute Resources - US, Europe and Asia zones - Fast SDN-Based virtual networking Consistently Fast - Fast VM Provisioning - Consistent Performance Cost Effective - Sub-Hour Billing - No IOPS charges for Block Storage - VMs not required for Load balancing Google Compute Engine
44. #dfua, @tekgrrl, @googlecloud 1 2 4 8 (Shared) 16 32 Compute Engine VMs: • Debian, CentOS, SUSE, RHEL, Ubuntu, Windows • shared core or 1 - 32 cores • Up to 208 GB of RAM VM Instances to suit diverse workloads
45. #dfua, @tekgrrl, @googlecloud create/ destroy VMs Instance Group Manager Instance Template Instance Template Managed Instance Group VM VM VM Instance Groups • Grouping of identical VM instances • Provisioned and monitored by Instance Group Manager • Instance Templates separate configuration from provisioning Managed Instance Group
46. #dfua, @tekgrrl, @googlecloud Managed Instance Group create/ destroy VMs Instance Group Manager actuator monitor Autoscaler Cloud Monitoring utilisation VM VM VM Pub/Sub Topic Compute Engine Autoscaler
47. #dfua, @tekgrrl, @googlecloud Extractor Workers create/ destroy VMs Instance Group Manager actuator monitor Autoscaler Cloud Monitoring utilisation VM VM Container VM Extract Topic Containerization
48. Wrap-up
49. #dfua, @tekgrrl, @googlecloud Input Bucket Stitching Q Devices Extraction Q Notification Processor Extractor Stitcher Video Uploads Object Change Notifications Output Bucket Coordinator Summary
50. Coordination
51. #dfua, @tekgrrl, @googlecloud Extract Q Temp Store Extractor Stitch Q Extractor Extractors READY Stitcher Online DB All READY? Retrieve Images Coordination
52. #dfua, @tekgrrl, @googlecloud • Cloud based Data Storage • No Server Side Coding • Updates pushed to Clients in Real Time Requirements
53. #dfua, @tekgrrl, @googlecloud • NoSQL, JSON database • Pushes updates in milliseconds when things change • Security model allows direct access from client devices • Maps each piece of data to a URL The Firebase Realtime Database
54. Demo Time
55. #dfua, @tekgrrl, @googlecloud Uses Kubernetes to deploy, run and manage Docker containers Spin up clusters of VMs purpose-built for containers Any Docker file can be installed across the cluster in one command Deploy a container as easily as you can install a mobile app Future: Container Engine
56. Wrap-up
57. #dfua, @tekgrrl, @googlecloud Summary and Hex Test
58. #dfua, @tekgrrl, @googlecloud • Bret McGowen • Francesc Campoy • Kathy Kam • Ray Tsang Demo Builders
59. Thanks! @googlecloudspin
60. #dfua, @tekgrrl, @googlecloud Subscriber Subscriber Pub/Sub Pub/Sub Push Subscription Pull Subscription msg ack msg ack RPC Send RPC Return