Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Platform end to end: How we built Cloud Spin

Cloud Platform end to end: How we built Cloud Spin

Version of the GCP Next Google Cloud Platform overview talk given at DevFest Ukraine 2015

GoogleCloudPlatform

October 24, 2015
Tweet

More Decks by GoogleCloudPlatform

Other Decks in Technology

Transcript

  1. Cloud Platform End to End
    or “How we built Cloud Spin”

    View Slide

  2. Mandy Waite
    Developer Advocate
    +MandyWaite
    @tekgrrl

    View Slide

  3. Cloud Spin
    How we got all Matrix-y in the Cloud

    View Slide

  4. #dfua, @tekgrrl, @googlecloud
    Resulting Image
    https://twitter.com/francesc/status/649405037303631873

    View Slide

  5. #dfua, @tekgrrl, @googlecloud
    backdrop (16 ft x 14 ft)
    2 ft
    8 ft
    8 ft
    8 ft
    19 x Nexus 6
    Engineer’s View
    10 Degrees
    Large Display

    View Slide

  6. View Slide

  7. #dfua, @tekgrrl, @googlecloud
    Video of Cloud Spin
    https://www.youtube.com/watch?v=6oGWFiSrPBc

    View Slide

  8. #dfua, @tekgrrl, @googlecloud
    Input
    Bucket
    Stitching Q
    Devices
    Extraction Q
    Notification
    Processor
    Extractor
    Stitcher
    Video
    Uploads
    Object Change
    Notifications
    Output
    Bucket
    Coordinator
    Backend Architecture

    View Slide

  9. Video Uploads and
    Notifications

    View Slide

  10. #dfua, @tekgrrl, @googlecloud
    Input
    Bucket
    Stitching Q
    Devices
    Extraction Q
    Notification
    Processor
    Extractor
    Stitcher
    Video
    Uploads
    Notification
    Output
    Bucket
    Coordinator
    Video Uploads and notifications

    View Slide

  11. #dfua, @tekgrrl, @googlecloud
    ● Storage for large amounts of unstructured data
    ● Immediate access and strong consistency on
    reads
    ● Notifications fired for all newly arrived objects
    Requirements

    View Slide

  12. #dfua, @tekgrrl, @googlecloud
    Object Store
    Change
    Notifications
    Store any amount of data, up
    to TBs per Object. Data
    encrypted at rest.
    Has full versioning, change
    notifications and resumable
    uploads and downloads
    99.9% SLA through highly
    available, geo-redundant
    data-replication system.
    Strong read-after-write
    consistency for each object
    Reliable and
    Consistent
    Google Cloud Storage

    View Slide

  13. #dfua, @tekgrrl, @googlecloud
    Standard Storage
    Durable Reduced
    Availability Storage
    speed
    availability
    durability
    cost
    Nearline
    Universal cloud storage
    suitable for any workload
    speed
    availability
    durability
    cost
    speed
    availability
    durability
    cost
    For use cases that don’t
    require high availability and
    high performance
    For long term storage of
    infrequently accessed content
    Cloud Storage Options

    View Slide

  14. #dfua, @tekgrrl, @googlecloud
    Cloud
    Storage Bucket
    [myapp-input-bucket]
    $ gsutil notification watchbucket \
    https://myapp.appspot.com gs://myapp-input-bucket
    Notification
    processing
    endpoint
    https://myapp.appspot.com
    Object Change Notifications

    View Slide

  15. #dfua, @tekgrrl, @googlecloud
    Cloud
    Storage Bucket
    [myapp-input-bucket]
    Client
    Notification
    processing
    endpoint
    Add/update/remove object
    Object Change Notifications

    View Slide

  16. #dfua, @tekgrrl, @googlecloud
    Cloud
    Storage Bucket
    [myapp-input-bucket]
    Notification
    processing
    endpoint
    Object Change Notification
    POST /ApplicationUrlPath
    Content-Length: 1097
    Content-Type: application/json; charset="utf-8"
    Host: ApplicationUrlHost
    X-Goog-Resource-Uri: https://www.googleapis.com/storage/v1/b/BucketName/o?
    alt=json
    {
    "kind": "storage#object",
    "id": "BucketName/ObjectName",
    "selfLink": "https://www.googleapis.com/storage/v1/b/BucketName/o/ObjectName",
    "name": "ObjectName",
    "bucket": "BucketName",
    "generation": "1367014943964000",
    "metageneration": "1",
    "contentType": "binary/octet-stream",
    "updated": "2013-04-26T22:22:23.832Z",
    "size": "10",
    "md5Hash": "xHZY0QLVuYng2gnOQD90Yw==",
    }
    Object Change Notifications

    View Slide

  17. #dfua, @tekgrrl, @googlecloud
    Edge Caching

    View Slide

  18. #dfua, @tekgrrl, @googlecloud
    Input
    Bucket
    Stitching Q
    Devices
    Extraction Q
    Notification
    Processor
    Extractor
    Stitcher
    Video
    Uploads
    Object Change
    Notifications
    Output
    Bucket
    Coordinator
    Summary

    View Slide

  19. #dfua, @tekgrrl, @googlecloud
    Videos Uploaded
    https://www.youtube.com/watch?v=EtSc9lZ7g74

    View Slide

  20. Notification processing

    View Slide

  21. #dfua, @tekgrrl, @googlecloud
    Input
    Bucket
    Stitching Q
    Devices
    Extraction Q
    Notification
    Processor
    Extractor
    Stitcher
    Video
    Uploads
    Object Change
    Notifications
    Output
    Bucket
    Coordinator
    Notification Processing

    View Slide

  22. #dfua, @tekgrrl, @googlecloud
    ● Handle HTTPS POST Requests
    ● Simple Authentication
    ● Process requests and push tasks on to a queue
    Requirements

    View Slide

  23. #dfua, @tekgrrl, @googlecloud
    Choice of Runtimes
    - Java, Python, Go, PHP
    - NodeJS (Beta)
    - Custom (Containers)
    Easy to develop
    - Build and test locally
    - Focus on App Code
    - Versioning
    - Traffic Splitting
    Trivial to manage
    - Fully managed
    - No patches/updates
    - 24x7 operation by Google SREs
    - Autoscale
    Google App Engine

    View Slide

  24. #dfua, @tekgrrl, @googlecloud
    Sandbox
    webserver user code
    sandbox runtime
    VM
    Container
    webserver user code
    optional packages
    standard/custom runtime
    Instances
    App Engine Hosting Environment VM-based Hosting Environment
    App Engine Hosting Environments

    View Slide

  25. #dfua, @tekgrrl, @googlecloud
    Input
    Bucket
    Stitching Q
    Devices
    Extraction Q
    Go
    Application on
    Google App Engine
    Extractor
    Stitcher
    Video
    Uploads
    Object Change
    Notifications
    Output
    Bucket
    Coordinator
    +
    Summary

    View Slide

  26. Messaging and Pipelines

    View Slide

  27. #dfua, @tekgrrl, @googlecloud
    Input
    Bucket
    Stitching Q
    Devices
    Extraction Q
    Notification
    Processor
    Extractor
    Stitcher
    Video
    Uploads
    Object Change
    Notifications
    Output
    Bucket
    Coordinator
    Building Pipelines

    View Slide

  28. #dfua, @tekgrrl, @googlecloud
    ● Asynchronous Messaging
    ● Pull Subscriptions
    Requirements

    View Slide

  29. #dfua, @tekgrrl, @googlecloud
    Reliable and
    real-time
    messaging
    Designed for
    Fast Data
    Provides reliable, real-time,
    many-to-many, asynchronous
    messaging between
    applications
    Send data into processing
    pipelines and out to other
    apps, devices and Google Cloud
    Services
    Global by design and highly
    available. ‘fire-and-forget’
    with minimal latency.
    Redundant forwarding paths
    Designed for
    Google scale
    Cloud Pub/Sub

    View Slide

  30. #dfua, @tekgrrl, @googlecloud
    Pub/Sub
    Topic B
    Subscriber X
    Publisher B
    Topic A
    Publisher A
    Topic C
    Publisher C
    message 1
    message 1
    message 3
    message 2
    message 2
    Subscriber Y
    message 3
    Subscriber Z
    message 3
    Subscription XA Subscription XB Subscription YC Subscription ZC
    Publishers and Subscribers

    View Slide

  31. #dfua, @tekgrrl, @googlecloud
    Pub/Sub
    Topics
    Pull Subscriptions Push Subscriptions
    Google Cloud Platform apps
    3rd Party network apps
    Desktop / command line apps
    Mobile apps
    Javascript / html5 clients
    Publisher and Subscriber Endpoints

    View Slide

  32. #dfua, @tekgrrl, @googlecloud
    Extract Topic
    Pub/Sub
    Extractor Extractor Extractor Extractor
    Shared Subscription
    App Engine Extraction Tasks
    Use Case: Work Queue (Frame Extraction)

    View Slide

  33. #dfua, @tekgrrl, @googlecloud
    Frames Extracted

    View Slide

  34. #dfua, @tekgrrl, @googlecloud
    Stitch Topic
    Pub/Sub
    Stitcher
    Use Case: Work Queue (Stitching)
    Extractor Extractor Extractor Extractor
    Stitching Messages

    View Slide

  35. #dfua, @tekgrrl, @googlecloud
    Frames Stitched

    View Slide

  36. Pipeline Processing

    View Slide

  37. #dfua, @tekgrrl, @googlecloud
    Input
    Bucket
    Stitching Q
    Devices
    Extraction Q
    Notification
    Processor
    Extractor
    Stitcher
    Video
    Uploads
    Object Change
    Notifications
    Output
    Bucket
    Coordinator
    Pipeline Processing

    View Slide

  38. #dfua, @tekgrrl, @googlecloud
    ● Long running service with on-demand scaling of
    workers based on queue depth
    ● Able to act as a secure pub/sub subscriber
    ● Simple to develop, deploy and manage
    Requirements

    View Slide

  39. #dfua, @tekgrrl, @googlecloud
    Compute Resources
    - US, Europe and
    Asia zones
    - Fast SDN-Based
    virtual networking
    Innovative
    - Live Migration
    - Preemptible VMs
    - SSD + Local SSD
    Cost Effective
    - Sub-Hour Billing
    - No IOPS charges PD
    - Sustained Usage
    Discounts (with no
    upfront charges)
    Google Compute Engine

    View Slide

  40. #dfua, @tekgrrl, @googlecloud
    1
    2
    4
    8
    (Shared)
    16
    32
    Compute Engine VMs:
    ● Debian, CentOS, SUSE, RHEL,
    Ubuntu, Windows
    ● shared core or 1 - 32 cores
    ● Up to 208 GB of RAM
    VM Instances to suit diverse workloads

    View Slide

  41. #dfua, @tekgrrl, @googlecloud
    create/
    destroy VMs
    Instance
    Group
    Manager
    Instance
    Template
    Instance
    Template
    Managed
    Instance
    Group
    VM
    VM
    VM
    Instance Groups
    ● Grouping of identical VM instances
    ● Provisioned and monitored by
    Instance Group Manager
    ● Instance Templates separate
    configuration from provisioning
    Managed Instance Group

    View Slide

  42. #dfua, @tekgrrl, @googlecloud
    Managed
    Instance
    Group
    create/
    destroy VMs
    Instance Group
    Manager
    actuator
    monitor
    Autoscaler
    Cloud Monitoring
    utilisation
    VM
    VM
    VM
    Pub/Sub Topic
    Compute Engine Autoscaler

    View Slide

  43. #dfua, @tekgrrl, @googlecloud
    Extractor
    Workers
    create/
    destroy VMs
    Instance Group
    Manager
    actuator
    monitor
    Autoscaler
    Cloud Monitoring
    utilisation
    VM
    VM
    Container
    VM
    Extract Topic
    Containerization

    View Slide

  44. #dfua, @tekgrrl, @googlecloud
    Input
    Bucket
    Stitching Q
    Devices
    Extraction Q
    Notification
    Processor
    Extractor
    Stitcher
    Video
    Uploads
    Object Change
    Notifications
    Output
    Bucket
    Coordinator
    Summary

    View Slide

  45. Coordination

    View Slide

  46. #dfua, @tekgrrl, @googlecloud
    Extract Q
    Temp
    Store
    Extractor
    Stitch Q
    Extractor
    Extractors
    READY
    Stitcher
    Online DB
    All READY?
    Retrieve Images
    Coordination

    View Slide

  47. #dfua, @tekgrrl, @googlecloud
    ● Cloud based Data Storage
    ● No Server Side Coding
    ● Updates pushed to Clients in Real Time
    Requirements

    View Slide

  48. #dfua, @tekgrrl, @googlecloud
    ● NoSQL, JSON database
    ● Pushes updates in milliseconds when
    things change
    ● Security model allows direct access
    from client devices
    ● Maps each piece of data to a URL
    The Firebase Realtime Database

    View Slide

  49. Demo Time

    View Slide

  50. #dfua, @tekgrrl, @googlecloud
    Uses Kubernetes to deploy, run and manage Docker
    containers
    Spin up clusters of VMs purpose-built for containers
    Any Docker file can be installed across the cluster in
    one command
    Deploy a container as easily as you can install a
    mobile app
    Future: Container Engine

    View Slide

  51. Wrap-up

    View Slide

  52. #dfua, @tekgrrl, @googlecloud
    Summary and Hex Test

    View Slide

  53. #dfua, @tekgrrl, @googlecloud
    ● Bret McGowen
    ● Francesc Campoy
    ● Kathy Kam
    ● Ray Tsang
    Demo Builders

    View Slide

  54. #dfua, @tekgrrl, @googlecloud
    If You Only Remember Three Things…
    Full Platform
    Complete set of services
    for building, hosting, and
    managing apps and
    services
    Leading
    Infrastructure
    Industry-leading
    hardware and services
    unavailable on-prem or
    other clouds
    Innovation Platform
    Be faster to market,
    respond rapidly to
    change, and focus on
    what matters to you.

    View Slide

  55. Thanks!
    @googlecloudspin

    View Slide

  56. #dfua, @tekgrrl, @googlecloud
    Subscriber Subscriber
    Pub/Sub Pub/Sub
    Push Subscription Pull Subscription
    msg ack msg ack
    RPC Send
    RPC Return

    View Slide