Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Google Cloud Workflows

Google Cloud Workflows

Workflows: orchestrate & automate API services with serverless workflows

- Workflows at a glance, benefits, key features, use cases
- UI interface in Google Cloud console
- Deep dive into the Workflows syntax
- Workflows connectors
- Demos

Guillaume Laforge

June 04, 2021
Tweet

More Decks by Guillaume Laforge

Other Decks in Technology

Transcript

  1. Proprietary + Confidential
    Workflows
    Orchestrate & automate
    API services with
    serverless workflows
    Guillaume Laforge — @glaforge
    Developer Advocate, Google Cloud
    May 25th 2021

    View full-size slide

  2. Proprietary + Confidential
    Workflows at a glance,
    benefits, and key features

    View full-size slide

  3. Proprietary + Confidential
    Workflows — At a glance
    Serverless
    Compute
    External
    API’s
    Google
    API’s
    etc...
    Workflows - orchestrate & integrate
    SaaS
    API’s
    Private
    API’s
    Other
    Clouds

    View full-size slide

  4. Proprietary + Confidential
    Orchestrate work across
    any services & APIs you use
    Easy-to-use workflow
    orchestration managing the
    work across Google Cloud
    products or any HTTP-based
    APIs, including SaaS or
    private APIs.
    Workflows — Benefits
    Serverless scalability and
    managed infrastructure
    Focus on modeling your
    workflow logic and let
    Workflows completely
    manage the infrastructure
    with rapid scaling.
    Pay-per-use pricing model
    Pay only if your workflows
    run: scale your costs down
    to zero during times of
    inactivity.

    View full-size slide

  5. Proprietary + Confidential
    Define workflows with a YAML or JSON syntax.
    Visual representation of your workflows.
    Expression formulas supporting decision points,
    conditional step executions, and operations on variables.
    Passing information between steps with built-in JSON
    parsing and expression-based variable manipulations.
    Workflow definition
    and visualisation
    Built-in decisions and
    conditional step executions
    Passing variable values
    between workflow steps
    Workflows — Features

    View full-size slide

  6. Proprietary + Confidential
    Reliable workflow execution
    Low latency of execution
    Workflows — Features
    Execute workflows with reliability required
    for enterprise and line-of-business applications.
    Fast scheduling of workflow executions
    and transitions between tasks.
    Predictable performance with no cold starts.

    View full-size slide

  7. Proprietary + Confidential
    Built-in authentication for
    Google Cloud products
    Support for external API calls
    Built-in error handling
    Workflows — Features
    Orchestrate work of any Google Cloud product
    without worrying about authentication. Use a proper
    service account and let Workflows do the rest.
    Out-of-the-box support for calls to API endpoints
    outside of Google Cloud.
    Out of the box error handling for your workflow steps
    with configurable retry policies.

    View full-size slide

  8. Proprietary + Confidential
    Workflows — Use cases
    Reliable
    transactions
    Low-latency, conditional processes
    with 3rd party integration
    IT infrastructure
    automation

    View full-size slide

  9. Proprietary + Confidential
    Workflows
    UI interface

    View full-size slide

  10. Proprietary + Confidential
    Gcloud commands
    # Deploy a workflow
    gcloud workflows deploy my-workflow \
    --source=workflow.yaml
    # Execute a workflow
    gcloud workflows execute my-workflow
    # See the result
    gcloud workflows executions \
    describe \
    --workflow my-workflow
    Deploy and execute
    a workflow.
    Inspect the result of
    the execution
    of a workflow.

    View full-size slide

  11. Proprietary + Confidential
    Workflows syntax
    deep dive

    View full-size slide

  12. Proprietary + Confidential
    Sequences of steps
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    - shipItems:
    call: http.post
    args:
    url: https://.../cloudfunctions.net/ship
    body:
    address: ${processResult.body.address}
    result: shipResult
    - notifyUser:
    call: http.post
    ...

    View full-size slide

  13. Proprietary + Confidential
    Variable passing &
    JSON parsing
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    - shipItems:
    call: http.post
    args:
    url: https://.../cloudfunctions.net/ship
    body:
    address: ${processResult.body.address}
    result: shipResult
    - notifyUser:
    call: http.post
    ...

    View full-size slide

  14. Proprietary + Confidential
    Calling HTTP APIs
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    - shipItems:
    call: http.post
    args:
    url: https://.../cloudfunctions.net/ship
    body:
    address: ${processResult.body.address}
    result: shipResult
    - notifyUser:
    call: http.post
    ...

    View full-size slide

  15. Proprietary + Confidential
    Authentication
    (OAuth2 | OIDC)
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    auth:
    type: OIDC
    result: processResult
    ...
    AUTHENTICATION

    View full-size slide

  16. Proprietary + Confidential
    Pause
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    WAIT
    - pause:
    call: sys.sleep
    args:
    seconds: 60

    View full-size slide

  17. Proprietary + Confidential
    Logging
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    LOG
    - log-processed:
    call: sys.log
    args:
    text: "Payment processed"
    severity: INFO

    View full-size slide

  18. Proprietary + Confidential
    base64
    ● encode
    ● decode
    text
    ● encode
    ● decode
    ● substring
    json
    ● encode
    ● encode_to_string
    ● decode
    retry
    ● always
    ● never
    Built-in functions
    sys
    ● get_env
    ● sleep
    ● now
    ● log
    errors
    ● type_error
    ● value_error
    ● index_error
    ● key_error
    ● not_implemented_error
    ● recursion_error
    ● zero_division_error
    ● system_error
    ● timeout_error
    ● resource_limit_error
    http
    ● http_error
    ● connection_error
    ● auth_error
    ● get
    ● post
    ● put
    ● patch
    ● delete
    ● request
    ● default_retry_predicate
    experimental.executions
    ● execution_error
    ● run
    ● map

    View full-size slide

  19. Proprietary + Confidential
    Error handling,
    conditionals, jumps
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    try:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    except:
    as: e
    steps:
    - known_errors:
    switch:
    - condition: ${not("HttpError" in e.tags)}
    next: connectionError
    - condition: ${e.code == 404}
    return: "Sorry, URL wasn't found."
    - unhandled_exception:
    raise: ${e}
    ERROR
    CHECKING

    View full-size slide

  20. Proprietary + Confidential
    Retry & backoff
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    try:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    retry:
    max_retries: 5
    backoff:
    initial_delay: 1
    max_delay: 60
    multiplier: 2
    MAX: 5 times
    BACKOFF

    View full-size slide

  21. Proprietary + Confidential
    Like programming language
    subroutines or functions
    Subworkflows main:
    steps:
    - call_fullname:
    call: get_fullname
    args:
    first_name: "Sherlock"
    last_name: "Holmes"
    result: output
    - return_message:
    return: ${output}
    get_fullname:
    params: [first_name, last_name]
    steps:
    - prepMessage:
    return: ${first_name + " " + last_name}

    View full-size slide

  22. Proprietary + Confidential
    Connectors

    View full-size slide

  23. Proprietary + Confidential
    Simplifies access to Google Cloud products within a workflow:
    ● No need to tweak the URLs to call, or specify authentication
    ● Transparent handling of errors and retries
    (improves reliability and service SLA through retries)
    ● Handles long-running operations
    (transparent polling till result is ready, using a backoff)
    Connectors

    View full-size slide

  24. Proprietary + Confidential
    BigQuery
    Cloud Build
    Cloud Functions
    Cloud Natural Language
    Cloud Scheduler
    Cloud SQL
    Cloud Storage
    Cloud Tasks
    Cloud Translation
    Existing connectors
    Compute Engine
    Dataflow
    Firestore
    Google Kubernetes Engine
    Pub/Sub
    Secret Manager
    Storage Transfer Service
    Workflows & Workflows execution

    View full-size slide

  25. Proprietary + Confidential
    Stopping a Compute Engine
    VM without connectors
    Need to poll till the VM
    is really stopped
    Connector example - stop_machine:
    try:
    call: http.post
    args:
    url: ${"https://compute.googleapis.com/compute/v1/projects/"+project+"/zones/"+zone+"/instances/"+instanceName+"/stop"}
    auth:
    type: OAuth2
    result: stop_resp
    retry: ${http.default_retry}
    - check_status:
    try:
    steps:
    - sleep:
    call: sys.sleep
    args:
    seconds: ${polling_delay}
    - adjust_delay:
    assign:
    - polling_delay: ${polling_delay * multiplier}
    - poll_status:
    call: http.get
    args:
    url: ${stop_resp.body.selfLink}
    auth:
    type: OAuth2
    result: status_resp
    - compare:
    switch:
    - condition: ${status_resp.body.status == "DONE"}
    next: successfully_stopped
    - condition: ${status_resp.body.status == "RUNNING" or status_resp.body.status == "PENDING"}
    next: poll_status
    - condition: ${"error" in status_resp.body}
    next: failed
    retry: ${http.default_retry}
    - successfully_stopped:
    return: "VM instance successfully stopped!"
    - failed:
    return: ${status_resp.body.error}

    View full-size slide

  26. Proprietary + Confidential
    Stopping a Compute Engine
    VM with the connector
    No need to poll: the connector
    waits for the end of a “long
    running operation”
    Connector example
    - stop_machine:
    call: googleapis.compute.v1.instances.stop
    args:
    instance: ${instanceName}
    project: ${project}
    zone: ${zone}
    # Optional connector parameters
    connector_params:
    timeout: 100
    polling_policy:
    initial_delay: 1
    multiplier: 1.25

    View full-size slide

  27. Proprietary + Confidential
    Workflows in action!

    View full-size slide

  28. Proprietary + Confidential
    ● How to invoke a workflow execution programmatically
    ○ Using the multi-language client libraries
    ● How to schedule a workflow execution
    ○ Thanks to Cloud Scheduler
    ● How to use functions to palliate the lack of expressiveness
    ○ Taking advantage of Cloud Functions for logic impossible with Workflows
    ● How to access secrets from Secret Manager
    ○ Avoid hard-coding secrets
    ● How to send a Pub/Sub message
    Workflows in action!

    View full-size slide

  29. Proprietary + Confidential
    Márton Kodok (Google Developer Expert)
    shared concrete use cases with Workflows
    https://martonkodok.medium.com/
    ● Automate the execution of BigQuery queries with Cloud Workflows
    ● Firestore backups the easy way with Cloud Workflows
    ● Run shell commands and orchestrate Compute Engine VMs with Cloud Workflows
    ● Using Cloud Workflows to load Cloud Storage files into BigQuery
    Concrete examples from the Community

    View full-size slide

  30. Proprietary + Confidential
    Guillaume Laforge — @glaforge
    Developer Advocate, Google Cloud
    May 2021
    Thanks for your attention!
    Q & A

    View full-size slide