Google Cloud Workflows

Workflows: orchestrate & automate API services with serverless workflows

- Workflows at a glance, benefits, key features, use cases
- UI interface in Google Cloud console
- Deep dive into the Workflows syntax
- Workflows connectors
- Demos

Transcript

1. Proprietary + Confidential
   Workflows
   Orchestrate & automate
   API services with
   serverless workflows
   Guillaume Laforge — @glaforge
   Developer Advocate, Google Cloud
   May 25th 2021
   

   View Slide

2. Proprietary + Confidential
   Workflows at a glance,
   benefits, and key features
   

   View Slide

3. Proprietary + Confidential
   Workflows — At a glance
   Serverless
   Compute
   External
   API’s
   Google
   API’s
   etc...
   Workflows - orchestrate & integrate
   SaaS
   API’s
   Private
   API’s
   Other
   Clouds
   

   View Slide

4. Proprietary + Confidential
   Orchestrate work across
   any services & APIs you use
   Easy-to-use workflow
   orchestration managing the
   work across Google Cloud
   products or any HTTP-based
   APIs, including SaaS or
   private APIs.
   Workflows — Benefits
   Serverless scalability and
   managed infrastructure
   Focus on modeling your
   workflow logic and let
   Workflows completely
   manage the infrastructure
   with rapid scaling.
   Pay-per-use pricing model
   Pay only if your workflows
   run: scale your costs down
   to zero during times of
   inactivity.
   

   View Slide

5. Proprietary + Confidential
   Define workflows with a YAML or JSON syntax.
   Visual representation of your workflows.
   Expression formulas supporting decision points,
   conditional step executions, and operations on variables.
   Passing information between steps with built-in JSON
   parsing and expression-based variable manipulations.
   Workflow definition
   and visualisation
   Built-in decisions and
   conditional step executions
   Passing variable values
   between workflow steps
   Workflows — Features
   

   View Slide

6. Proprietary + Confidential
   Reliable workflow execution
   Low latency of execution
   Workflows — Features
   Execute workflows with reliability required
   for enterprise and line-of-business applications.
   Fast scheduling of workflow executions
   and transitions between tasks.
   Predictable performance with no cold starts.
   

   View Slide

7. Proprietary + Confidential
   Built-in authentication for
   Google Cloud products
   Support for external API calls
   Built-in error handling
   Workflows — Features
   Orchestrate work of any Google Cloud product
   without worrying about authentication. Use a proper
   service account and let Workflows do the rest.
   Out-of-the-box support for calls to API endpoints
   outside of Google Cloud.
   Out of the box error handling for your workflow steps
   with configurable retry policies.
   

   View Slide

8. Proprietary + Confidential
   Workflows — Use cases
   Reliable
   transactions
   Low-latency, conditional processes
   with 3rd party integration
   IT infrastructure
   automation
   

   View Slide

9. Proprietary + Confidential
   Workflows
   UI interface
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. Proprietary + Confidential
    Gcloud commands
    # Deploy a workflow
    gcloud workflows deploy my-workflow \
    --source=workflow.yaml
    # Execute a workflow
    gcloud workflows execute my-workflow
    # See the result
    gcloud workflows executions \
    describe \
    --workflow my-workflow
    Deploy and execute
    a workflow.
    Inspect the result of
    the execution
    of a workflow.
    

    View Slide

19. Proprietary + Confidential
    Workflows syntax
    deep dive
    

    View Slide

20. Proprietary + Confidential
    Sequences of steps
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    - shipItems:
    call: http.post
    args:
    url: https://.../cloudfunctions.net/ship
    body:
    address: ${processResult.body.address}
    result: shipResult
    - notifyUser:
    call: http.post
    ...
    

    View Slide

21. Proprietary + Confidential
    Variable passing &
    JSON parsing
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    - shipItems:
    call: http.post
    args:
    url: https://.../cloudfunctions.net/ship
    body:
    address: ${processResult.body.address}
    result: shipResult
    - notifyUser:
    call: http.post
    ...
    

    View Slide

22. Proprietary + Confidential
    Calling HTTP APIs
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    - shipItems:
    call: http.post
    args:
    url: https://.../cloudfunctions.net/ship
    body:
    address: ${processResult.body.address}
    result: shipResult
    - notifyUser:
    call: http.post
    ...
    

    View Slide

23. Proprietary + Confidential
    Authentication
    (OAuth2 | OIDC)
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    auth:
    type: OIDC
    result: processResult
    ...
    AUTHENTICATION
    

    View Slide

24. Proprietary + Confidential
    Pause
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    WAIT
    - pause:
    call: sys.sleep
    args:
    seconds: 60
    

    View Slide

25. Proprietary + Confidential
    Logging
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    LOG
    - log-processed:
    call: sys.log
    args:
    text: "Payment processed"
    severity: INFO
    

    View Slide

26. Proprietary + Confidential
    base64
    ● encode
    ● decode
    text
    ● encode
    ● decode
    ● substring
    json
    ● encode
    ● encode_to_string
    ● decode
    retry
    ● always
    ● never
    Built-in functions
    sys
    ● get_env
    ● sleep
    ● now
    ● log
    errors
    ● type_error
    ● value_error
    ● index_error
    ● key_error
    ● not_implemented_error
    ● recursion_error
    ● zero_division_error
    ● system_error
    ● timeout_error
    ● resource_limit_error
    http
    ● http_error
    ● connection_error
    ● auth_error
    ● get
    ● post
    ● put
    ● patch
    ● delete
    ● request
    ● default_retry_predicate
    experimental.executions
    ● execution_error
    ● run
    ● map
    

    View Slide

27. Proprietary + Confidential
    Error handling,
    conditionals, jumps
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    try:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    except:
    as: e
    steps:
    - known_errors:
    switch:
    - condition: ${not("HttpError" in e.tags)}
    next: connectionError
    - condition: ${e.code == 404}
    return: "Sorry, URL wasn't found."
    - unhandled_exception:
    raise: ${e}
    ERROR
    CHECKING
    

    View Slide

28. Proprietary + Confidential
    Retry & backoff
    Payment Processor
    Cloud Run
    Authorize & charge CC
    Notifier
    Cloud Run
    Notify user
    Shipper
    Cloud Functions
    Prepare & ship items
    - processPayment:
    try:
    call: http.post
    args:
    url: https://payment-processor.run.app/...
    body:
    input: ${paymentDetails}
    result: processResult
    retry:
    max_retries: 5
    backoff:
    initial_delay: 1
    max_delay: 60
    multiplier: 2
    MAX: 5 times
    BACKOFF
    

    View Slide

29. Proprietary + Confidential
    Like programming language
    subroutines or functions
    Subworkflows main:
    steps:
    - call_fullname:
    call: get_fullname
    args:
    first_name: "Sherlock"
    last_name: "Holmes"
    result: output
    - return_message:
    return: ${output}
    get_fullname:
    params: [first_name, last_name]
    steps:
    - prepMessage:
    return: ${first_name + " " + last_name}
    

    View Slide

30. Proprietary + Confidential
    Connectors
    

    View Slide

31. Proprietary + Confidential
    Simplifies access to Google Cloud products within a workflow:
    ● No need to tweak the URLs to call, or specify authentication
    ● Transparent handling of errors and retries
    (improves reliability and service SLA through retries)
    ● Handles long-running operations
    (transparent polling till result is ready, using a backoff)
    Connectors
    

    View Slide

32. Proprietary + Confidential
    BigQuery
    Cloud Build
    Cloud Functions
    Cloud Natural Language
    Cloud Scheduler
    Cloud SQL
    Cloud Storage
    Cloud Tasks
    Cloud Translation
    Existing connectors
    Compute Engine
    Dataflow
    Firestore
    Google Kubernetes Engine
    Pub/Sub
    Secret Manager
    Storage Transfer Service
    Workflows & Workflows execution
    

    View Slide

33. Proprietary + Confidential
    Stopping a Compute Engine
    VM without connectors
    Need to poll till the VM
    is really stopped
    Connector example - stop_machine:
    try:
    call: http.post
    args:
    url: ${"https://compute.googleapis.com/compute/v1/projects/"+project+"/zones/"+zone+"/instances/"+instanceName+"/stop"}
    auth:
    type: OAuth2
    result: stop_resp
    retry: ${http.default_retry}
    - check_status:
    try:
    steps:
    - sleep:
    call: sys.sleep
    args:
    seconds: ${polling_delay}
    - adjust_delay:
    assign:
    - polling_delay: ${polling_delay * multiplier}
    - poll_status:
    call: http.get
    args:
    url: ${stop_resp.body.selfLink}
    auth:
    type: OAuth2
    result: status_resp
    - compare:
    switch:
    - condition: ${status_resp.body.status == "DONE"}
    next: successfully_stopped
    - condition: ${status_resp.body.status == "RUNNING" or status_resp.body.status == "PENDING"}
    next: poll_status
    - condition: ${"error" in status_resp.body}
    next: failed
    retry: ${http.default_retry}
    - successfully_stopped:
    return: "VM instance successfully stopped!"
    - failed:
    return: ${status_resp.body.error}
    

    View Slide

34. Proprietary + Confidential
    Stopping a Compute Engine
    VM with the connector
    No need to poll: the connector
    waits for the end of a “long
    running operation”
    Connector example
    - stop_machine:
    call: googleapis.compute.v1.instances.stop
    args:
    instance: ${instanceName}
    project: ${project}
    zone: ${zone}
    # Optional connector parameters
    connector_params:
    timeout: 100
    polling_policy:
    initial_delay: 1
    multiplier: 1.25
    

    View Slide

35. Proprietary + Confidential
    Workflows in action!
    

    View Slide

36. Proprietary + Confidential
    ● How to invoke a workflow execution programmatically
    ○ Using the multi-language client libraries
    ● How to schedule a workflow execution
    ○ Thanks to Cloud Scheduler
    ● How to use functions to palliate the lack of expressiveness
    ○ Taking advantage of Cloud Functions for logic impossible with Workflows
    ● How to access secrets from Secret Manager
    ○ Avoid hard-coding secrets
    ● How to send a Pub/Sub message
    Workflows in action!
    

    View Slide

37. Proprietary + Confidential
    Márton Kodok (Google Developer Expert)
    shared concrete use cases with Workflows
    https://martonkodok.medium.com/
    ● Automate the execution of BigQuery queries with Cloud Workflows
    ● Firestore backups the easy way with Cloud Workflows
    ● Run shell commands and orchestrate Compute Engine VMs with Cloud Workflows
    ● Using Cloud Workflows to load Cloud Storage files into BigQuery
    Concrete examples from the Community
    

    View Slide

38. Proprietary + Confidential
    Guillaume Laforge — @glaforge
    Developer Advocate, Google Cloud
    May 2021
    Thanks for your attention!
    Q & A
    

    View Slide