Proprietary + Confidential Workflows Orchestrate & automate API services with serverless workflows Guillaume Laforge — @glaforge Developer Advocate, Google Cloud May 25th 2021

Proprietary + Confidential Workflows at a glance, benefits, and key features

Proprietary + Confidential Workflows — At a glance Serverless Compute External API’s Google API’s etc... Workflows - orchestrate & integrate SaaS API’s Private API’s Other Clouds

Proprietary + Confidential Orchestrate work across any services & APIs you use Easy-to-use workflow orchestration managing the work across Google Cloud products or any HTTP-based APIs, including SaaS or private APIs. Workflows — Benefits Serverless scalability and managed infrastructure Focus on modeling your workflow logic and let Workflows completely manage the infrastructure with rapid scaling. Pay-per-use pricing model Pay only if your workflows run: scale your costs down to zero during times of inactivity.

Proprietary + Confidential Define workflows with a YAML or JSON syntax. Visual representation of your workflows. Expression formulas supporting decision points, conditional step executions, and operations on variables. Passing information between steps with built-in JSON parsing and expression-based variable manipulations. Workflow definition and visualisation Built-in decisions and conditional step executions Passing variable values between workflow steps Workflows — Features

Proprietary + Confidential Reliable workflow execution Low latency of execution Workflows — Features Execute workflows with reliability required for enterprise and line-of-business applications. Fast scheduling of workflow executions and transitions between tasks. Predictable performance with no cold starts.

Proprietary + Confidential Built-in authentication for Google Cloud products Support for external API calls Built-in error handling Workflows — Features Orchestrate work of any Google Cloud product without worrying about authentication. Use a proper service account and let Workflows do the rest. Out-of-the-box support for calls to API endpoints outside of Google Cloud. Out of the box error handling for your workflow steps with configurable retry policies.

Proprietary + Confidential Workflows — Use cases Reliable transactions Low-latency, conditional processes with 3rd party integration IT infrastructure automation

Proprietary + Confidential Workflows UI interface

No content

No content

No content

No content

No content

No content

No content

No content

Proprietary + Confidential Gcloud commands # Deploy a workflow gcloud workflows deploy my-workflow \ --source=workflow.yaml # Execute a workflow gcloud workflows execute my-workflow # See the result gcloud workflows executions \ describe \ --workflow my-workflow Deploy and execute a workflow. Inspect the result of the execution of a workflow.

Proprietary + Confidential Workflows syntax deep dive

Proprietary + Confidential Sequences of steps Payment Processor Cloud Run Authorize & charge CC Notifier Cloud Run Notify user Shipper Cloud Functions Prepare & ship items - processPayment: call: http.post args: url: https://payment-processor.run.app/... body: input: ${paymentDetails} result: processResult - shipItems: call: http.post args: url: https://.../cloudfunctions.net/ship body: address: ${processResult.body.address} result: shipResult - notifyUser: call: http.post ...

Proprietary + Confidential Variable passing & JSON parsing Payment Processor Cloud Run Authorize & charge CC Notifier Cloud Run Notify user Shipper Cloud Functions Prepare & ship items - processPayment: call: http.post args: url: https://payment-processor.run.app/... body: input: ${paymentDetails} result: processResult - shipItems: call: http.post args: url: https://.../cloudfunctions.net/ship body: address: ${processResult.body.address} result: shipResult - notifyUser: call: http.post ...

Proprietary + Confidential Calling HTTP APIs Payment Processor Cloud Run Authorize & charge CC Notifier Cloud Run Notify user Shipper Cloud Functions Prepare & ship items - processPayment: call: http.post args: url: https://payment-processor.run.app/... body: input: ${paymentDetails} result: processResult - shipItems: call: http.post args: url: https://.../cloudfunctions.net/ship body: address: ${processResult.body.address} result: shipResult - notifyUser: call: http.post ...

Proprietary + Confidential Authentication (OAuth2 | OIDC) Payment Processor Cloud Run Authorize & charge CC Notifier Cloud Run Notify user Shipper Cloud Functions Prepare & ship items - processPayment: call: http.post args: url: https://payment-processor.run.app/... body: input: ${paymentDetails} auth: type: OIDC result: processResult ... AUTHENTICATION

Proprietary + Confidential Pause Payment Processor Cloud Run Authorize & charge CC Notifier Cloud Run Notify user Shipper Cloud Functions Prepare & ship items WAIT - pause: call: sys.sleep args: seconds: 60

Proprietary + Confidential Logging Payment Processor Cloud Run Authorize & charge CC Notifier Cloud Run Notify user Shipper Cloud Functions Prepare & ship items LOG - log-processed: call: sys.log args: text: "Payment processed" severity: INFO

Proprietary + Confidential base64 ● encode ● decode text ● encode ● decode ● substring json ● encode ● encode_to_string ● decode retry ● always ● never Built-in functions sys ● get_env ● sleep ● now ● log errors ● type_error ● value_error ● index_error ● key_error ● not_implemented_error ● recursion_error ● zero_division_error ● system_error ● timeout_error ● resource_limit_error http ● http_error ● connection_error ● auth_error ● get ● post ● put ● patch ● delete ● request ● default_retry_predicate experimental.executions ● execution_error ● run ● map

Proprietary + Confidential Error handling, conditionals, jumps Payment Processor Cloud Run Authorize & charge CC Notifier Cloud Run Notify user Shipper Cloud Functions Prepare & ship items - processPayment: try: call: http.post args: url: https://payment-processor.run.app/... body: input: ${paymentDetails} result: processResult except: as: e steps: - known_errors: switch: - condition: ${not("HttpError" in e.tags)} next: connectionError - condition: ${e.code == 404} return: "Sorry, URL wasn't found." - unhandled_exception: raise: ${e} ERROR CHECKING

Proprietary + Confidential Retry & backoff Payment Processor Cloud Run Authorize & charge CC Notifier Cloud Run Notify user Shipper Cloud Functions Prepare & ship items - processPayment: try: call: http.post args: url: https://payment-processor.run.app/... body: input: ${paymentDetails} result: processResult retry: max_retries: 5 backoff: initial_delay: 1 max_delay: 60 multiplier: 2 MAX: 5 times BACKOFF

Proprietary + Confidential Like programming language subroutines or functions Subworkflows main: steps: - call_fullname: call: get_fullname args: first_name: "Sherlock" last_name: "Holmes" result: output - return_message: return: ${output} get_fullname: params: [first_name, last_name] steps: - prepMessage: return: ${first_name + " " + last_name}

Proprietary + Confidential Connectors

Proprietary + Confidential Simplifies access to Google Cloud products within a workflow: ● No need to tweak the URLs to call, or specify authentication ● Transparent handling of errors and retries (improves reliability and service SLA through retries) ● Handles long-running operations (transparent polling till result is ready, using a backoff) Connectors

Proprietary + Confidential BigQuery Cloud Build Cloud Functions Cloud Natural Language Cloud Scheduler Cloud SQL Cloud Storage Cloud Tasks Cloud Translation Existing connectors Compute Engine Dataflow Firestore Google Kubernetes Engine Pub/Sub Secret Manager Storage Transfer Service Workflows & Workflows execution

Proprietary + Confidential Stopping a Compute Engine VM without connectors Need to poll till the VM is really stopped Connector example - stop_machine: try: call: http.post args: url: ${"https://compute.googleapis.com/compute/v1/projects/"+project+"/zones/"+zone+"/instances/"+instanceName+"/stop"} auth: type: OAuth2 result: stop_resp retry: ${http.default_retry} - check_status: try: steps: - sleep: call: sys.sleep args: seconds: ${polling_delay} - adjust_delay: assign: - polling_delay: ${polling_delay * multiplier} - poll_status: call: http.get args: url: ${stop_resp.body.selfLink} auth: type: OAuth2 result: status_resp - compare: switch: - condition: ${status_resp.body.status == "DONE"} next: successfully_stopped - condition: ${status_resp.body.status == "RUNNING" or status_resp.body.status == "PENDING"} next: poll_status - condition: ${"error" in status_resp.body} next: failed retry: ${http.default_retry} - successfully_stopped: return: "VM instance successfully stopped!" - failed: return: ${status_resp.body.error}

Proprietary + Confidential Stopping a Compute Engine VM with the connector No need to poll: the connector waits for the end of a “long running operation” Connector example - stop_machine: call: googleapis.compute.v1.instances.stop args: instance: ${instanceName} project: ${project} zone: ${zone} # Optional connector parameters connector_params: timeout: 100 polling_policy: initial_delay: 1 multiplier: 1.25

Proprietary + Confidential Workflows in action!

Proprietary + Confidential ● How to invoke a workflow execution programmatically ○ Using the multi-language client libraries ● How to schedule a workflow execution ○ Thanks to Cloud Scheduler ● How to use functions to palliate the lack of expressiveness ○ Taking advantage of Cloud Functions for logic impossible with Workflows ● How to access secrets from Secret Manager ○ Avoid hard-coding secrets ● How to send a Pub/Sub message Workflows in action!

Proprietary + Confidential Márton Kodok (Google Developer Expert) shared concrete use cases with Workflows https://martonkodok.medium.com/ ● Automate the execution of BigQuery queries with Cloud Workflows ● Firestore backups the easy way with Cloud Workflows ● Run shell commands and orchestrate Compute Engine VMs with Cloud Workflows ● Using Cloud Workflows to load Cloud Storage files into BigQuery Concrete examples from the Community

Proprietary + Confidential Guillaume Laforge — @glaforge Developer Advocate, Google Cloud May 2021 Thanks for your attention! Q & A