Dynamic Workflow Execution in the Cloud using Steep

Slide 1

Slide 1 text

Hendrik M. Würz - 12.05.2022 — Dynamic Workflow Execution in the Cloud using Steep

Slide 2

Slide 2 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 About me 12.05.2022 Page 2 Hi, I'm Hendrik ● Studied Computer Science at TU Darmstadt ● Researcher at Fraunhofer Institute for Computer Graphics ● Interested in Cloud Computing, Function as a Service, Microservices, Workflow Management, ... © Fraunhofer IGD

Slide 3

Slide 3 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Agenda 12.05.2022 Page 3 © Fraunhofer IGD Dynamic Workflows Execution in the Cloud Live Demo Requirements

Slide 4

Slide 4 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Requirements What do we want? 12.05.2022 Page 4 Cloud-based © Fraunhofer IGD Start VMs on demand Allocate only “fitting” resources Robust execution Works for large data sets Efficient scheduling Heterogeneous services Enable loops Easy customization of the execution Maximize parallelization

Slide 5

Slide 5 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Requirements What do we want? 12.05.2022 Page 5 © Fraunhofer IGD Efficient scheduling Heterogeneous services ● I can execute custom software ● I can call external APIs ● I can pass arguments in any form Enable loops ● Data driven loops ● Adjust iteration during run time ● Collect outputs of the iterations Easy customization ● Resource allocation can be manipulated ● Output of Services can be changed ● Scheduled tasks can be adjusted before execution Maximize parallelization ● Find independent parts ● Limit parallelism ● Prioritize execution

Slide 6

Slide 6 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Requirements What do we want? 12.05.2022 Page 6 Cloud-based © Fraunhofer IGD Start VMs on demand ● If no workflow is running, no VM is running ● Reuse running VMs if there if more work to do ● Automatic VM provisioning Allocate only “fitting” resources ● A CPU task does not need a GPU ● Select good values for RAM, disk size, CPU cores, network connection, … Robust execution ● A VM can be unreachable → Retry its task on another VM ● Cleanup broken VMs ● Rolling Updates Works for large data sets ● Strong separation between data processing and execution management ● Use databases for tasks

Slide 7

Slide 7 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Executing a workflow 12.05.2022 Page 7 1. (Prepare base images for VMs) 2. Submit Workflow 3. Schedule Tasks 4. Determine required capabilities 5. Start VMs via OpenStack Driver 6. Run provisioning scripts 7. Assign tasks to VMs 8. Monitor execution 9. Destroy idle VMs © Fraunhofer IGD 2 3 4 5 6 7 8 9

Slide 8

Slide 8 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Executing a workflow 12.05.2022 Page 8 1. (Prepare base images for VMs) 2. Submit Workflow 3. Schedule Tasks 4. Determine required capabilities 5. Start VMs via OpenStack Driver 6. Run provisioning scripts 7. Assign tasks to VMs 8. Monitor execution 9. Destroy idle VMs © Fraunhofer IGD 2 3 4 5 6 7 8 9

Slide 9

Slide 9 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Generating Process Chains for Execution 12.05.2022 Page 9 1. Read Workflow from Database 2. Generate Process Chains 3. Save Process Chains in Database 4. Wait for results 5. GoTo 2. © Fraunhofer IGD

Slide 10

Slide 10 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Loops 12.05.2022 Page 10 © Fraunhofer IGD Multiple instances with a priori design-time knowledge Multiple instances with a priori run-time knowledge Multiple instances without a priori run-time knowledge

Slide 11

Slide 11 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Loops 12.05.2022 Page 11 © Fraunhofer IGD Multiple instances with a priori design-time knowledge Multiple instances with a priori run-time knowledge Multiple instances without a priori run-time knowledge

Slide 12

Slide 12 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Loops 12.05.2022 Page 12 © Fraunhofer IGD

Slide 13

Slide 13 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Loops 12.05.2022 Page 13 © Fraunhofer IGD

Slide 14

Slide 14 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Executing a workflow 12.05.2022 Page 14 1. (Prepare base images for VMs) 2. Submit Workflow 3. Schedule Tasks 4. Determine required capabilities 5. Start VMs via OpenStack Driver 6. Run provisioning scripts 7. Assign tasks to VMs 8. Monitor execution 9. Destroy idle VMs © Fraunhofer IGD 2 3 4 5 6 7 8 9

Slide 15

Slide 15 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 15 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU Process Chain 1 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler

Slide 16

Slide 16 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 16 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU Process Chain 1 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler get distinct required capability sets

Slide 17

Slide 17 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 17 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU Process Chain 1 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler GPU GPU 2TB 2x 1x

Slide 18

Slide 18 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 18 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU Process Chain 1 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler GPU GPU 2TB 2x 1x

Slide 19

Slide 19 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 19 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU Process Chain 1 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler GPU GPU 2TB busy available 2x 1x

Slide 20

Slide 20 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 20 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU Process Chain 1 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler GPU GPU 2TB Cloud Manager VM2 Get Process Chain with [GPU] 2x 1x

Slide 21

Slide 21 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 21 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU Process Chain 1 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager assign

Slide 22

Slide 22 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 22 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager

Slide 23

Slide 23 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 23 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager GPU GPU 2TB 1x 1x get distinct required capability sets

Slide 24

Slide 24 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 24 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager GPU GPU 2TB 1x 1x Start VMs Find Setups

Slide 25

Slide 25 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 25 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager GPU GPU 2TB 1x 1x Start VMs VM 3 GPU VM 4 GPU 2TB Start and provision VMs

Slide 26

Slide 26 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 26 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager GPU GPU 2TB 1x 1x VM 3 GPU VM 4 GPU 2TB

Slide 27

Slide 27 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 27 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager GPU GPU 2TB 1x 1x VM 3 GPU VM 4 GPU 2TB busy busy available available

Slide 28

Slide 28 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 28 © Fraunhofer IGD Process Chain 2 GPU 2TB Process Chain 3 GPU VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager VM 3 GPU VM 4 GPU 2TB assign

Slide 29

Slide 29 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 29 © Fraunhofer IGD VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager VM 3 GPU VM 4 GPU 2TB

Slide 30

Slide 30 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 30 © Fraunhofer IGD VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager VM 3 GPU VM 4 GPU 2TB

Slide 31

Slide 31 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 31 © Fraunhofer IGD VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager VM 3 GPU VM 4 GPU 2TB

Slide 32

Slide 32 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 32 © Fraunhofer IGD VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager VM 3 GPU VM 4 GPU 2TB Destroy

Slide 33

Slide 33 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Steep Start VMs for Process Chains 12.05.2022 Page 33 © Fraunhofer IGD VM 1 GPU VM 2 GPU Registered Process Chains Running VMs Setups Setup 1 GPU Setup 2 GPU 2TB Setup 3 CPU Scheduler Cloud Manager VM 3 GPU

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Thanks for your attention —

Slide 36

Slide 36 text

Contact — Hendrik M. Würz GEO Department [email protected] Fraunhofer IGD Fraunhoferstraße 5 64283 Darmstadt www.igd.fraunhofer.de

Slide 37

Slide 37 text

Public IGD-Folienvorlage-de.potx, Version 3.2, 01.04.2022 Image Sources 12.05.2022 Seite 37 ● Cloud by Iconixar, Flaticon License, https://www.flaticon.com/free-icon/cloud_1163726 (accessed 2022-04-26) ● Dynamic by bearicons, Flaticon License, https://www.flaticon.com/free-icon/workflow_6687163 (accessed 2022-04-26; modified) ● Open Stack Logo by OpenStack community https://www.openstack.org/brand/openstack-logo/logo-download/ (accessed 2021-03-31) ● Terminal by Royyan Wijaya, Flaticon License, https://www.flaticon.com/free-icon/terminal_6617073 (accessed 2022-04-28) ● Workflow by Freepik, Flaticon License, https://www.flaticon.com/free-icon/workflow_1415396 (accessed 2022-04-28; modified) ● Loading by Eucalyp, Flaticon License, https://www.flaticon.com/premium-icon/loading_3598265 (accessed 2022-04-28) ● Delete by Freepik, Flaticon License, https://www.flaticon.com/free-icon/delete_3221845 (accessed 2022-04-28) ● File by Pixel perfect, Flaticon License, https://www.flaticon.com/free-icon/file_633585 (accessed 2022-04-28) ● Assignment by Darius Dan, Flaticon License https://www.flaticon.com/premium-icon/assignment_3995766 (accessed 2022-04-28) ● Work in Progress by Freepik https://www.flaticon.com/free-icon/arrows_1716838 (accessed 2022-05-09) ● Checkmark by Stockio https://www.flaticon.com/premium-icon/checkmark_656971 (accessed 2022-05-10) © Fraunhofer IGD