Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Capability-based scheduling of scientific workflows in the cloud

Michel Krämer
July 07, 2020
Research
0
410

Capability-based scheduling of scientific workflows in the cloud

This is a presentation I held at the DATA conference 2020. The talk is about my research paper entitled "Capability-based scheduling of scientific workflows in the cloud".

I presented a distributed task scheduling algorithm and a software architecture for a system executing scientific workflows in the Cloud. The main challenges I addressed were (i) capability-based scheduling, which means that individual workflow tasks may require specific capabilities from highly heterogeneous compute machines in the Cloud, (ii) a dynamic environment where resources can be added and removed on demand, (iii) scalability in terms of scientific workflows consisting of hundreds of thousands of tasks, and (iv) fault tolerance because in the Cloud, faults can happen at any time. My software architecture consists of loosely coupled components communicating with each other through an event bus and a shared database. Workflow graphs are converted to process chains that can be scheduled independently. My scheduling algorithm collects distinct required capability sets for the process chains, asks the agents which of these sets they can manage, and then assigns process chains accordingly. I presented the results of four experiments I conducted to evaluate if my approach meets the aforementioned challenges. An implementation of my algorithm and software architecture is publicly available with the open-source workflow management system “Steep”.

Michel Krämer

July 07, 2020
Tweet

More Decks by Michel Krämer

See All by Michel Krämer
Smart City Clouds
michelkraemer
0
1.4k
Processing Smart City Data in the Cloud with Domain-Specific Languages
michelkraemer
0
1.6k
Visualizing Large 3D city models in the Web
michelkraemer
1
1.8k
Presentation 101
michelkraemer
2
1.9k
It's getting faster: from relational databases to MongoDB
michelkraemer
1
3k

Other Decks in Research

See All in Research
音楽土木工学を設計する——音楽プログラミング言語mimiumの開発を通じて
tomoyanonymous
0
190
Win ratioとは何か？
shuntaros
0
790
ランダムフォレストによる因果推論と最近の展開
tomoshige_n
11
6.8k
ClimaX: A foundation model for weather and climate
cocomoff
0
120
【論文紹介】Google. 2023. PaLM 2 Technical Report
chokkan
PRO
9
4k
LaCAM: Search-Based Algorithm for Quick Multi-Agent Pathfinding
kei18
0
450
大規模言語モデルに基づいた対話型AIによる研究支援に関する初歩的分析
niryuu
0
110
単語の通時的な意味変化のモデル化
seiichiinoue
3
680
AWS CDKでもMacを 管理したい
moba
0
280
論文 Capturability-based Analysis and Control of Legged Locomotion を読む (Part 1) / Review on "Capturability-based Analysis and Control of Legged Locomotion (Part 1)"
ssryuki
1
150
Generative A.I. + Law - Background, Applications and Use Cases Including GPT-4 Passes the Bar Exam
danielkatz
PRO
3
3k
Part 5: Efforts for Responsible LLMs, PAKDD 2023 Tutorial 2: A Gentle Introduction to Technologies Behind Language Models and Recent Achievement in ChatGPT
chokkan
PRO
1
310

Featured

See All Featured
The Illustrated Children's Guide to Kubernetes
chrisshort
25
44k
jQuery: Nuts, Bolts and Bling
dougneiner
57
6.7k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
658
120k
How to Ace a Technical Interview
jacobian
271
22k
Large-scale JavaScript Application Architecture
addyosmani
500
110k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
35
9.5k
Statistics for Hackers
jakevdp
787
210k
Fontdeck: Realign not Redesign
paulrobertlloyd
74
4.5k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
9
770
Design by the Numbers
sachag
272
18k
Fashionably flexible responsive web design (full day workshop)
malarkey
395
64k
The Power of CSS Pseudo Elements
geoffreycrofte
54
4.5k

Transcript

  1. CAPABILITY-BASED SCHEDULING
    OF SCIENTIFIC WORKFLOWS
    IN THE CLOUD
    MICHEL KRÄMER

    View Slide

  2. THAT’S ME

    View Slide

  3. Data processing
    requirements
    Very large
    data sets
    Heterogeneous
    services
    Automated data
    processing

    View Slide

  4. Data processing
    requirements
    Very large
    data sets
    Heterogeneous
    services
    Automated data
    processing

    View Slide

  5. Data processing
    requirements
    Very large
    data sets
    Heterogeneous
    services
    Automated data
    processing

    View Slide

  6. Data processing
    requirements
    Very large
    data sets
    Heterogeneous
    services
    Automated data
    processing

    View Slide

  7. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    A
    B D
    E
    C
    Scienti ic Work lows
    Automated data processing
    Independent services
    Distributed environments

    View Slide

  8. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    A
    B D
    E
    C
    Scienti ic Work lows
    Automated data processing
    Independent services
    Distributed environments

    View Slide

  9. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    A
    B D
    E
    C
    Scienti ic Work lows
    Automated data processing
    Independent services
    Distributed environments

    View Slide

  10. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    A
    B D
    E
    C
    Scienti ic Work lows
    Automated data processing
    Independent services
    Distributed environments

    View Slide

  11. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    A
    B D
    E
    C
    Scienti ic Work lows
    Automated data processing
    Independent services
    Distributed environments

    View Slide

  12. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    A
    B D
    E
    C
    Scienti ic Work lows
    Automated data processing
    Independent services
    Distributed environments

    View Slide

  13. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    A
    B D
    E
    C
    Scienti ic Work lows
    Automated data processing
    Independent services
    Distributed environments

    View Slide

  14. Challenges
    of cloud-based work low management
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows

    View Slide

  15. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    Challenges
    of cloud-based work low management
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows

    View Slide

  16. A
    B D
    E
    C
    Cloud-based
    Scienti ic Work lows
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows
    Challenges
    of cloud-based work low management
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows

    View Slide

  17. Challenges
    of cloud-based work low management
    Dynamic environment
    Scalability
    Fault tolerance
    Deelman et al. (2018). The future of scienti ic work lows

    View Slide

  18. How to schedule
    heterogeneous
    processing services?

    View Slide

  19. Naïve scheduling
    Service
    instances
    Virtual
    Machines
    ...
    ...
    A
    VM1
    VM2
    VM3
    VM4
    VMn
    Docker
    A
    Docker
    B
    GPU
    C
    In-memory
    C
    In-memory
    D
    Docker GPU

    View Slide

  20. Naïve scheduling
    Service
    instances
    Virtual
    Machines
    ...
    ...
    A
    VM1
    VM2
    VM3
    VM4
    VMn
    Docker
    A
    Docker
    B
    GPU
    C
    In-memory
    C
    In-memory
    D
    Docker GPU

    View Slide

  21. Naïve scheduling
    Service
    instances
    Virtual
    Machines
    ...
    ...
    A
    VM1
    VM2
    VM3
    VM4
    VMn
    Docker
    Docker
    GPU
    In-memory
    ...
    Docker
    GPU
    In-memory
    ...
    Docker
    GPU
    In-memory
    ...
    Docker
    GPU
    In-memory
    ...
    Docker
    GPU
    In-memory
    ...
    A
    Docker
    B
    GPU
    C
    In-memory
    C
    In-memory
    D
    Docker GPU

    View Slide

  22. Contribution
    Dynamic environment
    Scalability
    Fault
    tolerance

    View Slide

  23. Contribution
    Dynamic environment
    Capability-based scheduling
    Scalability
    Fault
    tolerance

    View Slide

  24. Contribution
    Dynamic environment
    Capability-based scheduling
    Scalability
    Fault
    tolerance
    Software architecture
    + Algorithm

    View Slide

  25. Software
    architecture

    View Slide

  26. Overview
    HTTP
    server
    Controller Scheduler
    Instance 1 Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M

    View Slide

  27. Overview
    HTTP
    server
    Controller Scheduler
    Instance 1 Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M

    View Slide

  28. Overview
    HTTP
    server
    Controller Scheduler
    Instance 1 Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M

    View Slide

  29. Overview
    HTTP
    server
    Controller
    Instance 1 Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M
    Scheduler

    View Slide

  30. Overview
    HTTP
    server
    Scheduler
    Instance 1 Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M
    Controller

    View Slide

  31. Overview
    HTTP
    server
    Controller Scheduler
    Instance 1 Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M

    View Slide

  32. Overview
    HTTP
    server
    Controller Scheduler
    Instance 1 Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M

    View Slide

  33. Overview
    HTTP
    server
    Controller Scheduler
    Instance 1 Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M

    View Slide

  34. Overview
    HTTP
    server
    Controller Scheduler
    Instance n
    Agent
    ...
    Database
    Event bus
    Cloud
    manager
    H C S A M
    Instance 1

    View Slide

  35. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  36. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  37. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  38. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  39. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  40. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  41. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  42. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  43. A
    B D
    E
    C
    Controller
    Read work low from database
    Generate new process chains
    Save process chains into database
    Wait for results

    View Slide

  44. Scheduling
    algorithm

    View Slide

  45. 500 process chains
    Scheduler
    PC1
    Docker
    A1
    C++
    PC2
    Docker
    PCi
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    A2
    (busy)
    Docker
    2 agents

    View Slide

  46. 500 process chains
    Scheduler
    distinct
    required
    capability sets
    PC1
    Docker
    A1
    C++
    PC2
    Docker
    PCi
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    2 agents
    A2
    (busy)
    Docker

    View Slide

  47. 500 process chains 2 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU

    View Slide

  48. 500 process chains 2 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU

    View Slide

  49. 500 process chains 2 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    busy
    ???
    Docker
    Python
    TensorFlow
    GPU

    View Slide

  50. 500 process chains 2 agents
    Scheduler
    Cloud Manager
    PC1
    Docker
    A1
    C++
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU
    request agents

    View Slide

  51. 500 process chains 2 agents
    Scheduler
    Cloud Manager
    PC1
    Docker
    A1
    C++
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    create
    Docker
    Python
    TensorFlow
    GPU

    View Slide

  52. 500 process chains 5 agents
    Scheduler
    Cloud Manager
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A5
    GPU
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    create
    Docker
    Python
    TensorFlow
    GPU

    View Slide

  53. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    distinct
    required
    capability sets
    A5
    GPU
    Docker

    View Slide

  54. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU
    A5
    GPU
    Docker

    View Slide

  55. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU
    A5
    GPU
    Docker

    View Slide

  56. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU
    busy
    OK
    OK
    OK
    ???
    A5
    GPU
    Docker

    View Slide

  57. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU
    A5
    GPU
    Docker
    A5
    GPU
    Docker
    A5
    GPU
    Docker

    View Slide

  58. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU
    A5
    GPU
    Docker
    A5
    GPU
    Docker

    View Slide

  59. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Docker
    Python
    TensorFlow
    GPU
    A5
    GPU
    Docker
    A5
    GPU
    Docker
    fetch
    process chain

    View Slide

  60. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    Docker
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU
    A5
    GPU
    Docker
    A5
    GPU
    Docker

    View Slide

  61. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    (busy)
    Docker
    A3
    Docker
    A2
    (busy)
    Docker
    PC2
    Docker
    PCi
    Python
    TensorFlow A4
    Python
    TensorFlow
    A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Docker
    Python
    TensorFlow
    GPU
    A5
    GPU
    Docker
    A5
    GPU
    Docker

    View Slide

  62. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    (busy)
    Docker
    A2
    (busy)
    Docker
    PCi
    Python
    TensorFlow A4
    (busy)
    Python
    TensorFlow
    A4
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    Python
    TensorFlow
    GPU
    A5
    GPU
    Docker
    A5
    GPU
    Docker

    View Slide

  63. 500 process chains 5 agents
    Scheduler
    PC1
    Docker
    A1
    C++
    A3
    (busy)
    Docker
    A2
    (busy)
    Docker
    A4
    (busy)
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    PCi+1
    GPU
    ...
    GPU
    A5
    (busy)
    GPU
    Docker
    A5
    GPU
    Docker

    View Slide

  64. 500 process chains 5 agents
    Scheduler
    Repeat
    PC1
    Docker
    A1
    C++
    A3
    (busy)
    Docker
    A2
    (busy)
    Docker
    A4
    (busy)
    Python
    TensorFlow
    PCj+1
    Docker
    ... ...
    PCj
    GPU
    ...
    A5
    (busy)
    GPU
    Docker

    View Slide

  65. Evaluation

    View Slide

  66. Contribution
    Dynamic environment
    Capability-based scheduling
    Scalability
    Fault
    tolerance
    Software architecture
    + Algorithm

    View Slide

  67. Experiment 1
    Capability-based scheduling
    100 process chains
    4 distinct capability sets
    Correct allocation
    R1
    R2
    R3
    R4
    R3+R4
    Process chain
    Start End
    Agent killed
    Fault

    View Slide

  68. Experiment 2
    Dynamic environment
    1000 process chains
    1 agent at the beginning
    8 agents at the end
    R1
    R2
    R3
    R4
    R3+R4
    Process chain
    Start End
    Agent killed
    Fault

    View Slide

  69. Experiment 3
    Scalability (process chains)
    150.000 process chains
    up to 8 agents
    Load managed well
    R1
    R2
    R3
    R4
    R3+R4
    Process chain
    Start End
    Agent killed
    Fault

    View Slide

  70. Experiment 4
    Fault tolerance
    1000 process chains
    Agents randomly killed
    Successful recovery

    View Slide

  71. Implementation of the
    software architecture
    and algorithm
    Open Source
    https://steep-wms.github.io/
    Steep

    View Slide

  72. Thanks for listening!
    MICHEL KRÄMER
    Fraunhofer IGD, Germany
    [email protected]
    github.com/michel-kraemer
    steep-wms.github.io
    Icons by Freepik from www. laticon.com

    View Slide