Share
Embed
Pwrake: Distributed Workflow Engine based on Rake
Presentation for RubyKaigi2016@Kyoto
http://rubykaigi.org/2016/presentations/masa16tanaka.html
Masahiro Tanaka 田中昌宏
September 09, 2016
Transcript
52 slides
-
Pwrake: Distributed Workflow Engine based on Rake Masahiro TANAKA -
田中昌宏 Center for Computational Science, University of Tsukuba 筑波大学 計算科学研究センター Sep.9, 2016 RubyKaigi2016@Kyoto 1 Japan Science and Technology Agency -
Masahiro Tanaka ▶ Research Fellow at ◦ Center for Computational
Sciences, University of Tsukuba ▶ Majored in Astronomy ▶ The author of Ruby/NArray since 1999 ◦ Equivalent of Numpy ◦ Presentation at RubyKaigi 2010 at Tsukuba • http://rubykaigi.org/2010/ja/events/83 Sep.9, 2016 RubyKaigi2016@Kyoto 2 -
NArray Progress ▶ The name of new version: Numo::NArray. ◦
https://github.com/ruby-numo/narray ▶ Basic functionality work is almost complete. ▶ Future work: ◦ binding to numerical libraries. ◦ binding to plotting libraries. ◦ interface to I/O. ◦ speedup with SIMD etc. ◦ binding to GPU APIs. ◦ use case such as machine learning ◦ … ▶ Contributions are welcome. Sep.9, 2016 RubyKaigi2016@Kyoto 3 -
Today’s Topic: Pwrake ▶ Parallel Workflow extension for Rake ▶
Purpose: ◦ Scientific data processing on computer cluster. ▶ Run Rake tasks concurrently. ▶ Execute sh command line on remote computing nodes. Sep.9, 2016 RubyKaigi2016@Kyoto 4 Pwrake cc -o a.o -c a.c cc -o b.o -c b.c cc -o c.o -c c.c rule ".o" => ".c" do |x| sh "cc -o #{x} -c #{x.source}" end Rakefile https://github.com/masa16/pwrake -
Contents ▶ Background: Scientific Workflow ▶ Workflow Definition Language ▶
Pwrake Structure ▶ Gfarm Distributed File System ▶ Locality-Aware Task Scheduling ▶ Fault Tolerance ▶ Science Data Processing with Pwrake & Gfarm Sep.9, 2016 RubyKaigi2016@Kyoto 5 -
Contents ▶Background: Scientific Workflow ▶ Workflow Definition Language ▶ Pwrake
Structure ▶ Gfarm Distributed File System ▶ Locality-Aware Task Scheduling ▶ Fault Tolerance ▶ Science Data Processing with Pwrake & Gfarm Sep.9, 2016 RubyKaigi2016@Kyoto 6 -
Combine multiple-shots of Astronomical Images and produce a custom
Mosaic Image. http://montage.ipac.caltech.edu/ Sep.9, 2016 RubyKaigi2016@Kyoto 7 Workflow Example: Montage -
Combine multiple-shots of Astronomical Images and produce a custom
Mosaic Image. http://montage.ipac.caltech.edu/ Sep.9, 2016 RubyKaigi2016@Kyoto 8 Workflow Example: Montage -
Example of scientific workflow: Montage (Astronomy image processing) mProjectPP mDiff+mFitplane
mBGModel mBackground mShrink mAdd mAdd mJPEG Output image Workflow is expressed as DAG (Directed Acyclic Graph) Input images … Process file output input Task Sep.9, 2016 RubyKaigi2016@Kyoto 9 -
Contents ▶ Background: Scientific Workflow ▶Workflow Definition Language ▶ Pwrake
Structure ▶ Gfarm Distributed File System ▶ Locality-Aware Task Scheduling ▶ Fault Tolerance ▶ Science Data Processing with Pwrake & Gfarm Sep.9, 2016 RubyKaigi2016@Kyoto 10 -
▶ Use Markup Language (e.g. XML) ◦ e.g. DAX (for
Pegasus workflow system) ◦ Necessary to write a script to define many tasks. ▶ Design a new language ◦ e.g. Swift (Wilde et al. 2011) (Different language from Apple’s) ◦ Learning cost, small user community. ▶ Use an existing language ◦ e.g. GXP Make (Taura et al. 2013) Workflow Definition Language Sep.9, 2016 RubyKaigi2016@Kyoto 11 -
Workflow to Build Program Sep.9, 2016 12 RubyKaigi2016@Kyoto DAG a.o
a.c b.o b.c c.o c.c foo -
Workflow to Build Program Sep.9, 2016 13 RubyKaigi2016@Kyoto a.o a.c
b.o b.c c.o c.c foo DAG Makefile (GNU make) SRCS := $(wildcard *.c) OBJS := $(subst .c,.o,$(SRCS)) all: foo %.o : %.c cc -o $@ -c $< foo: $(OBJS) cc -o $@ $^ -
Workflow to Build Program Sep.9, 2016 14 Makefile (GNU make)
Rakefile RubyKaigi2016@Kyoto SRCS := $(wildcard *.c) OBJS := $(subst .c,.o,$(SRCS)) all: foo %.o : %.c cc -o $@ -c $< foo: $(OBJS) cc -o $@ $^ SRCS = FileList["*.c"] OBJS = SRCS.ext("o") task :default => "foo" rule ".o" => ".c" do |x| sh "cc -o #{x} -c #{x.source}" end file "foo" => OBJS do |x| sh "cc -o #{x} #{OBJS}" end Rakefile is a Ruby Script. -
Useful features of Rake ▶ Ruby Scripting ▶ Pathmap Sep.9,
2016 RubyKaigi2016@Kyoto 15 -
▶ For-Loop Ruby Scripting enabled by Internal DSL Sep.9, 2016
RubyKaigi2016@Kyoto 16 INPUT = FileList["r/*.fits"] OUTPUT = [] for src in INPUT OUTPUT << dst = "p/"+File.basename(src) file dst => src do |t| sh "mProjectPP #{t.prerequisites[0]} #{t.name} region.hdr" end end task :default => OUTPUT -
▶ Replaces %-format to the specified part of the path
name. ▶ Applicable to FileList, String, Prerequisites Pathmap Sep.9, 2016 RubyKaigi2016@Kyoto 17 INPUT = FileList["r/*.fits"] OUTPUT = INPUT.pathmap("p/%f") rule /^p¥/.*¥.fits$/ => "r/%n.fits" do |t| sh "mProjectPP #{t.prerequisites[0]} #{t.name} region.hdr" end task :default => OUTPUT -
Pathmap Examples ▶ See Rake manual for detail. p 'a/b/c/file.txt'.pathmap("%p")
#=> "a/b/c/file.txt" p 'a/b/c/file.txt'.pathmap("%f") #=> "file.txt" p 'a/b/c/file.txt'.pathmap("%n") #=> "file" p 'a/b/c/file.txt'.pathmap("%x") #=> ".txt" p 'a/b/c/file.txt'.pathmap("%X") #=> "a/b/c/file" p 'a/b/c/file.txt'.pathmap("%d") #=> "a/b/c" p 'a/b/c/file.txt'.pathmap("%2d") #=> "a/b" p 'a/b/c/file.txt'.pathmap("%-2d") #=> "b/c“ p 'a/b/c/file.txt'.pathmap("%d%s%{file,out}f") #=> "a/b/c/out.txt“ p 'a/b/c/file.txt'.pathmap("%X%{.*,*}x"){|ext| ext.upcase} #=> "a/b/c/file.TXT“ Sep.9, 2016 RubyKaigi2016@Kyoto 18 -
▶ Requires two files as prerequisites. ▶ Useful to define
complex workflows. Prerequisite Map by Block Sep.9, 2016 RubyKaigi2016@Kyoto 19 FILEMAP = {"d/d00.fits“=>["p/p00.fits","p/01.fits"], ...} rule /^d¥/.*¥.fits$/ => proc{|x| FILEMAP[x]} do |t| p1,p2 = t.prerequisites sh "mDiff #{p1} #{p2} #{t.name} region.hdr" end -
Rake as a WfDL ▶ Rake is a powerful WfDL
to define Complex and Many-Task Scientific Workflow. ◦ Rule ◦ Pathmap ◦ Internal DSL • For-Loop • Prerequisite map by block ▶ We use Rake as WfDL for Pwrake system. Sep.9, 2016 RubyKaigi2016@Kyoto 20 -
Contents ▶ Background: Scientific Workflow ▶ Workflow Definition Language ▶Pwrake
Structure ▶ Gfarm Distributed File System ▶ Locality-Aware Task Scheduling ▶ Fault Tolerance ▶ Science Data Processing with Pwrake & Gfarm Sep.9, 2016 RubyKaigi2016@Kyoto 21 -
Pwrake master fiber pool fiber sh fiber sh fiber sh
pwrake worker communicator Pwrake Structure Worker nodes Master node enq deq Task Graph Task Queue fiber sh pwrake worker communicator Scheduling Gfarm files files files files process process process process SSH SSH Sep.9, 2016 22 RubyKaigi2016@Kyoto -
Task Queueing ▶ Rake ◦ Depth-First Search • Same as
Topological Sort • No parallelization ▶ Pwrake ◦ Task Queue • Search Ready-to-Execute Tasks • Enqueue it. • Scheduling = Select Task on deq Sep.9, 2016 RubyKaigi2016@Kyoto 23 enq Topological Sort A B D C E F Task Queue A B C Workflow DAG A B C D E F deq -
Thread vs. Fiber ▶ Pwrake is initially implemented using Thread.
▶ Thread is annoying… ◦ Limited by max user processes (ulimit –u) ◦ Hard to find the reason of deadlock. ◦ Which part of code should be synchronized??? ◦ Need to synchronize puts. ▶ Fiber is currently used. ◦ Most of time, waiting I/O from worker nodes. ◦ Easier coding due to explicit context switch. ▶ But requires Asynchronous I/O. Sep.9, 2016 RubyKaigi2016@Kyoto 24 -
Asynchronous I/O ▶ Bartender (Asynchronous I/O) by Seki-san ◦ https://github.com/seki/bartender
◦ Single Fiber for one I/O ▶ Pwrake Asynchronous I/O ◦ Multiple Fibers for one I/O ◦ Timeout handling Sep.9, 2016 RubyKaigi2016@Kyoto 25 -
Other Features ▶ Task options defined with desc ◦ ncore,
allow, deny, … ▶ Logging ▶ Report statics as a HTML page. ▶ Output DAG in Graphviz form. Sep.9, 2016 RubyKaigi2016@Kyoto 26 -
File Sharing ▶ File sharing is necessary for multi-node workflows
▶ File Staging by Workflow Systems ◦ Transfer files to/from worker nodes. ◦ Managed by Workflow Systems. ▶ File Sharing with Distributed File System (DFS) ◦ NFS, Lustre, GPFS, Gluster, … ◦ We choose Gfarm file system for Pwrake. Sep.9, 2016 RubyKaigi2016@Kyoto 27 -
Comparison of Network File Systems Sep.9, 2016 RubyKaigi2016@Kyoto 28 Storage
CPU CPU CPU file1 file2 file3 Storage CPU CPU CPU Storage Storage file1 file2 file3 NFS Distributed file systems (Lutre, GPFS, etc) Storage CPU CPU CPU Storage Storage file1 file2 file3 Gfarm file system Concentration of storage Network limitation Scalable Performance with local access -
Contents ▶ Background: Scientific Workflow ▶ Workflow Definition Language ▶
Pwrake Structure ▶Gfarm Distributed File System ▶ Locality-Aware Task Scheduling ▶ Fault Tolerance ▶ Science Data Processing with Pwrake & Gfarm Sep.9, 2016 RubyKaigi2016@Kyoto 29 -
Gfarm File System ▶ http://oss-tsukuba.org/software/gfarm ▶ Distributed File System constructed
by local storage of compute nodes. ▶ Designed for wide-area file sharing ◦ across institutes connected through the Internet. ▶ Open-Source project by Prof. Tatebe ◦ Since 2000. ◦ Gfarm ver. 2 since 2007. ◦ Current version: version 2.6.12 ▶ Reference: ◦ Osamu Tatebe, Kohei Hiraga, Noriyuki Soda, "Gfarm Grid File System", New Generation Computing, 2010, Vol.28, issue 3, p.257 Sep.9, 2016 RubyKaigi2016@Kyoto 30 -
Global Directory Tree / /dir1 file1 file2 /dir2 file3 file4
file2 file3 file1 Metadata Server (MDS) Store content of file to local storage. Manages inode and file location. File System Nodes (FSN) Local Storage Client Directory lookup File access … compute process local access FSN is also compute node. Gfarm File System Components Sep.9, 2016 RubyKaigi2016@Kyoto 31 -
Use Cases of Gfarm ▶ HPCI (High Performance Computing Infrastructure)
◦ http://www.hpci-office.jp/ ◦ Computational environment connecting the K computer and other supercomputers of research institutions in Japan by SINET5. ▶ NICT Science Cloud ◦ http://sc-web.nict.go.jp/ ▶ Commercial Uses ◦ Active! mail by QUALITIA • http://www.qualitia.co.jp/product/am/ Sep.9, 2016 RubyKaigi2016@Kyoto 32 -
Gfarm Features ▶ Scalable Capacity ◦ By adding FSN ◦
Commodity hardware ▶ Fault Tolerance ◦ Standby slave MDS ◦ Automatic file replication (mirroring) ▶ High Performance ◦ Parallel access scales ◦ Local access Sep.9, 2016 RubyKaigi2016@Kyoto 33 -
Gfarm Issues ▶ MDS is stand alone, not scalable. ◦
File creation speed is limited by DB performance. ◦ Use SSD for MDS DB storage. ▶ Sequential access performance does not increase. ◦ Gfarm does not support network RAID except 1. ◦ Use RAID0/5 for FSN spool. ▶ Maybe improved in the future. Sep.9, 2016 RubyKaigi2016@Kyoto 34 -
Gfarm Information Source ▶ NPO - OSS Tsukuba ◦ http://oss-tsukuba.org/
▶ Gfarm Symposium/Workshop ◦ http://oss-tsukuba.org/event ◦ Next Workshop: Oct 21, 2016 @Kobe • http://oss-tsukuba.org/event/gw2016 ▶ Mailing List ◦ https://sourceforge.net/p/gfarm/mailman/ ▶ Paid Support ◦ http://oss-tsukuba.org/support Sep.9, 2016 RubyKaigi2016@Kyoto 35 -
Pwrake master Supporting Gfarm by Pwrake Sep.9, 2016 RubyKaigi2016@Kyoto 36
Pwrake worker process process / tmp/ pwrake_john_000/ Rakefile file01.dat file02.dat pwrake_john_001/ Rakefile file01.dat file02.dat / tmp/ john/ Rakefile file01.dat file02.dat Gfarm MDS Gfarm / Rakefile file01.dat file02.dat Find FSN where a file is stored. (gfwhere-pipe) Mount Gfarm FS for each core. (gfarm2fs) Pwrake Master Node Worker Node mount mount Check Gfarm FS? communicator -
Contents ▶ Background: Scientific Workflow ▶ Workflow Definition Language ▶
Pwrake Structure ▶ Gfarm Distributed File System ▶Locality-Aware Task Scheduling ▶ Fault Tolerance ▶ Science Data Processing with Pwrake & Gfarm Sep.9, 2016 RubyKaigi2016@Kyoto 37 -
Locality in Gfarm File System ▶ Large Scientific Data ◦
File I/O is a bottleneck ▶ Data Locality is a key ◦ Write a File • Select local storage • to write output file ◦ Read a File • Assign a task to the node where input file exits • Workflow System Sep.9, 2016 38 RubyKaigi2016@Kyoto Local Storage Local Storage File Task Task Local Storage Local Storage Task File File Write Read -
▶ NodeQueue in TaskQueue: Assigned to worker node. ▶ enq:
put a task into NodeQueue assigned to candidate nodes. ▶ deq: get a task from NodeQueue assigned to worker thread node. ▶ Load-balancing by deq-ing from another NodeQueue (Task-stealing) Sep.9, 2016 RubyKaigi2016@Kyoto Locality-Aware Task Queue TaskQueue Node 1 Node 2 Node 3 deq enq NodeQueue RemoteQueue Task 39 worker thread -
1. Naïve Locality Scheduling ◦ Define “candidate nodes” where input
files are stored. ◦ Default of Pwrake 2. Scheduling based on Graph Partitioning ◦ Method using MCGP (Multi-Constraint Graph Partitioning) ◦ Publication: CCGrid 2012 • M. Tanaka and O. Tatebe, "Workflow Scheduling to Minimize Data Movement Using Multi-constraint Graph Partitioning," in CCGrid 2012, p.65. Locality-aware Scheduling Methods Sep.9, 2016 RubyKaigi2016@Kyoto 40 -
▶ Find Candidate Nodes for Task based on Input File
Location ◦ Note: Input files for a task can be stored in multiple nodes. ▶ Method to define candidate node: ◦ Calculate the total size of input files for each node. ◦ Find candidate nodes having more file size than half of maximum size. Sep.9, 2016 RubyKaigi2016@Kyoto Naïve Locality Scheduling Task t × input files Node 3 Node 2 Node 1 file A file B file C ½ max filesize file A file B file C ◦ ◦ file C file C 41 -
Graph Partitioning on DAG Standard Graph Partitioning Proposed method using
Multi-Constraint Graph Partitioning Node-A Node-B Node-C Node-D Former Tasks Latter Tasks Not aware of task parallelization Sep.9, 2016 RubyKaigi2016@Kyoto 42 Parallelize in every stage Node-A Node-B Node-C Node-D -
Platform for Evaluation ▶ Cluster used for Evaluation ▶ Input
File: 2MASS Image CPU Xeon E5410 (2.3GHz) Main Memory 16 GB Network GbE # of Nodes 8 Total # of Cores 32 Data size of each File 2.1 MB or 1.7 MB # of Input Files 607 Total Data size of Input Files 1270 MB Data I/O size during Workflow ~24 GB Total # of Tasks = # of Vertices 3090 At first, All the Input files are stored at a single node. Sep.9, 2016 43 RubyKaigi2016@Kyoto -
Data Transfer between nodes 87.9 47.4 14.0 0 10 20
30 40 50 60 70 80 90 100 A(Unconcern) B(Naïve locality) C(MCGP) Data Size Ratio (%) Sep.9, 2016 RubyKaigi2016@Kyoto 44 -
Workflow Execution Time 0 20 40 60 80 100 120
140 160 180 200 A(Unconcern) B(Naïve locality) C(MCGP) Elapsed Time (sec) 31% down 22% down Includes time to solve MCGP (30 ms) Sep.9, 2016 RubyKaigi2016@Kyoto 45 -
Contents ▶ Background: Scientific Workflow ▶ Workflow Definition Language ▶
Pwrake Structure ▶ Gfarm Distributed File System ▶ Locality-Aware Task Scheduling ▶Fault Tolerance ▶ Science Data Processing with Pwrake & Gfarm Sep.9, 2016 RubyKaigi2016@Kyoto 46 -
Fault Tolerance in Pwrake ▶ Master Failure: ◦ Rerun Pwrake
and resume interrupted workflow. • based on time stamp of Input/Output files. ▶ Worker Failure: ◦ Policy: • Workflow does not stop even after one of worker nodes fails. ◦ Approaches: • Automatic file replication by Gfarm FS • Task retry, Worker dropout by Pwrake (ver 2.1) Sep.9, 2016 RubyKaigi2016@Kyoto 47 -
Experiment of Worker Failure ▶ Kill worker process and gfsd
at 20 sec. ▶ Reduce # of cores 64 → 56, Storage is unavailable after kill. ▶ Final result was correct. Workflow continued successfully. Sep.9, 2016 RubyKaigi2016@Kyoto 48 0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70 80 # of running processes time (sec) Kill processes and gfsd in a worker node -
Contents ▶ Background: Scientific Workflow ▶ Workflow Definition Language ▶
Pwrake Structure ▶ Gfarm Distributed File System ▶ Locality-Aware Task Scheduling ▶ Fault Tolerance ▶Science Data Processing with Pwrake & Gfarm ◦ NICT Science Cloud ◦ HSC in Subaru Telescope Sep.9, 2016 RubyKaigi2016@Kyoto 49 -
NICT Science Cloud Sep.9, 2016 RubyKaigi2016@Kyoto 50 Himawari-8 realtime web
http://himawari8.nict.go.jp/ http://sc-web.nict.go.jp/ Presentation at Gfarm Symposium 2015 - http://oss-tsukuba.org/event/gs2015 -
Hyper Suprime-Cam(HSC) in Subaru Telescope Sep.9, 2016 RubyKaigi2016@Kyoto 51 (Image
credit: NAOJ・HSC project) HSC Outlook HSC focal plane CCD Subaru Telescope Field of View: 1.5 degree (x 3 than Suprime-Cam) # of CCDs: 116 CCD pixels: 4272×2272 Generates ~300 GB data per night One of HSC targets: Discovery of Super Nova -
Conclusion ▶ Scientific Workflow System is required for processing of
science data on multi-node cluster. ▶ Rake is powerful as a Workflow Definition Language. ▶ Pwrake workflow system is developed based on Rake an Gfarm file system. ▶ Study on Locality-Aware Task Scheduling. ▶ Fault Tolerance features. ▶ Pwrake & Gfarm use cases: ◦ NICT Science Cloud ◦ Subaru HSC Sep.9, 2016 RubyKaigi2016@Kyoto 52