NEW CONTRIBUTORS Sven Fillinger Quantitative Biology Center, Germany Lorenz Gerber Genome Institute Singapore Ólafur Haukur Flygenring WuXi NextCode, Island Jonathan Sheffi Google Genomics, USA Rad Suchecki CSIRO, Australia Luke Goodsell Achilles Therapeutics, UK
CHANNELS DEDUP Channel.fromPath('/something/*') .into{ this_ch; that_ch } proces foo { input: file x from this_ch """ your_command $x """ } proces bar { input: file x from that_ch """ your_command $x """ } Channel.fromPath('/something/*') .fork(2) .set{ this_ch } proces foo { input: file x from this_ch """ your_command $x """ } proces bar { input: file x from this_ch """ your_command $x """ } →
EVENTS & METADATA proces foo { input: set pair_id, file(reads) from something_ch onSuccess: // do something onError: // do something else metadata_ch << [ name: task.name, failed: true, sample: pair_id, cmd_ver: task.version ] onTerminate: // an action when the last task has complete script: """ your_command $x """ }
RUNTIME CLEANUP • Workflow execution may produce a lot of temporary data • Deleting it affects the resume ability • Work in progress to cleaned it up at runtime w/o breaking the resume (issues # 452) p1 q1 t3 t2 t1 ω1 fromPath('/data/*') p1 t2 t1 q1
RESOURCES PREDICTION • Train a prediction model with real resources usage and inputs metadata for first n tasks • Predict following task execution of previous trained model process foo { time auto memory auto input: file 'some-in.txt' from in_ch output: file 'some-out.bam' into out_ch script: """ your_command --mem $task.memory """ }
WORKFLOW COMPOSITION? nextflow run prj/foo nextflow run prj/bar input: file > from ch_1 output: file > into ch_2 input: file > from ch_2 output: file > into ch_3 prj/foo + prj/bar + ... proces foo { script: workflow('prj/foo') } process bar { script: workflow('prj/bar') }
MY TAKE • Genomics data and health records are exploding • Data will be fragmented and siloed in many different public clouds and private orgs • Heterogeneous computing environment eg. clouds, HPC clusters, interactive notebooks, etc. • Hardly there could be one-size fit-all solution • Increasingly need of portable workflows and hybrid computing enabling transparent multi-cloud and platform deployments