How to pass the conformance test #1

This is a digest of how to pass the first conformance test for Common Workflow Language (CWL).
This slide is mainly for the developers of workflow engines for CWL.

Transcript

1. How to pass the conformance test #1 Tomoya Tanjo National Institute of Informatics 2018.12.6 1
2. 2 Main target of this talk Users who write CWL Users who execute CWL workflows written by other people Developers of CWL platforms (workflow engines) User requirements
3. 3 Main target of this talk Users who write CWL Users who execute CWL workflows written by other people Developers of CWL platforms (workflow engines) User requirements
4. • CWL project provides conformance tests to check how a workflow engine supports CWL specification. – It provides 195 tests. 4 How to test your engine? $ git clone git@github.com:common-workflow-language/common-workflow-language.git $ cd common-workflow-language $ ./run_test.sh –help
5. • Run all the tests: • It uses `cwl-runner` as a default workflow engine. 5 Running conformance test $ ./run_test.sh ... 153 tests passed, 13 failures, 29 unsupported features 1 tool tests failed
6. • Run specific tests: • Run with your workflow engine: • Run the tests in parallel: 6 Running conformance test $ ./run_test.sh –n=1,2,4-10,… $ ./run_test.sh RUNNER=/path/to/your-own-engine $ ./run_test.sh –j=32
7. • Let’s start with conformance test #1! – A test for CommandLineTool – Execute a given (or default) python script and capture the output • It consists of: – bwa-mem-tool.cwl (Tool description) – bwa-mem-job.json (Input parameters) – args.py (Python script) 7 The first conformance test
8. 8 The first conformance test cwlVersion: v1.0 class: CommandLineTool hints: - class: ResourceRequirement coresMin: 2 - class: DockerRequirement dockerPull: python:2-slim inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string bwa-mem-tool.cwl
9. 9 The first conformance test cwlVersion: v1.0 class: CommandLineTool hints: - class: ResourceRequirement coresMin: 2 - class: DockerRequirement dockerPull: python:2-slim inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string bwa-mem-tool.cwl Input schema Output schema Arguments Command name stdout
10. 10 The first conformance test { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] } bwa-mem-job.json args.py #!/usr/bin/env python import sys import json import os args = [os.path.basename(a) for a in sys.argv[1:]] with open("cwl.output.json", "w") as f: json.dump({"args": args}, f)
11. • Execution example: 11 The first conformance test $ cwltool --quiet bwa-mem-tool.cwl bwa-mem-job.json { "args": [ "bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.pe_1.fastq", "example_human_Illumina.pe_2.fastq” ] }
12. • Build the commandline from the CWL file and input object • Run the command • Capture the output object 12 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
13. • Build the commandline from the CWL file and input object • Run the command • Capture the output object 13 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
14. 14 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam … inputs: - id: reference … - id: reads … - id: minimum_seed_length … - id: min_std_max_min … - id: args.py … baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam
15. • First, collect CommandLineBinding (CLB) objects from `arguments` with their sorting keys 15 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [?, ?] “bwa” [?, ?] “mem” [?, ?] { valueFrom: … }
16. • First, collect CommandLineBinding (CLB) objects from `arguments` with their sorting keys – 4.1.(1) Assign a sorting key `[position, i]` where `position` is `CommandLineBinding.position` and … – 5.1.2. Default position is 0. 16 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [?, ?] “bwa” [?, ?] “mem” [1, ?] { valueFrom: … }
17. • First, collect CommandLineBinding (CLB) objects from `arguments` with their sorting keys – 4.1.(1) Assign a sorting key `[position, i]` where `position` is `CommandLineBinding.position` and … – 5.1.2. Default position is 0. 17 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [0, ?] “bwa” [0, ?] “mem” [1, ?] { valueFrom: … }
18. • First, collect CommandLineBinding (CLB) objects from `arguments` with their sorting keys – 4.1.(1) …, and `i` is the index in the `arguments` list. 18 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … }
19. • Next, collect CLB object (`inputBinding` field) from `inputs` 19 How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference reads minimum_s eed_length min_std_m ax_min args.py
20. • Next, collect CLB object (`inputBinding` field) from `inputs` 20 How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads minimum_s eed_length min_std_m ax_min args.py
21. • Next, collect CLB object (`inputBinding` field) from `inputs` 21 How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length min_std_m ax_min args.py
22. • Next, collect CLB object (`inputBinding` field) from `inputs` 22 How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min args.py
23. • Next, collect CLB object (`inputBinding` field) from `inputs` 23 How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py
24. • Next, collect CLB object (`inputBinding` field) from `inputs` 24 How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 }
25. • Next, collect CLB object (`inputBinding` field) from `inputs` 25 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 }
26. • Associate CLB objects with values from the input object 26 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
27. • Associate CLB objects with values from the input object 27 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
28. • Associate CLB objects with values from the input object 28 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
29. • Associate CLB objects with values from the input object 29 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
30. • Associate CLB objects with values from the input object 30 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
31. • Associate CLB objects with values from the input object – 5.0. If an input parameter is missing …, it must be assigned a value of `null` (or the value of `default` for that parameter, if provided) 31 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
32. • Create a sorting key for each object – Basically use `position` – `id` for tie breaker 32 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
33. • Create a sorting key for each object – Basically use `position` – `id` for tie breaker 33 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
34. • Create a sorting key for each object – Basically use `position` – `id` for tie breaker 34 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
35. • Create a sorting key for each object – Basically use `position` – `id` for tie breaker 35 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
36. • Create a sorting key for each object – Basically use `position` – `id` for tie breaker 36 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
37. • Create a sorting key for each object – Basically use `position` – `id` for tie breaker 37 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … }
38. • Create a sorting key for each object – Basically use `position` – `id` for tie breaker 38 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … }
39. • Create a sorting key for each object – Basically use `position` – `id` for tie breaker 39 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … } No longer needed
40. • Sort collected CLB objects – 4.1.(4) Numeric entries sort before strings. 40 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … }
41. • Sort collected CLB objects – 4.1.(4) Numeric entries sort before strings. 41 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … } [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ]
42. • Sort collected CLB objects – 4.1.(4) Numeric entries sort before strings. 42 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … } [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ] No longer needed
43. • Apply conversion rules in section 5.1.2. 43 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { … } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
44. • Apply conversion rules in section 5.1.2. • args.py 44 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
45. • Apply conversion rules in section 5.1.2. • args.py bwa 45 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
46. • Apply conversion rules in section 5.1.2. • args.py bwa mem 46 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
47. • Apply conversion rules in section 5.1.2. • args.py bwa mem -t 2 47 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ] Note: see “parameter reference” (3.4, 3.5), “runtime environment” (4.2) and “ResourceRequirement” (5.12) for more details
48. • Apply conversion rules in section 5.1.2. • args.py bwa mem -t 2 -I 1,2,3,4 48 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
49. • Apply conversion rules in section 5.1.2. • args.py bwa mem -t 2 -I 1,2,3,4 -m 3 49 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
50. • Apply conversion rules in section 5.1.2. • args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa 50 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ {…}, {…} ]
51. • Apply conversion rules in section 5.1.2. • args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 51 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ { …,location: ...pe_ 1.fastq}, {…, location: ….pe_2.fastq } ]
52. • Apply conversion rules in section 5.1.2. • args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 52 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ { …,location: ...pe_ 1.fastq}, {…, location: ….pe_2.fastq } ]
53. • Apply conversion rules in section 5.1.2. • args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 53 How to build the commandline: section 4.1.(5)
54. • 4.1.(6) Insert the value of `baseCommand` at the beginning of the command line. • python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 54 How to build the commandline: section 4.1.(6) baseCommand: python
55. • Finally, redirect the standard output to `output.sam` • python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam 55 How to build the commandline: capturning stdout stdout: output.sam
56. • Finally, redirect the standard output to `output.sam` • python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam 56 How to build the commandline: capturning stdout
57. • Build the commandline from the CWL file and input object • Run the command • Capture the output object 57 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
58. • Build the commandline from the CWL file and input object • Run the command • Capture the output object 58 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
59. • Run! 59 How to build the commandline: run command $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam $ ls -l cwl.output.json output.sam $ cat cwl.output.json {"args": ["bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.pe_1.fastq", "example_human_Illumina.pe_2.fastq"]}
60. • Build the commandline from the CWL file and input object • Run the command • Capture the output object 60 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
61. • 4.4. If the output directory contains a file named "cwl.output.json", that file must be loaded and used as the output object. 61 How to build the commandline: capture the output object (4.4)
62. • 4.4. If the output directory contains a file named "cwl.output.json", that file must be loaded and used as the output object. 62 How to build the commandline: capture the output object (4.4) $ cat cwl.output.json {"args": ["bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.p e_1.fastq", "example_human_Illumina.p e_2.fastq"]} Output object
63. • Note: `outputBinding` and `glob` objects are not used for this test. 63 How to build the commandline: capture the output object (4.4) outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
64. • Now you can develop your own workflow engine for CWL! • Easier first step: ExpressionTool (such as #14, #15) – Executing JavaScript codes – Almost the same but no need to construct the commandline • Advanced topics for #1 – DockerRequirement 64 Conclusion