Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to pass the conformance test #1

Tomoya Tanjo
December 06, 2018

How to pass the conformance test #1

This is a digest of how to pass the first conformance test for Common Workflow Language (CWL).
This slide is mainly for the developers of workflow engines for CWL.

Tomoya Tanjo

December 06, 2018
Tweet

Other Decks in Science

Transcript

  1. How to pass the conformance test #1 Tomoya Tanjo National

    Institute of Informatics 2018.12.6 1
  2. 2 Main target of this talk Users who write CWL

    Users who execute CWL workflows written by other people Developers of CWL platforms (workflow engines) User requirements
  3. 3 Main target of this talk Users who write CWL

    Users who execute CWL workflows written by other people Developers of CWL platforms (workflow engines) User requirements
  4. • CWL project provides conformance tests to check how a

    workflow engine supports CWL specification. – It provides 195 tests. 4 How to test your engine? $ git clone [email protected]:common-workflow-language/common-workflow-language.git $ cd common-workflow-language $ ./run_test.sh –help
  5. • Run all the tests: • It uses `cwl-runner` as

    a default workflow engine. 5 Running conformance test $ ./run_test.sh ... 153 tests passed, 13 failures, 29 unsupported features 1 tool tests failed
  6. • Run specific tests: • Run with your workflow engine:

    • Run the tests in parallel: 6 Running conformance test $ ./run_test.sh –n=1,2,4-10,… $ ./run_test.sh RUNNER=/path/to/your-own-engine $ ./run_test.sh –j=32
  7. • Let’s start with conformance test #1! – A test

    for CommandLineTool – Execute a given (or default) python script and capture the output • It consists of: – bwa-mem-tool.cwl (Tool description) – bwa-mem-job.json (Input parameters) – args.py (Python script) 7 The first conformance test
  8. 8 The first conformance test cwlVersion: v1.0 class: CommandLineTool hints:

    - class: ResourceRequirement coresMin: 2 - class: DockerRequirement dockerPull: python:2-slim inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string bwa-mem-tool.cwl
  9. 9 The first conformance test cwlVersion: v1.0 class: CommandLineTool hints:

    - class: ResourceRequirement coresMin: 2 - class: DockerRequirement dockerPull: python:2-slim inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string bwa-mem-tool.cwl Input schema Output schema Arguments Command name stdout
  10. 10 The first conformance test { "reference": { "class": "File",

    "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] } bwa-mem-job.json args.py #!/usr/bin/env python import sys import json import os args = [os.path.basename(a) for a in sys.argv[1:]] with open("cwl.output.json", "w") as f: json.dump({"args": args}, f)
  11. • Execution example: 11 The first conformance test $ cwltool

    --quiet bwa-mem-tool.cwl bwa-mem-job.json { "args": [ "bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.pe_1.fastq", "example_human_Illumina.pe_2.fastq” ] }
  12. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 12 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  13. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 13 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  14. 14 How to build the commandline: overview $ python args.py

    bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam … inputs: - id: reference … - id: reads … - id: minimum_seed_length … - id: min_std_max_min … - id: args.py … baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam
  15. • First, collect CommandLineBinding (CLB) objects from `arguments` with their

    sorting keys 15 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [?, ?] “bwa” [?, ?] “mem” [?, ?] { valueFrom: … }
  16. • First, collect CommandLineBinding (CLB) objects from `arguments` with their

    sorting keys – 4.1.(1) Assign a sorting key `[position, i]` where `position` is `CommandLineBinding.position` and … – 5.1.2. Default position is 0. 16 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [?, ?] “bwa” [?, ?] “mem” [1, ?] { valueFrom: … }
  17. • First, collect CommandLineBinding (CLB) objects from `arguments` with their

    sorting keys – 4.1.(1) Assign a sorting key `[position, i]` where `position` is `CommandLineBinding.position` and … – 5.1.2. Default position is 0. 17 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [0, ?] “bwa” [0, ?] “mem” [1, ?] { valueFrom: … }
  18. • First, collect CommandLineBinding (CLB) objects from `arguments` with their

    sorting keys – 4.1.(1) …, and `i` is the index in the `arguments` list. 18 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … }
  19. • Next, collect CLB object (`inputBinding` field) from `inputs` 19

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference reads minimum_s eed_length min_std_m ax_min args.py
  20. • Next, collect CLB object (`inputBinding` field) from `inputs` 20

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads minimum_s eed_length min_std_m ax_min args.py
  21. • Next, collect CLB object (`inputBinding` field) from `inputs` 21

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length min_std_m ax_min args.py
  22. • Next, collect CLB object (`inputBinding` field) from `inputs` 22

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min args.py
  23. • Next, collect CLB object (`inputBinding` field) from `inputs` 23

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py
  24. • Next, collect CLB object (`inputBinding` field) from `inputs` 24

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 }
  25. • Next, collect CLB object (`inputBinding` field) from `inputs` 25

    How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 }
  26. • Associate CLB objects with values from the input object

    26 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  27. • Associate CLB objects with values from the input object

    27 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  28. • Associate CLB objects with values from the input object

    28 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  29. • Associate CLB objects with values from the input object

    29 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  30. • Associate CLB objects with values from the input object

    30 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  31. • Associate CLB objects with values from the input object

    – 5.0. If an input parameter is missing …, it must be assigned a value of `null` (or the value of `default` for that parameter, if provided) 31 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  32. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 32 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  33. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 33 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  34. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 34 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  35. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 35 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  36. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 36 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  37. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 37 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … }
  38. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 38 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … }
  39. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 39 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … } No longer needed
  40. • Sort collected CLB objects – 4.1.(4) Numeric entries sort

    before strings. 40 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … }
  41. • Sort collected CLB objects – 4.1.(4) Numeric entries sort

    before strings. 41 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … } [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ]
  42. • Sort collected CLB objects – 4.1.(4) Numeric entries sort

    before strings. 42 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … } [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ] No longer needed
  43. • Apply conversion rules in section 5.1.2. 43 How to

    build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { … } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  44. • Apply conversion rules in section 5.1.2. • args.py 44

    How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  45. • Apply conversion rules in section 5.1.2. • args.py bwa

    45 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  46. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem 46 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  47. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 47 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ] Note: see “parameter reference” (3.4, 3.5), “runtime environment” (4.2) and “ResourceRequirement” (5.12) for more details
  48. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 48 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  49. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 49 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  50. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 chr20.fa 50 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ {…}, {…} ]
  51. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 51 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ { …,location: ...pe_ 1.fastq}, {…, location: ….pe_2.fastq } ]
  52. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 52 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ { …,location: ...pe_ 1.fastq}, {…, location: ….pe_2.fastq } ]
  53. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 53 How to build the commandline: section 4.1.(5)
  54. • 4.1.(6) Insert the value of `baseCommand` at the beginning

    of the command line. • python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 54 How to build the commandline: section 4.1.(6) baseCommand: python
  55. • Finally, redirect the standard output to `output.sam` • python

    args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam 55 How to build the commandline: capturning stdout stdout: output.sam
  56. • Finally, redirect the standard output to `output.sam` • python

    args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam 56 How to build the commandline: capturning stdout
  57. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 57 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  58. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 58 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  59. • Run! 59 How to build the commandline: run command

    $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam $ ls -l cwl.output.json output.sam $ cat cwl.output.json {"args": ["bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.pe_1.fastq", "example_human_Illumina.pe_2.fastq"]}
  60. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 60 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  61. • 4.4. If the output directory contains a file named

    "cwl.output.json", that file must be loaded and used as the output object. 61 How to build the commandline: capture the output object (4.4)
  62. • 4.4. If the output directory contains a file named

    "cwl.output.json", that file must be loaded and used as the output object. 62 How to build the commandline: capture the output object (4.4) $ cat cwl.output.json {"args": ["bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.p e_1.fastq", "example_human_Illumina.p e_2.fastq"]} Output object
  63. • Note: `outputBinding` and `glob` objects are not used for

    this test. 63 How to build the commandline: capture the output object (4.4) outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  64. • Now you can develop your own workflow engine for

    CWL! • Easier first step: ExpressionTool (such as #14, #15) – Executing JavaScript codes – Almost the same but no need to construct the commandline • Advanced topics for #1 – DockerRequirement 64 Conclusion