Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to pass the conformance test #1

1c644e39916561a6fdcf85430865e094?s=47 Tomoya Tanjo
December 06, 2018

How to pass the conformance test #1

This is a digest of how to pass the first conformance test for Common Workflow Language (CWL).
This slide is mainly for the developers of workflow engines for CWL.

1c644e39916561a6fdcf85430865e094?s=128

Tomoya Tanjo

December 06, 2018
Tweet

Other Decks in Science

Transcript

  1. How to pass the conformance test #1 Tomoya Tanjo National

    Institute of Informatics 2018.12.6 1
  2. 2 Main target of this talk Users who write CWL

    Users who execute CWL workflows written by other people Developers of CWL platforms (workflow engines) User requirements
  3. 3 Main target of this talk Users who write CWL

    Users who execute CWL workflows written by other people Developers of CWL platforms (workflow engines) User requirements
  4. • CWL project provides conformance tests to check how a

    workflow engine supports CWL specification. – It provides 195 tests. 4 How to test your engine? $ git clone git@github.com:common-workflow-language/common-workflow-language.git $ cd common-workflow-language $ ./run_test.sh –help
  5. • Run all the tests: • It uses `cwl-runner` as

    a default workflow engine. 5 Running conformance test $ ./run_test.sh ... 153 tests passed, 13 failures, 29 unsupported features 1 tool tests failed
  6. • Run specific tests: • Run with your workflow engine:

    • Run the tests in parallel: 6 Running conformance test $ ./run_test.sh –n=1,2,4-10,… $ ./run_test.sh RUNNER=/path/to/your-own-engine $ ./run_test.sh –j=32
  7. • Let’s start with conformance test #1! – A test

    for CommandLineTool – Execute a given (or default) python script and capture the output • It consists of: – bwa-mem-tool.cwl (Tool description) – bwa-mem-job.json (Input parameters) – args.py (Python script) 7 The first conformance test
  8. 8 The first conformance test cwlVersion: v1.0 class: CommandLineTool hints:

    - class: ResourceRequirement coresMin: 2 - class: DockerRequirement dockerPull: python:2-slim inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string bwa-mem-tool.cwl
  9. 9 The first conformance test cwlVersion: v1.0 class: CommandLineTool hints:

    - class: ResourceRequirement coresMin: 2 - class: DockerRequirement dockerPull: python:2-slim inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string bwa-mem-tool.cwl Input schema Output schema Arguments Command name stdout
  10. 10 The first conformance test { "reference": { "class": "File",

    "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] } bwa-mem-job.json args.py #!/usr/bin/env python import sys import json import os args = [os.path.basename(a) for a in sys.argv[1:]] with open("cwl.output.json", "w") as f: json.dump({"args": args}, f)
  11. • Execution example: 11 The first conformance test $ cwltool

    --quiet bwa-mem-tool.cwl bwa-mem-job.json { "args": [ "bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.pe_1.fastq", "example_human_Illumina.pe_2.fastq” ] }
  12. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 12 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  13. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 13 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  14. 14 How to build the commandline: overview $ python args.py

    bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam … inputs: - id: reference … - id: reads … - id: minimum_seed_length … - id: min_std_max_min … - id: args.py … baseCommand: python arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t stdout: output.sam
  15. • First, collect CommandLineBinding (CLB) objects from `arguments` with their

    sorting keys 15 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [?, ?] “bwa” [?, ?] “mem” [?, ?] { valueFrom: … }
  16. • First, collect CommandLineBinding (CLB) objects from `arguments` with their

    sorting keys – 4.1.(1) Assign a sorting key `[position, i]` where `position` is `CommandLineBinding.position` and … – 5.1.2. Default position is 0. 16 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [?, ?] “bwa” [?, ?] “mem” [1, ?] { valueFrom: … }
  17. • First, collect CommandLineBinding (CLB) objects from `arguments` with their

    sorting keys – 4.1.(1) Assign a sorting key `[position, i]` where `position` is `CommandLineBinding.position` and … – 5.1.2. Default position is 0. 17 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [0, ?] “bwa” [0, ?] “mem” [1, ?] { valueFrom: … }
  18. • First, collect CommandLineBinding (CLB) objects from `arguments` with their

    sorting keys – 4.1.(1) …, and `i` is the index in the `arguments` list. 18 How to build the commandline: section 4.1.(1) arguments: - bwa - mem - valueFrom: $(runtime.cores) position: 1 prefix: -t Sorting key [position, i] CLB Object [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … }
  19. • Next, collect CLB object (`inputBinding` field) from `inputs` 19

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference reads minimum_s eed_length min_std_m ax_min args.py
  20. • Next, collect CLB object (`inputBinding` field) from `inputs` 20

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads minimum_s eed_length min_std_m ax_min args.py
  21. • Next, collect CLB object (`inputBinding` field) from `inputs` 21

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length min_std_m ax_min args.py
  22. • Next, collect CLB object (`inputBinding` field) from `inputs` 22

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min args.py
  23. • Next, collect CLB object (`inputBinding` field) from `inputs` 23

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py
  24. • Next, collect CLB object (`inputBinding` field) from `inputs` 24

    How to build the commandline: section 4.1.(2) inputs: - id: reference type: File inputBinding: { position: 2 } - id: reads type: type: array items: File inputBinding: { position: 3 } - id: minimum_seed_length type: int inputBinding: { position: 1, prefix: -m } - id: min_std_max_min type: { type: array, items: int } inputBinding: { position: 1, prefix: -I, itemSeparator: ",” } - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 }
  25. • Next, collect CLB object (`inputBinding` field) from `inputs` 25

    How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 }
  26. • Associate CLB objects with values from the input object

    26 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  27. • Associate CLB objects with values from the input object

    27 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  28. • Associate CLB objects with values from the input object

    28 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  29. • Associate CLB objects with values from the input object

    29 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  30. • Associate CLB objects with values from the input object

    30 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  31. • Associate CLB objects with values from the input object

    – 5.0. If an input parameter is missing …, it must be assigned a value of `null` (or the value of `default` for that parameter, if provided) 31 How to build the commandline: section 4.1.(2) inputs: … - id: args.py type: File default: class: File location: args.py inputBinding: { position: -1 } id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_s eed_length { position: 1, prefix: -m } 3 min_std_m ax_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … } { "reference": { "class": "File", "location": "chr20.fa” }, "reads": [ { "class": "File", "location": "example_human_Illumina.pe_1.fastq" }, { "class": "File", "location": "example_human_Illumina.pe_2.fastq" } ], "minimum_seed_length": 3, "min_std_max_min": [1, 2, 3, 4] }
  32. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 32 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  33. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 33 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  34. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 34 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  35. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 35 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  36. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 36 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] args.py { position: -1 } { … }
  37. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 37 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … }
  38. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 38 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … }
  39. • Create a sorting key for each object – Basically

    use `position` – `id` for tie breaker 39 How to build the commandline: section 4.1.(3) Sorting key id CLB Object Value [2, “reference”] reference { position: 2 } { … } [3, “reads”] reads { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] minimum_seed_len gth { position: 1, prefix: -m } 3 [1, “min_std_max_min”] min_std_max_min { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] args.py { position: -1 } { … } No longer needed
  40. • Sort collected CLB objects – 4.1.(4) Numeric entries sort

    before strings. 40 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … }
  41. • Sort collected CLB objects – 4.1.(4) Numeric entries sort

    before strings. 41 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … } [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ]
  42. • Sort collected CLB objects – 4.1.(4) Numeric entries sort

    before strings. 42 How to build the commandline: section 4.1.(4) Sorting key CLB Object Value [-1, “args.py”] { position: -1 } { … } [0, 0] “bwa” [0, 1] “mem” [1, 2] { valueFrom: … } [1, “min_std_max_min”] { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] [1, “minimum_seed_length”] { position: 1, prefix: -m } 3 [2, “reference”] { position: 2 } { … } [3, “reads”] { position: 3 } [ {…}, {…} ] No longer needed
  43. • Apply conversion rules in section 5.1.2. 43 How to

    build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { … } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  44. • Apply conversion rules in section 5.1.2. • args.py 44

    How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  45. • Apply conversion rules in section 5.1.2. • args.py bwa

    45 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  46. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem 46 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: … } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  47. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 47 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ] Note: see “parameter reference” (3.4, 3.5), “runtime environment” (4.2) and “ResourceRequirement” (5.12) for more details
  48. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 48 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  49. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 49 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { … } { position: 3 } [ {…}, {…} ]
  50. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 chr20.fa 50 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ {…}, {…} ]
  51. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 51 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ { …,location: ...pe_ 1.fastq}, {…, location: ….pe_2.fastq } ]
  52. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 52 How to build the commandline: section 4.1.(5) CLB Object Value { position: -1 } { class: File, location: args.py } “bwa” “mem” { valueFrom: $(runtime.cores), potition: 1, prefix: -t } { position: 1, prefix: -l, itemSeparator: “,”} [ 1, 2, 3, 4 ] { position: 1, prefix: -m } 3 { position: 2 } { class: File, location: chr20.fa } { position: 3 } [ { …,location: ...pe_ 1.fastq}, {…, location: ….pe_2.fastq } ]
  53. • Apply conversion rules in section 5.1.2. • args.py bwa

    mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 53 How to build the commandline: section 4.1.(5)
  54. • 4.1.(6) Insert the value of `baseCommand` at the beginning

    of the command line. • python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq 54 How to build the commandline: section 4.1.(6) baseCommand: python
  55. • Finally, redirect the standard output to `output.sam` • python

    args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam 55 How to build the commandline: capturning stdout stdout: output.sam
  56. • Finally, redirect the standard output to `output.sam` • python

    args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam 56 How to build the commandline: capturning stdout
  57. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 57 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  58. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 58 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  59. • Run! 59 How to build the commandline: run command

    $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam $ ls -l cwl.output.json output.sam $ cat cwl.output.json {"args": ["bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.pe_1.fastq", "example_human_Illumina.pe_2.fastq"]}
  60. • Build the commandline from the CWL file and input

    object • Run the command • Capture the output object 60 How to build the commandline: overview $ python args.py bwa mem -t 2 -I 1,2,3,4 -m 3 chr20.fa example_human_Illumina.pe_1.fastq example_human_Illumina.pe_2.fastq > ~/output.sam outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  61. • 4.4. If the output directory contains a file named

    "cwl.output.json", that file must be loaded and used as the output object. 61 How to build the commandline: capture the output object (4.4)
  62. • 4.4. If the output directory contains a file named

    "cwl.output.json", that file must be loaded and used as the output object. 62 How to build the commandline: capture the output object (4.4) $ cat cwl.output.json {"args": ["bwa", "mem", "-t", "2", "-I", "1,2,3,4", "-m", "3", "chr20.fa", "example_human_Illumina.p e_1.fastq", "example_human_Illumina.p e_2.fastq"]} Output object
  63. • Note: `outputBinding` and `glob` objects are not used for

    this test. 63 How to build the commandline: capture the output object (4.4) outputs: - id: sam type: ["null", File] outputBinding: { glob: output.sam } - id: args type: type: array items: string
  64. • Now you can develop your own workflow engine for

    CWL! • Easier first step: ExpressionTool (such as #14, #15) – Executing JavaScript codes – Almost the same but no need to construct the commandline • Advanced topics for #1 – DockerRequirement 64 Conclusion