OHBM 'Big Data' Workshop: Dissecting Nipype Workflows

A description of the Nipype (nipy.org/nipype) platform for analyzing brain imaging data with Python workflows.

1. Dissec&ng  Nipype  Workflows an  fMRI  processing  story satrajit ghosh satra@mit.edu massachusetts institute of technology
2. CONTRIBUTORS h<p://nipy.org/nipype/about.html#code-­‐ contributors FUNDING 1R03EB008673-­‐01  (NIBIB) 5R01MH081909-­‐02  (NIMH) INCF CONFLICT  OF  INTEREST TankThink  Labs,  LLC
3. Outline What  is  Nipype? Unwrapping  a  workflow Where  to  go  from  here?
4. Outline What  is  Nipype? Unwrapping  a  workflow Where  to  go  from  here?
5. Neuroimaging Pipelines and Interfaces nipy.org/nipype Gorgolewski et al., 2010 What is Nipype? Python In Neuroscience Brain Imaging Software Nipype
6. Poline et al., 2012
7. data source: pymvpa.org 1990 92 94 96 98 2000 02 04 06 08 2010 Afni Brainvoyager Freesurfer R Caret Fmristat FSL MVPA NiPy ANTS SPM Brainvisa software
8. ? Structural Diffusion Functional transformations
9. What is Nipype? A community developed, opensource, lightweight Python library Exposes formal, common semantics Interactive exploration of brain imaging algorithms Workflow creation and execution Flexible and adaptive
10. Nipype architecture
11. Outline What  is  Nipype? Unwrapping  a  workflow Where  to  go  from  here?
12. Openfmri Workflow All datasets available at openfmri.org (also on Amazon s3) Workflow itself is part of Nipype example workflows $ python fmri_openfmri.py --datasetdir ds107
13. OpenfMRI data organization
14. 30000 feet view Data and info Preprocessing Task Modeling MNI Registration Storage
15. Close-up view
16. Nodes are workflows
17. The workflow object >>> from nipype.pipeline.engine import Workflow >>> my_workflow = Workflow(name=‘concept’) :: Each node of a Workflow can be a Workflow # 30000 ft overview my_workflow.write_graph(dotfilename='ograph.dot', graph2use='orig') # close-up view my_workflow.write_graph(dotfilename='ograph.dot', graph2use='exec')
18. What are these nodes?
19. What are these nodes?
20. The node object >>> from nipype.pipeline.engine import Node >>> from nipype.interfaces.spm import Segment >>> from nipype.interfaces.fsl import ImageMaths >>> from nipype.interfaces.freesurfer import MRIConvert :: All nodes encapsulate Interfaces which wrap external programs. >>> segment = Node(Segment(), name=‘segmenter’) >>> binarize = Node(ImageMaths(op_string = ‘-nan -thr 0.5 -bin’), name=‘binarize’) >>> convert = Node(MRIConvert(out_type=‘nii’), name=‘converter’)
21. How about them edges? :: One cannot connect two outputs to a single input. >>> convert = Node(MRIConvert(out_type=‘nii’), name=‘converter’) >>> segment = Node(Segment(), name=‘segmenter’) >>> binarize = Node(ImageMaths(op_string = ‘-nan -thr 0.5 -bin’), name=‘binarize’) my_workflow.connect(convert, ‘out_file’, segment, ‘data’) my_workflow.connect(segment, ‘native_wm_image’, binarize, ‘in_file’)
22. ! Inputs and outputs
23. Inputs and outputs >>> from nipype.interfaces.camino import DTIFit >>> from nipype.interfaces.spm import Realign >>> DTIFit.help() >>> Realign.help()
24. thus far Creating a Workflow Creating Nodes Connecting Nodes Finding inputs/outputs
25. Running a workflow
26. Running a workflow >>> my_workflow.run()
27. Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run()
28. Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘MultiProc’, plugin_args={‘nprocs’: 64})
29. Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘MultiProc’, plugin_args={‘nprocs’: 64}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBS’, plugin_args={‘qsub_args’: ‘-q max500’})
30. Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘MultiProc’, plugin_args={‘nprocs’: 64}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBS’, plugin_args={‘qsub_args’: ‘-q max500’}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBSGraph’, plugin_args={‘qsub_args’: ‘-q max500’})
31. Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘MultiProc’, plugin_args={‘nprocs’: 64}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBS’, plugin_args={‘qsub_args’: ‘-q max500’}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBSGraph’, plugin_args={‘qsub_args’: ‘-q max500’}) Local Distributed
32. Running a workflow :: Nipype plugins define how a graph is executed. Current plugins: Linear, MultiProc, SGE/PBS/ LSF, PBSGraph, Condor/CondorDAGMan, IPython
33. things to do Run on many subjects Set parameters Rerunning Inserting your code
34. Repeating subgraphs: iterables >>> subjects = [‘sub001’, ‘sub002’, ‘sub003’ ...] >>> infosource.iterables = [('subject_id', subjects)]
35. Repeating subgraphs: iterables >>> subjects = [‘sub001’, ‘sub002’, ‘sub003’ ...] >>> infosource.iterables = [('subject_id', subjects), (‘model_id’, [1, 2])] >>> smoothnode.iterables = [(‘fwhm’, [0, 3, 10])
36. Repeating subgraphs: iterables >>> subjects = [‘sub001’, ‘sub002’, ‘sub003’ ...] >>> infosource.iterables = [('subject_id', subjects), (‘model_id’, [1, 2])] >>> smoothnode.iterables = [(‘fwhm’, [0, 3, 10]) :: iterables are like nested for-loops, that you create simply by setting a property.
37. Rerun affected nodes only >>> subjects = [‘sub001’] >>> infosource.iterables = [('subject_id', subjects), (‘model_id’, [1, 2])] >>> smoothnode.iterables = [(‘fwhm’, [6]) :: Using content or timestamp hashing Nipype tracks inputs and only reruns nodes with changed inputs >>> subjects = [‘sub001’, ‘sub002’, ‘sub003] >>> infosource.iterables = [('subject_id', subjects), (‘model_id’, [1, 2])] >>> smoothnode.iterables = [(‘fwhm’, [0, 6])
38. Running your own code def get_contrasts(base_dir, model_id): contrast_file = os.path.join(base_dir, 'models', 'model%03d' % model_id,'task_contrasts.txt') ... return contrasts contrastgen = Node(Function(input_names=['base_dir', 'model_id'], output_names=['contrasts'], function=get_contrasts), name='generate_contrasts') >>> from nipype.interfaces.utility import Function
39. Nipype features covered Crea&ng  Nodes  and  Workflows Distributed  computa&on Using  Iterables Func&on  nodes Rerunning  par&al  workflows
40. Other Nipype features Using  Nipype  as  an  interface  library Nipype  caching Crea&ng  MapReduce  like  nodes Connec&ng  to  XNAT Execu&on  configura&on  op&ons
41. Outline What  is  Nipype? Unwrapping  a  workflow Where  to  go  from  here?
42. What next? Explore interfaces and workflows: - PredictHD project workflows - CPAC workflows - BIPS workflows http://nipy.org/nipype/quickstart.html Contribute back: - Report issues - Create new interfaces and workflows - Write tests for existing code
43. nipy.org/nipype