Tweet
Tweet

Slide 1

Slide 1 text

Dissec&ng  Nipype  Workflows an  fMRI  processing  story satrajit ghosh [email protected] massachusetts institute of technology

Slide 2

Slide 2 text

CONTRIBUTORS h

Slide 3

Slide 3 text

Outline What  is  Nipype? Unwrapping  a  workflow Where  to  go  from  here?

Slide 4

Slide 4 text

Outline What  is  Nipype? Unwrapping  a  workflow Where  to  go  from  here?

Slide 5

Slide 5 text

Neuroimaging Pipelines and Interfaces nipy.org/nipype Gorgolewski et al., 2010 What is Nipype? Python In Neuroscience Brain Imaging Software Nipype

Slide 6

Slide 6 text

Poline et al., 2012

Slide 7

Slide 7 text

data source: pymvpa.org 1990 92 94 96 98 2000 02 04 06 08 2010 Afni Brainvoyager Freesurfer R Caret Fmristat FSL MVPA NiPy ANTS SPM Brainvisa software

Slide 8

Slide 8 text

? Structural Diffusion Functional transformations

Slide 9

Slide 9 text

What is Nipype? A community developed, opensource, lightweight Python library Exposes formal, common semantics Interactive exploration of brain imaging algorithms Workflow creation and execution Flexible and adaptive

Slide 10

Slide 10 text

Nipype architecture

Slide 11

Slide 11 text

Outline What  is  Nipype? Unwrapping  a  workflow Where  to  go  from  here?

Slide 12

Slide 12 text

Openfmri Workflow All datasets available at openfmri.org (also on Amazon s3) Workflow itself is part of Nipype example workflows $ python fmri_openfmri.py --datasetdir ds107

Slide 13

Slide 13 text

OpenfMRI data organization

Slide 14

Slide 14 text

30000 feet view Data and info Preprocessing Task Modeling MNI Registration Storage

Slide 15

Slide 15 text

Close-up view

Slide 16

Slide 16 text

Nodes are workflows

Slide 17

Slide 17 text

The workflow object >>> from nipype.pipeline.engine import Workflow >>> my_workflow = Workflow(name=‘concept’) :: Each node of a Workflow can be a Workflow # 30000 ft overview my_workflow.write_graph(dotfilename='ograph.dot', graph2use='orig') # close-up view my_workflow.write_graph(dotfilename='ograph.dot', graph2use='exec')

Slide 18

Slide 18 text

What are these nodes?

Slide 19

Slide 19 text

What are these nodes?

Slide 20

Slide 20 text

The node object >>> from nipype.pipeline.engine import Node >>> from nipype.interfaces.spm import Segment >>> from nipype.interfaces.fsl import ImageMaths >>> from nipype.interfaces.freesurfer import MRIConvert :: All nodes encapsulate Interfaces which wrap external programs. >>> segment = Node(Segment(), name=‘segmenter’) >>> binarize = Node(ImageMaths(op_string = ‘-nan -thr 0.5 -bin’), name=‘binarize’) >>> convert = Node(MRIConvert(out_type=‘nii’), name=‘converter’)

Slide 21

Slide 21 text

How about them edges? :: One cannot connect two outputs to a single input. >>> convert = Node(MRIConvert(out_type=‘nii’), name=‘converter’) >>> segment = Node(Segment(), name=‘segmenter’) >>> binarize = Node(ImageMaths(op_string = ‘-nan -thr 0.5 -bin’), name=‘binarize’) my_workflow.connect(convert, ‘out_file’, segment, ‘data’) my_workflow.connect(segment, ‘native_wm_image’, binarize, ‘in_file’)

Slide 22

Slide 22 text

! Inputs and outputs

Slide 23

Slide 23 text

Inputs and outputs >>> from nipype.interfaces.camino import DTIFit >>> from nipype.interfaces.spm import Realign >>> DTIFit.help() >>> Realign.help()

Slide 24

Slide 24 text

thus far Creating a Workflow Creating Nodes Connecting Nodes Finding inputs/outputs

Slide 25

Slide 25 text

Running a workflow

Slide 26

Slide 26 text

Running a workflow >>> my_workflow.run()

Slide 27

Slide 27 text

Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run()

Slide 28

Slide 28 text

Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘MultiProc’, plugin_args={‘nprocs’: 64})

Slide 29

Slide 29 text

Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘MultiProc’, plugin_args={‘nprocs’: 64}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBS’, plugin_args={‘qsub_args’: ‘-q max500’})

Slide 30

Slide 30 text

Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘MultiProc’, plugin_args={‘nprocs’: 64}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBS’, plugin_args={‘qsub_args’: ‘-q max500’}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBSGraph’, plugin_args={‘qsub_args’: ‘-q max500’})

Slide 31

Slide 31 text

Running a workflow >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run() >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘MultiProc’, plugin_args={‘nprocs’: 64}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBS’, plugin_args={‘qsub_args’: ‘-q max500’}) >>> my_workflow.base_dir = ‘/some/place’ >>> my_workflow.run(plugin=‘PBSGraph’, plugin_args={‘qsub_args’: ‘-q max500’}) Local Distributed

Slide 32

Slide 32 text

Running a workflow :: Nipype plugins define how a graph is executed. Current plugins: Linear, MultiProc, SGE/PBS/ LSF, PBSGraph, Condor/CondorDAGMan, IPython

Slide 33

Slide 33 text

things to do Run on many subjects Set parameters Rerunning Inserting your code

Slide 34

Slide 34 text

Repeating subgraphs: iterables >>> subjects = [‘sub001’, ‘sub002’, ‘sub003’ ...] >>> infosource.iterables = [('subject_id', subjects)]

Slide 35

Slide 35 text

Repeating subgraphs: iterables >>> subjects = [‘sub001’, ‘sub002’, ‘sub003’ ...] >>> infosource.iterables = [('subject_id', subjects), (‘model_id’, [1, 2])] >>> smoothnode.iterables = [(‘fwhm’, [0, 3, 10])

Slide 36

Slide 36 text

Repeating subgraphs: iterables >>> subjects = [‘sub001’, ‘sub002’, ‘sub003’ ...] >>> infosource.iterables = [('subject_id', subjects), (‘model_id’, [1, 2])] >>> smoothnode.iterables = [(‘fwhm’, [0, 3, 10]) :: iterables are like nested for-loops, that you create simply by setting a property.

Slide 37

Slide 37 text

Rerun affected nodes only >>> subjects = [‘sub001’] >>> infosource.iterables = [('subject_id', subjects), (‘model_id’, [1, 2])] >>> smoothnode.iterables = [(‘fwhm’, [6]) :: Using content or timestamp hashing Nipype tracks inputs and only reruns nodes with changed inputs >>> subjects = [‘sub001’, ‘sub002’, ‘sub003] >>> infosource.iterables = [('subject_id', subjects), (‘model_id’, [1, 2])] >>> smoothnode.iterables = [(‘fwhm’, [0, 6])

Slide 38

Slide 38 text

Running your own code def get_contrasts(base_dir, model_id): contrast_file = os.path.join(base_dir, 'models', 'model%03d' % model_id,'task_contrasts.txt') ... return contrasts contrastgen = Node(Function(input_names=['base_dir', 'model_id'], output_names=['contrasts'], function=get_contrasts), name='generate_contrasts') >>> from nipype.interfaces.utility import Function

Slide 39

Slide 39 text

Nipype features covered Crea&ng  Nodes  and  Workflows Distributed  computa&on Using  Iterables Func&on  nodes Rerunning  par&al  workflows

Slide 40

Slide 40 text

Other Nipype features Using  Nipype  as  an  interface  library Nipype  caching Crea&ng  MapReduce  like  nodes Connec&ng  to  XNAT Execu&on  configura&on  op&ons

Slide 41

Slide 41 text

Outline What  is  Nipype? Unwrapping  a  workflow Where  to  go  from  here?

Slide 42

Slide 42 text

What next? Explore interfaces and workflows: - PredictHD project workflows - CPAC workflows - BIPS workflows http://nipy.org/nipype/quickstart.html Contribute back: - Report issues - Create new interfaces and workflows - Write tests for existing code

Slide 43

Slide 43 text

nipy.org/nipype