Bio Genomics 2017

3ee44f53c39bcd4bc663a2ea0e21d526?s=47 James Taylor
February 22, 2017

Bio Genomics 2017

Galaxy, BioConda, BioContainers at http://biogenomics2017.org/

3ee44f53c39bcd4bc663a2ea0e21d526?s=128

James Taylor

February 22, 2017
Tweet

Transcript

  1. Accessible, transparent, reproducible genomic analysis with @jxtx / #usegalaxy https://speakerdeck.com/jxtx

  2. What happens to traditional research outputs when an area of

    science rapidly become data intensive?
  3. Idea Experiment Raw Data Tidy Data Summarized data Results Experimental

    design Data collection Data cleaning Data analysis Inference Data Pipeline, inspired by Leek and Peng, Nature 2015 The part we are considering here The part that ends up in the Publication
  4. Galaxy’s goals: Accessibility: Eliminate barriers for researchers wanting to use

    complex methods, make these methods available to everyone Transparency: Facilitate communication of analyses and results in ways that are easy to understand while providing all details Reproducibility*: Ensure that analysis performed in the system can be reproduced precisely and practically *The state of which is still frighteningly bad, see doi:10.1038/nrg3305, doi:10.7717/peerj.148
  5. Galaxy: accessible analysis system

  6. A free (for everyone) web service integrating a wealth of

    tools, compute resources, terabytes of reference data and permanent storage Open source software that makes integrating your own tools and data and customizing for your own site simple An open extensible platform for sharing tools, datatypes, workflows, ...
  7. Describe analysis tool behavior abstractly

  8. Describe analysis tool behavior abstractly Analysis environment automatically and transparently

    tracks details
  9. Describe analysis tool behavior abstractly Analysis environment automatically and transparently

    tracks details Workflow system for complex analysis, constructed explicitly or automatically
  10. Describe analysis tool behavior abstractly Analysis environment automatically and transparently

    tracks details Workflow system for complex analysis, constructed explicitly or automatically Pervasive sharing, and publication of documents with integrated analysis
  11. Visualization and visual analytics

  12. More Powerful Workflows Arbitrary # of Inputs (... paired). Run

    applications in parallel (one per input). Merged output for subsequent processing. Dataset collections: map/reduce workflows over 1000s of datasets Interactive tours for building realtime interactive training Interactive Environments: custom analysis in Galaxy workflows using Jupyter, …
  13. Ways to use Galaxy The public web service at http://usegalaxy.org

    Install locally with many compute environments Deploy on a cloud using Cloudman Atmosphere
  14. Galaxy was designed to be a flexible configuration driven platform

    Galaxy can be easily customized to the needs of different types of analyses by assembly different tools, workflows, visualizations, datasets… For this to work, we need to make it as easy as possible for developers to integrate and share tools
  15. The Galaxy ToolShed For early Galaxy instances, tool wrapper management

    was very ad hoc. No tracking of wrapper version information in the Galaxy database, no standard way to share. ToolShed enables not just sharing, but global identifiers and versions across all Galaxy instances.
  16. None
  17. None
  18. The Galaxy ToolShed Easy to make, version, and share tool

    wrappers, but we also need to manage the underlying software packages This is hard, and best handled by a broader community
  19. Packaging software for reproducible research

  20. https://bioconda.github.io

  21. It is now reasonable to support one major server platform

    — Linux (this is great for portability and reproducibility, but scary for other reasons — monoculture leads to fragility)
  22. Builds on Conda packaging system, designed “for installing multiple versions

    of software packages and their dependencies and switching easily between them” ~2000 recipes for software packages* All packages are built in a minimal environment to ensure isolation and portability *not even including different versions!
  23. Submit recipe to GitHub Travis CI pulls recipes and builds

    in minimal docker container Successful builds from main repo uploaded to Anaconda to be installed anywhere
  24. Containers for composing an recreating complete environments

  25. rkt Singularity

  26. Containerization Builds on Linux kernel features enabling complete isolation from

    the kernel level up Containers — lightweight environments with isolation enforced at the OS level, complete control over all software Adds a complete ecosystem for sharing, versioning, managing containers — e.g. Docker hub
  27. Galaxy + Containers Run every analysis in a clean container

    — analysis are isolated and environment is the same every time Archive that container — containers are lightweight thanks to layers — and the analysis can always be recreated
  28. Bioconda + Containers Given a set of packages and versions

    in Conda/ Bioconda, we can build a container with just that software on a minimal base image If we use the same base image, we can reconstruct exactly the same container (since we archive all binary builds of all versions) With automation, these containers can be built automatically for every package with no manual modification or intervention (e.g. mulled)
  29. Bioconda + Containers + Virtualization If we run our containers

    inside a specific (ideally minimal) known VM we can control the kernel environment as well Atmosphere funded by the National Science Foundation
  30. Tool and dependency binaries, built in minimal environment with controlled

    libs Container defines minimum environment Virtual machine controls kernel and apparent hardware environment KVM, Xen, …. Increasingly precise environment control
  31. This is the best stack for complete reproducibility we have

    ever had in bioinformatics. With the right technologies, reproducibility is possible and practical.
  32. This is the best stack for complete reproducibility we have

    ever had in bioinformatics. With the right technologies, reproducibility is possible and practical. …and for Galaxy users, it makes getting new tools into a Galaxy instance incredibly easy
  33. Galaxy’s inherent flexibility makes it an ideal platform for new

    genome projects Many relevant tools are wrapped and can now be easily used in a fully reproducible way Custom data can easily be added to the system using “Data Managers” Custom genomes can be added on the fly and then used in any of Galaxy’s genome analysis tools
  34. None
  35. Galaxy is a community Users, tool developers, maintainers, training material

    authors, … The value of the Galaxy platform comes from these community contributions Join us!
  36. ACKnowledgements Galaxy: Enis Afgan, Dannon Baker, Daniel Blankenberg, Dave Bouvier,

    Martin Čech, John Chilton, Dave Clements, Nate Coraor, Carl Eberhard, Jeremy Goecks, Björn Grüning, Sam Guerler, Mo Heydarian, Jennifer Hillman-Jackson, Anton Nekrutenko, Eric Rasche, Nicola Soranzo, Marius van den Beek JHU Data Science: Jeff Leek, Roger Peng, … Jetstream: Craig Stewart, Ian Foster, Matthew Vaughn, Nirav Merchant BioConda: Johannes Köster, Björn Grüning, Ryan Dale, Chris Tomkins-Tinch, Brad Chapman, … Other lab members: Boris Brenerman, Min Hyung Cho, Peter DeFord, German Uritskiy, Mallory Freeberg NHGRI (HG005133, HG004909, HG005542, HG005573, HG006620) NIDDK (DK065806) and NSF (DBI 0543285, DBI 0850103)
  37. (fin)