Maximising the value of research data through software publishing

Beyond Data: How containers are making the sharing and publishing of complete computational environments a possibility:
Funders have already begun to request data sharing from their researchers, and the publishing industry is responding with initiatives such as enabling data citation and linking publications to primary data. In this short talk we will ask what might come next? Data itself rarely emerges into the world complete, but often needs a series of computational steps for collection, cleaning and analysis. Up to now sharing of software has usually been a fairly static affair, but software can be a notoriously difficult to get running, with dependencies on operating systems and libraries and host systems. Container technology is making it possible to encapsulate software in a lightweight and robust way that allows sharing of computational environments. We will give an overview of this technology, look at how it is being currently used by researchers, and discuss possible future implications for the publishing industry, both in terms of how we think about core infrastructure, as well as how we might provide better services to authors and readers.

Transcript

1. Ian Mulvany Head of product innovation SAGE Publishing Maximising the value of data Through software publishing Hi, I’m Ian and I work at SAGE to help build new products to support computational social science, but today I want to provoke some thinking around how we might work towards better software publishing, with some side benefits. I’m going to be talking about using container technology as a way to get there!
2. (This is obligatory pitch slide, we just launched courses to teach social science researchers how to code!, check it out at campus.sagepub.com)
3. But back to the talk. I do want to talk about how we maximise the value of data, but let’s think for a moment about what we mean by data. It’s presented in it’s final form in the research paper, but I think it’s useful to think about where it comes from.
4. It’s inexorably tied to the means of acquisition, and as our tools have become more powerful the scale of our data has grown.
5. Acquire Data But it’s never as simple as going from acquiring the data to having the data ready to publish!
6. Acquire Data Acquire Data Clean Analyse A huge amount of work often goes into cleaning the data, analysing the data.
7. Acquire Data Acquire Data Clean Analyse Generate Data Clean Analyse Increasingly data is actually generated “in silico”
8. Acquire Data Acquire Data Clean Analyse Generate Data Clean Analyse SOFTWARE All of these green steps require software, and often custom software created by the researcher
9. < > So my claim is that if we want to get serious about data, we have to get serious about software. I want us to treat software as a First Class Citizen. And I strongly believe that code will become the next object of interest from funders, after OA and data.
10. Just a file, right? Today, though, we often just treat software as a file that we expect people to download.
11. But the problem with this is that software often does not just work out of the box.
12. App Let’s think about the app that the researcher has written.
13. App Dependency Dependency Dependency It will often have dependencies of other scripts that the researcher has written.
14. App Dependency Dependency Dependency Specific language version It might require a specific version of a language to be installed on the users system, Python 3.4 vs Python 2.7, for example
15. App Dependency Dependency Dependency Specific language version Underlying Library 1 Underlying Library 2 There might be underlying system libraries that are needed
16. App Dependency Dependency Dependency Specific language version Underlying Library 1 Underlying Library 2 Database It might need to talk to a database
17. Specific Operating System App Dependency Dependency Dependency Specific language version Underlying Library 1 Underlying Library 2 Database It might need a specific version of the actual operating system
18. Specific Operating System App Dependency Dependency Dependency Specific language version Underlying Library 1 Underlying Library 2 Database App So we can think of the app as living on top of a large stack of other software
19. An software dependencies can get really complicated.
20. This is a problem that has a long history of potential solutions from the systems ops world and I’m going to talk a bit about two of these approaches.
21. http://www.nature.com/encode/#/threads Back in 2012 Researchers used one of these tools - virtualisation - to make all of their code and data available.
22. Let’s talk for a moment about how virtualisation works.
23. It provides a piece of software that emulates a computer, running inside your computer.
24. You create a Digital snapshot of the entire operating system that your app is running in, including all of the dependencies
25. And you load that image into the virtual machine running inside your machine
26. The data and code remain separate from the paper Hosting is $$$ Complex instructions 18 GB Large artefacts But this has some downsides for research use cases
27. So I want to talk about another technology that is getting a lot of traction over the last few years
28. That technology is containerisation, and I’m going to talk about the tool docker, which is not the only route to containerisation, but which gaining a lot of traction over the last few years
29. Physical Hardware (Host) Virtual Machine Approach Operating System VM system (Hpyervisor) App OS App OS App Guest OS Physical Hardware (Host) Container / Docker Approach Operating System Docker Engine Shared Libraries Shared Dependencies App 1 App 2 App 3 LOTS O’SPACE!! Containers -> Are a lot more lightweight than virtual machines <-
30. Virtual Machines Containers Hosts per box Tens Thousands Startup time Minutes Fractions of a second Containers are more preferment that virtual machines
31. However containers run a fixed binary of the application, and they can’t write data, or change state, while they are running, so orchestrating how containers work together, can be a bit complex
32. There is another aspect of containers that I’d like to tell you about. We used to care for our software like we care for our pets. We would feed them with updates, if they got sick we would ssh in and try to make them better.
33. In the containerised world we treat instances of the software more like livestock than like pets. As it’s cheap to just spin up a new container, if one container gets sick, we just kill it and replace it.
34. We have used this technology to provide a neat feature for our new online learning platform
35. We use a containerised version of Jupyterhub to provide a hub environment to each of our learners for one of our courses
36. These other companies are making heavy use of containers to create new classes of data/software driven products!
37. Video Metadata Text Code https://www.oreilly.com/learning/regex-golf-with-peter-norvig is amazing!
38. ~3300/s Google spin up about 3.3k containers per second (2B per week) to power search This is important because as a by-product of going all in on containers, google has created some create open source tools to help with the orchestration issue. https://kubernetes.io
39. Summary can solve the problem of software dependencies allows “live coding” to be delivered over the web can wrap code, data and environment together growing ecosystem of tools to support containers requires a level of infrastructural sophistication
40. Some resources to look at • Docker https://www.docker.com • Jupyterhub https://zero-to-jupyterhub.readthedocs.io/en/latest/ • Data packages http://frictionlessdata.io/data-packages/ (just metadata) • Kubernetes https://kubernetes.io • Executable Research Compendium http://o2r.info/erc-spec/spec/ • Smart Figures http://smartfigures.net • Singularity http://singularity.lbl.gov/faq • Kubernetes guide http://partiallyattended.com/2017/09/14/kubernetes_- _as_i_learn/