The Swinburne Pulsar Portal: Real-time Supercomputing Processing of Big Data

Presented at the Astroinformatics 2013: Knowledge from Data conference December 11, 2013

Conference website: http://www.atnf.csiro.au/research/workshops/2013/astroinformatics/program.html

Transcript

1. Arna Karick eResearch Consultant/Data Analyst/Astro Swinburne Research Swinburne Pulsar Portal Real-time Supercomputing Processing of Big Data
2. This project is an extension of The Swinburne University of Technology Metadata Stores Project and partly supported by the Australian National Data Service (ANDS) ANDS is supported by the Australian Government through the National Collaborative Research Infrastructure Strategy Program and the Education Investment Fund (EIF) Super Science Initiative
3. The ERA of All-Sky Science @ radio, optical and infrared wavelengths... ASKAP: Wallaby all-sky HI survey - 620 Giga voxels/2.5 TB data cubes MWA: All-sky radio survey, ~ 6 PB/yr archived data WISE: All-sky IR survey – recent AllWISE data release (Nov 2013) Source catalog: 747 million objects VST ATLAS: 4500 square deg. of the Southern sky (U, V, R, I, Z) VPHAS+: VST H-alpha Survey of the Southern Galactic Plane + Molonglo Observatory Synthesis Telescope (MOST) WISE IR Survey
4. Value of Data Access & Analysis Tools for researchers and citizen scientists... Hubble Legacy Archive HST & SDSS obviously.... • Dealing with the data deluge • Efficient research (less reinventing the wheel) • Greater Exposure • New Collaborations • Publications & increased citations • New Discoveries
5. Swinburne Pulsar Portal - The Who? Matthew Bailes: Research Astronomer & Pro-Vice Chancellor (Research) Andrew Jameson: Software Developer & Systems Engineer Chris Flynn: Research Astronomer (Molonglo Telescope) Willem van Straten: Research Astronomer (Software & Instrumentation) Ewan Barr: Pulsar Postdoc (HTRU reprocessing & Molonglo) Arna Karick: eResearch (Research Data Management & Policy) & Astronomer (optical: galaxy clusters & ETGs)
6. Swinburne Pulsar Portal - The What? online tool facilitating remote access to and processing of CSIRO Parkes pulsar data Survey snapshot
7. High Time Resolution Universe - HTRU (P630) • Paper I - Keith et al. (2010) + discovery papers • Collaborative Research (Swinburne, Manchester, ATNF, Cagliari) • Low-lat survey: thin strip Galactic Plane (deep) -> faint pulsars • Med-lat survey: bright MSP for timing array projevys • High-lat survey: snapshot of transient sky (Sth +10) • Rotating radio transients, short duration radio bursts • Running for ~5 years, over 100 new pulsars, including 26 ms pulsars • Survey has produced over 600 Tb of raw data (Total ~875 Tb) • Data archived to tape & streamed to Swinburne via 1 Gb/s link - cont. observing Pulsar Timing Array projects (P140) • Detection of gravitational waves High Time Resolution Universe North (HTRU - North) • Effelsburg Radio Telescope possibly... in consultation with research groups
8. Swinburne Pulsar Portal - The How? • User friendly web interface • Sophisicated analysis tools backed by significant processing power • MySQL data base with a PHP frontend. • Accesses a Pb scale database • XML headers for instrumental and astrophysical metadata - format independent & editable - facilitates easy indexing • Uses the supercomputer (gSTAR) batch queue system - email alerts - currently has ‘timeout’ in place • Modular - datasets and analysis tools can be added over time • Attempt to write ‘non-expert’ analysis tools
9. Swinburne Pulsar Portal - The Why? • Sharing of collaborative datasets - secure / proprietory periods • Target: project collaborators, registered astronomers.. public? • Enables users to query AND analyse data and download data products (metadata available via CSIROs Data Access Facility) • Alleviates Tb-Pb storage issues, and the guesswork associated with setup & maintenance of software & hardware infrastructure • Access pulsar observations, search object catalogues, process time-series data with sophisticated analysis software • Test & validate analysis techniques (for Parkes data, Molonglo & SKA) Improved multi-processing (eg. orbital solutions for high-eccenticity binaries) • Science-ready results & greater discovery potential
10. Swinburne Pulsar Portal – Data Processing
11. Swinburne Pulsar Portal – Data Tools standard routines + novel techniques candidate sorting folding & optimisation pulse periods plotting software editabe? user software? e.g Geophysics VO on NeCTar
12. Swinburne Pulsar Portal – Data Products failures ~2% e.g. beams with interference/ timeouts
13. Swinburne Pulsar Portal – Data Products
14. Swinburne Pulsar Portal – Data Products
15. Swinburne Pulsar Portal – Data Products
16. Coming Soon... Mid-2014
17. Other Projects • MyTardis@Swinburne data solutions - Brain Imaging: MEG & EEG - Microscopy/Eng: raman spectrometer & confocal microscope • Research Data Management & Policy - Instutional research storage, Cloudstor+ (file sharing & storage) - Research Conduct policy, analytics & strategy • Swinburne (ANDS) Metadata Store Project - Research data collections for Research Data Australia - Copyright, software licencing, DOIs • Astronomy Research - HST/ACS Coma Cluster Treasury Survey - HST imaging of the Atlas3D galaxy sample