John Swinbank

LOFAR Transients Software Overview

1. LOFAR Transients Software Overview swinbank@transientskp.org John Swinbank
2. The Goal • The Radio Sky Monitor • Large fraction of the sky • 1--1000s second cadence • Detect & respond in real time • Public alerts & mineable lightcurve archive
3. The (Immediate) Goal • Search all MSSS data for transients. • Calibrated visibilities provided. • We make our own images. • 5940 LBA, 7232 HBA observations. Figures: Heald, http://www.astron.nl/~heald/msss/
4. The Components • Now: • LOFAR imaging • The “Trap” • Sourcefinder • Database • Classification • Visualization • VOEvent system • Next: • LOFAR 2.0 • Real-time imaging • Responsive telescope • Trap upgrade • Speed • Volume • Classification
5. LOFAR Imaging Figure: Heald
6. LOFAR Imaging Figure: Heald
7. LOFAR Imaging Figure: Heald
8. LOFAR Imaging Figure: Heald
9. LOFAR Imaging Figure: Heald
10. LOFAR Imaging Figure: Heald
11. Flagging • How does AOFlagger respond to transients? • How can we effectively flag with minimal time- series available? • Tests: • Insert transients into existing data. • Experiment with new strategies. (Calvelo,) Cendes Figures: Calvelo
12. Imaging • awimager, the new LOFAR imager, is now in widespread use. • Extensive commissioning at ASTRON through this year to date. • Many TKP members participated! • How can we optimise speed and accuracy for transients? van der Horst, Rowlinson
13. The Grid Swinbank • The LOFAR Long Term Archive is based on “Grid” technologies. • Data storage. • (Re)processing. • TKP (with help from SARA) pioneered the use of NDPPP in the LTA. • Much of the Bell#1 data (Jess’s talk) was flagged/compressed on the Grid.
14. Real-time imaging GPU Cluster Imaging BlueGene/P Flagging and Compression Correlation CEP Cluster Transients Pipeline Calibration LOFAR Stations • Coming soon: LOFAR 2.0. • Flagging on BlueGene? • How long is calibration valid? • GPU imaging? • BlueGene successor? • ...? • We need to provide input!
15. It’s a trap! MonetDB Database Imaging Pipeline Image Cube Source Finding Lightcurve Storage Transient & Variability Analysis Source Association Archive Database Classification & Analysis Response Scheduling Send External Alert Re-run Image Analysis Schedule New Observation Other Facilities Receive External Alert Real-time Processing Off-line & External Systems On-line Processing Visibility Data Scheduler Molenaar, Rowlinson, (Rol,) Staley, Swinbank
16. Sourcefinder Carbone, Garsden, van der Horst, (Spreeuw), Swinbank • Essential both for trap and LOFAR imaging. • Historical duplication: PyBDSM, TKP. • Minimal development recently. • Two pronged approach: • Support for ASTRON sourcefinder comparison project. • Renewed discussion/development on TKP specific requirements. • No firm conclusion yet.
17. Database Scheers Archive Database Multiplex Funnel LTA 1 LTA 2 LTA 3 LTA 4 LTA 8 Hot Spare Pipeline Database "trap" pipeline ... Hot Spare • Prototype LOFAR long-term lightcurve archive. • 50-100 TB/year insertion. • Same infrastructure supports pipeline operations & archive. • Next steps: • LTA integration. • Next-generation query language (SciQL). http://www.monetdb.org/ http://www.scilens.org/
18. Classification • Essential to understand 100s of TB & respond promptly to new events. • Astronomer-defined and (post- MSSS) machine learning. • Both cases start by breaking the lightcurve into “features”. • ➜ Major effort post-MSSS. (Coenen, Rol)
19. trap: missing features • Initial version: • Cross-frequency source association. • Ingest quality control. • Store diagnostics in database. • Next generation: • Classification & response. • Performance. • Smarter transient identification & association.
20. Visualization Molenaar, Rowlinson, (Rol,) Staley, Swinbank http://archive.transientskp.org/ (password required)
21. Visualization Molenaar, Rowlinson, (Rol,) Staley, Swinbank http://archive.transientskp.org/ (password required)
22. Visualization Molenaar, Rowlinson, (Rol,) Staley, Swinbank http://archive.transientskp.org/ (password required)
23. Visualization Molenaar, Rowlinson, (Rol,) Staley, Swinbank http://archive.transientskp.org/ (password required)
24. Visualization Molenaar, Rowlinson, (Rol,) Staley, Swinbank http://archive.transientskp.org/ (password required)
25. VOEvent Staley, Swinbank http://www.voevent.org/ http://comet.transientskp.org/ https://github.com/timstaley/pysovo • Authenticity? • Transport? • Repositories? • “Building VOEventNet” Figures: Williams et al. (2012), IVOA VOEvent WG
26. Responsive LOFAR • On the agenda for LOFAR 2.0. • Integration of VOEventNet with LOFAR scheduling & control system.
27. Timeline 2012 2013 MSSS LBA MSSS HBA Regular LOFAR operations LO FA R 1.0 release Call for proposals LO FA R 2.0 freeze LO FA R 2.0 release A A RTFA AC correlator LO FA R 3.0 release System design Initial “trap” U pgraded “trap” Connect to A A RTFA AC pipeline M ining lightcurve archive BlueG ene/P replacem ent approximate dates LO FA R sending VO Events
28. Summary • The trap & associated infrastructure are shaping well up for MSSS. • Still a lot of work to make them suitable for real-time operations. • To capitalize on the upcoming LOFAR 2.0 (and beyond), we need to be prepared provide input now.