The Trap: Now & Next

John Swinbank

Transcript

1. TKP Project Meeting, Amsterdam / 2014-01-09 John Swinbank j.swinbank@uva.nl The Trap: Now and Next
2. The LOFAR Transients Pipeline Real-time Database Imaging Pipeline Image Cube Quality Control Lightcurve Storage Transient & Variability Analysis Source Association Archive Database Classification & Analysis Response Scheduling Send External Alert Re-run Image Analysis Schedule New Observation Other Facilities Receive External Alert Real-time Processing Off-line & External Systems On-line Processing Visibility Data Scheduler Source Finding
3. For each tim estep Perform quality control checks Blind source extraction Store im age properties & sky region in database Store rejection reason in database Q C Failure G o to next im age Q C Success Im age stack m ust have a dataset ID associated w ith it: either the database w ill generate one, or the user specifies a pre-existing ID . Source m easurem ent quality control For each m easurem ent Store m easurem ent, but do not use Q C Failure G o to next m easurem ent Store Q C results in database End for each m easurem ent Perform in-im age source association G et all im ages for tim estep For each im age Tim esteps m ust be processed in order N B All parsets corresponding to a given pipeline run m ust be stored in the database. W e suggestion creation of a new table ("pipelinerun"), w hich stores parsets and other configuration inform ation. Each dataset m ay consist of one or m ore pipeline runs. Each im age is uniquely associated w ith a pipeline run. W e store everything, but from here on w e are only using m easurem ents w hich assed the Q C step. Im age Stack R etrieve expected running catalogue sources Formal Specification and Design The Master Plan The basic functionality we are
 iterating towards in the future
4. Release Planning Even numbers 㱺 Science functionality
 Odd numbers 㱺 Technical capabilities •Release 0: May 2013 •In support of Cycle 0 data processing •10 new features; 49 bugs fixed •Release 1: November 2013 •14 new features; 78 bugs fixed •Release 2: “Early” 2014 •16 new features & 21 bugs currently targeted •Aiming for higher release cadence
5. Release 1 Highlights •Provide the same science capability as Cycle 0: •Persistence; QC; Source Finding; Null Detections; Source Association; Transient Detection. •No automatic classification or response. •Transition to a new pipeline architecture (based on Celery). •Multi-frequency source association. •Support wider range of image formats. •Revised & updated documentation. •Provide proper positional uncertainties. •More robust against database failures. •Many, many bug fixes.
6. Banana http://banana.transientskp.org/1/
7. Documentation http://docs.transientskp.org/
8. Standalone Sourcefinder: PySE
9. Coming in Release 2 (Or: the most egregious issues with the current release) •Handle “single-epoch” transients… …without an avalanche of false detections. •Simpler/faster/better handling of “null detections”. •Rework “monitoring list” for correct behaviour. •Dynamic (“query time”) definition of a transient. •Proper storage of meta- and logging data. •Developer documentation. •Open source!
10. The (near) future •Science: •Association with catalogue sources. •Improved (per-image) quality control. •Technical: •Database interface cleanup (SQLAlchemy, etc?). •Sourcefinder run-time performance improvements. •Logging system improvements. •Robustness.
11. The (further) future •Science: •Improved transient detection metrics. •Source classification. •Automatic response. •Archival database hosting & data mining interface. •Improved (per-source) quality control. •False Detection Rate algorithm support. •Full Stokes. •Image differencing. •Technical: •Streaming mode. •Further performance improvements.
12. Development Process Focus on Quality Planning Testing Reviewing Documenting Communicating
13. Issue Tracking: Redmine https://support.astron.nl/lofar_issuetracker/
14. Communication: HipChat
15. Source Control & Code Review: GitHub http://github.com/transientskp/tkp/
16. Continuous Integration: Jenkins http://jenkins.transientskp.org/
17. Conclusions •Trap Release 1 is now available: use it for science today. •Trap Release 2 is coming “soon”: it will be even more awesome! •We have lots of plans for the future… •…but appreciate your input. •…and your manpower! •Tightened development procedures with a focus on delivering quality, bug-free, documented software.