Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Trap: Now & Next

Ab44292d7d6f032baf342a98230a6654?s=47 transientskp
January 09, 2014

The Trap: Now & Next

John Swinbank



January 09, 2014


  1. TKP Project Meeting, Amsterdam / 2014-01-09 John Swinbank j.swinbank@uva.nl The

    Trap: Now and Next
  2. The LOFAR Transients Pipeline Real-time Database Imaging Pipeline Image Cube

    Quality Control Lightcurve Storage Transient & Variability Analysis Source Association Archive Database Classification & Analysis Response Scheduling Send External Alert Re-run Image Analysis Schedule New Observation Other Facilities Receive External Alert Real-time Processing Off-line & External Systems On-line Processing Visibility Data Scheduler Source Finding
  3. For each tim estep Perform quality control checks Blind source

    extraction Store im age properties & sky region in database Store rejection reason in database Q C Failure G o to next im age Q C Success Im age stack m ust have a dataset ID associated w ith it: either the database w ill generate one, or the user specifies a pre-existing ID . Source m easurem ent quality control For each m easurem ent Store m easurem ent, but do not use Q C Failure G o to next m easurem ent Store Q C results in database End for each m easurem ent Perform in-im age source association G et all im ages for tim estep For each im age Tim esteps m ust be processed in order N B All parsets corresponding to a given pipeline run m ust be stored in the database. W e suggestion creation of a new table ("pipelinerun"), w hich stores parsets and other configuration inform ation. Each dataset m ay consist of one or m ore pipeline runs. Each im age is uniquely associated w ith a pipeline run. W e store everything, but from here on w e are only using m easurem ents w hich assed the Q C step. Im age Stack R etrieve expected running catalogue sources Formal Specification and Design The Master Plan The basic functionality we are
 iterating towards in the future
  4. Release Planning Even numbers 㱺 Science functionality
 Odd numbers 㱺

    Technical capabilities •Release 0: May 2013 •In support of Cycle 0 data processing •10 new features; 49 bugs fixed •Release 1: November 2013 •14 new features; 78 bugs fixed •Release 2: “Early” 2014 •16 new features & 21 bugs currently targeted •Aiming for higher release cadence
  5. Release 1 Highlights •Provide the same science capability as Cycle

    0: •Persistence; QC; Source Finding; Null Detections; Source Association; Transient Detection. •No automatic classification or response. •Transition to a new pipeline architecture (based on Celery). •Multi-frequency source association. •Support wider range of image formats. •Revised & updated documentation. •Provide proper positional uncertainties. •More robust against database failures. •Many, many bug fixes.
  6. Banana http://banana.transientskp.org/1/

  7. Documentation http://docs.transientskp.org/

  8. Standalone Sourcefinder: PySE

  9. Coming in Release 2 (Or: the most egregious issues with

    the current release) •Handle “single-epoch” transients… …without an avalanche of false detections. •Simpler/faster/better handling of “null detections”. •Rework “monitoring list” for correct behaviour. •Dynamic (“query time”) definition of a transient. •Proper storage of meta- and logging data. •Developer documentation. •Open source!
  10. The (near) future •Science: •Association with catalogue sources. •Improved (per-image)

    quality control. •Technical: •Database interface cleanup (SQLAlchemy, etc?). •Sourcefinder run-time performance improvements. •Logging system improvements. •Robustness.
  11. The (further) future •Science: •Improved transient detection metrics. •Source classification.

    •Automatic response. •Archival database hosting & data mining interface. •Improved (per-source) quality control. •False Detection Rate algorithm support. •Full Stokes. •Image differencing. •Technical: •Streaming mode. •Further performance improvements.
  12. Development Process Focus on Quality Planning Testing Reviewing Documenting Communicating

  13. Issue Tracking: Redmine https://support.astron.nl/lofar_issuetracker/

  14. Communication: HipChat

  15. Source Control & Code Review: GitHub http://github.com/transientskp/tkp/

  16. Continuous Integration: Jenkins http://jenkins.transientskp.org/

  17. Conclusions •Trap Release 1 is now available: use it for

    science today. •Trap Release 2 is coming “soon”: it will be even more awesome! •We have lots of plans for the future… •…but appreciate your input. •…and your manpower! •Tightened development procedures with a focus on delivering quality, bug-free, documented software.