Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Trap: Now & Next

January 09, 2014

The Trap: Now & Next

John Swinbank


January 09, 2014

More Decks by transientskp

Other Decks in Science


  1. The LOFAR Transients Pipeline Real-time Database Imaging Pipeline Image Cube

    Quality Control Lightcurve Storage Transient & Variability Analysis Source Association Archive Database Classification & Analysis Response Scheduling Send External Alert Re-run Image Analysis Schedule New Observation Other Facilities Receive External Alert Real-time Processing Off-line & External Systems On-line Processing Visibility Data Scheduler Source Finding
  2. For each tim estep Perform quality control checks Blind source

    extraction Store im age properties & sky region in database Store rejection reason in database Q C Failure G o to next im age Q C Success Im age stack m ust have a dataset ID associated w ith it: either the database w ill generate one, or the user specifies a pre-existing ID . Source m easurem ent quality control For each m easurem ent Store m easurem ent, but do not use Q C Failure G o to next m easurem ent Store Q C results in database End for each m easurem ent Perform in-im age source association G et all im ages for tim estep For each im age Tim esteps m ust be processed in order N B All parsets corresponding to a given pipeline run m ust be stored in the database. W e suggestion creation of a new table ("pipelinerun"), w hich stores parsets and other configuration inform ation. Each dataset m ay consist of one or m ore pipeline runs. Each im age is uniquely associated w ith a pipeline run. W e store everything, but from here on w e are only using m easurem ents w hich assed the Q C step. Im age Stack R etrieve expected running catalogue sources Formal Specification and Design The Master Plan The basic functionality we are
 iterating towards in the future
  3. Release Planning Even numbers 㱺 Science functionality
 Odd numbers 㱺

    Technical capabilities •Release 0: May 2013 •In support of Cycle 0 data processing •10 new features; 49 bugs fixed •Release 1: November 2013 •14 new features; 78 bugs fixed •Release 2: “Early” 2014 •16 new features & 21 bugs currently targeted •Aiming for higher release cadence
  4. Release 1 Highlights •Provide the same science capability as Cycle

    0: •Persistence; QC; Source Finding; Null Detections; Source Association; Transient Detection. •No automatic classification or response. •Transition to a new pipeline architecture (based on Celery). •Multi-frequency source association. •Support wider range of image formats. •Revised & updated documentation. •Provide proper positional uncertainties. •More robust against database failures. •Many, many bug fixes.
  5. Coming in Release 2 (Or: the most egregious issues with

    the current release) •Handle “single-epoch” transients… …without an avalanche of false detections. •Simpler/faster/better handling of “null detections”. •Rework “monitoring list” for correct behaviour. •Dynamic (“query time”) definition of a transient. •Proper storage of meta- and logging data. •Developer documentation. •Open source!
  6. The (near) future •Science: •Association with catalogue sources. •Improved (per-image)

    quality control. •Technical: •Database interface cleanup (SQLAlchemy, etc?). •Sourcefinder run-time performance improvements. •Logging system improvements. •Robustness.
  7. The (further) future •Science: •Improved transient detection metrics. •Source classification.

    •Automatic response. •Archival database hosting & data mining interface. •Improved (per-source) quality control. •False Detection Rate algorithm support. •Full Stokes. •Image differencing. •Technical: •Streaming mode. •Further performance improvements.
  8. Conclusions •Trap Release 1 is now available: use it for

    science today. •Trap Release 2 is coming “soon”: it will be even more awesome! •We have lots of plans for the future… •…but appreciate your input. •…and your manpower! •Tightened development procedures with a focus on delivering quality, bug-free, documented software.