The Trap: Now & Next

John Swinbank

Transcript

1. TKP Project Meeting, Amsterdam / 2014-01-09
   John Swinbank j.swinbank@uva.nl
   The Trap:
   Now and Next
   

   View Slide

2. The LOFAR Transients Pipeline
   Real-time Database
   Imaging
   Pipeline
   Image
   Cube
   Quality
   Control
   Lightcurve
   Storage
   Transient &
   Variability
   Analysis
   Source
   Association
   Archive
   Database
   Classification
   & Analysis
   Response
   Scheduling
   Send
   External
   Alert
   Re-run
   Image
   Analysis
   Schedule
   New
   Observation
   Other
   Facilities
   Receive
   External
   Alert
   Real-time Processing
   Off-line & External Systems
   On-line
   Processing
   Visibility
   Data
   Scheduler
   Source
   Finding
   

   View Slide

3. For each
   tim
   estep
   Perform
   quality
   control checks
   Blind
   source
   extraction
   Store
   im
   age
   properties
   &
   sky
   region
   in
   database
   Store
   rejection
   reason
   in
   database
   Q
   C
   Failure
   G
   o
   to
   next
   im
   age
   Q
   C
   Success
   Im
   age
   stack
   m
   ust have
   a
   dataset ID
   associated
   w
   ith
   it: either the
   database
   w
   ill generate
   one, or the
   user
   specifies
   a
   pre-existing
   ID
   .
   Source
   m
   easurem
   ent
   quality
   control
   For each
   m
   easurem
   ent
   Store
   m
   easurem
   ent,
   but do
   not use
   Q
   C
   Failure
   G
   o
   to
   next
   m
   easurem
   ent
   Store
   Q
   C
   results
   in
   database
   End
   for each
   m
   easurem
   ent
   Perform
   in-im
   age
   source
   association
   G
   et all im
   ages
   for
   tim
   estep
   For each
   im
   age
   Tim
   esteps
   m
   ust be
   processed
   in
   order
   N
   B
   All parsets
   corresponding
   to
   a
   given
   pipeline
   run
   m
   ust be
   stored
   in
   the
   database.
   W
   e
   suggestion
   creation
   of a
   new
   table
   ("pipelinerun"), w
   hich
   stores
   parsets
   and
   other configuration
   inform
   ation. Each
   dataset m
   ay
   consist of one
   or m
   ore
   pipeline
   runs. Each
   im
   age
   is
   uniquely
   associated
   w
   ith
   a
   pipeline
   run.
   W
   e
   store
   everything, but
   from
   here
   on
   w
   e
   are
   only
   using
   m
   easurem
   ents
   w
   hich
   assed
   the
   Q
   C
   step.
   Im
   age
   Stack
   R
   etrieve
   expected
   running
   catalogue
   sources
   Formal Specification and Design
   The Master Plan
   The basic functionality we are

   iterating towards in the future
   

   View Slide

4. Release Planning
   Even numbers 㱺 Science functionality

   Odd numbers 㱺 Technical capabilities
   •Release 0: May 2013
   •In support of Cycle 0 data processing
   •10 new features; 49 bugs fixed
   •Release 1: November 2013
   •14 new features; 78 bugs fixed
   •Release 2: “Early” 2014
   •16 new features & 21 bugs currently targeted
   •Aiming for higher release cadence
   

   View Slide

5. Release 1 Highlights
   •Provide the same science capability as Cycle 0:
   •Persistence; QC; Source Finding; Null Detections; Source
   Association; Transient Detection.
   •No automatic classification or response.
   •Transition to a new pipeline architecture (based on Celery).
   •Multi-frequency source association.
   •Support wider range of image formats.
   •Revised & updated documentation.
   •Provide proper positional uncertainties.
   •More robust against database failures.
   •Many, many bug fixes.
   

   View Slide

6. Banana
   http://banana.transientskp.org/1/
   

   View Slide

7. Documentation
   http://docs.transientskp.org/
   

   View Slide

8. Standalone Sourcefinder: PySE
   

   View Slide

9. Coming in Release 2
   (Or: the most egregious issues with the current release)
   •Handle “single-epoch” transients…
   …without an avalanche of false detections.
   •Simpler/faster/better handling of “null detections”.
   •Rework “monitoring list” for correct behaviour.
   •Dynamic (“query time”) definition of a transient.
   •Proper storage of meta- and logging data.
   •Developer documentation.
   •Open source!
   

   View Slide

10. The (near) future
    •Science:
    •Association with catalogue sources.
    •Improved (per-image) quality control.
    •Technical:
    •Database interface cleanup (SQLAlchemy, etc?).
    •Sourcefinder run-time performance improvements.
    •Logging system improvements.
    •Robustness.
    

    View Slide

11. The (further) future
    •Science:
    •Improved transient detection metrics.
    •Source classification.
    •Automatic response.
    •Archival database hosting & data mining interface.
    •Improved (per-source) quality control.
    •False Detection Rate algorithm support.
    •Full Stokes.
    •Image differencing.
    •Technical:
    •Streaming mode.
    •Further performance improvements.
    

    View Slide

12. Development Process
    Focus on
    Quality
    Planning
    Testing
    Reviewing
    Documenting
    Communicating
    

    View Slide

13. Issue Tracking: Redmine
    https://support.astron.nl/lofar_issuetracker/
    

    View Slide

14. Communication: HipChat
    

    View Slide

15. Source Control & Code Review: GitHub
    http://github.com/transientskp/tkp/
    

    View Slide

16. Continuous Integration: Jenkins
    http://jenkins.transientskp.org/
    

    View Slide

17. Conclusions
    •Trap Release 1 is now available: use it for science today.
    •Trap Release 2 is coming “soon”: it will be even more
    awesome!
    •We have lots of plans for the future…
    •…but appreciate your input.
    •…and your manpower!
    •Tightened development procedures with a focus on
    delivering quality, bug-free, documented software.
    

    View Slide