Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Trap: Now & Next

transientskp
January 09, 2014

The Trap: Now & Next

John Swinbank

transientskp

January 09, 2014
Tweet

More Decks by transientskp

Other Decks in Science

Transcript

  1. TKP Project Meeting, Amsterdam / 2014-01-09
    John Swinbank [email protected]
    The Trap:
    Now and Next

    View full-size slide

  2. The LOFAR Transients Pipeline
    Real-time Database
    Imaging
    Pipeline
    Image
    Cube
    Quality
    Control
    Lightcurve
    Storage
    Transient &
    Variability
    Analysis
    Source
    Association
    Archive
    Database
    Classification
    & Analysis
    Response
    Scheduling
    Send
    External
    Alert
    Re-run
    Image
    Analysis
    Schedule
    New
    Observation
    Other
    Facilities
    Receive
    External
    Alert
    Real-time Processing
    Off-line & External Systems
    On-line
    Processing
    Visibility
    Data
    Scheduler
    Source
    Finding

    View full-size slide

  3. For each
    tim
    estep
    Perform
    quality
    control checks
    Blind
    source
    extraction
    Store
    im
    age
    properties
    &
    sky
    region
    in
    database
    Store
    rejection
    reason
    in
    database
    Q
    C
    Failure
    G
    o
    to
    next
    im
    age
    Q
    C
    Success
    Im
    age
    stack
    m
    ust have
    a
    dataset ID
    associated
    w
    ith
    it: either the
    database
    w
    ill generate
    one, or the
    user
    specifies
    a
    pre-existing
    ID
    .
    Source
    m
    easurem
    ent
    quality
    control
    For each
    m
    easurem
    ent
    Store
    m
    easurem
    ent,
    but do
    not use
    Q
    C
    Failure
    G
    o
    to
    next
    m
    easurem
    ent
    Store
    Q
    C
    results
    in
    database
    End
    for each
    m
    easurem
    ent
    Perform
    in-im
    age
    source
    association
    G
    et all im
    ages
    for
    tim
    estep
    For each
    im
    age
    Tim
    esteps
    m
    ust be
    processed
    in
    order
    N
    B
    All parsets
    corresponding
    to
    a
    given
    pipeline
    run
    m
    ust be
    stored
    in
    the
    database.
    W
    e
    suggestion
    creation
    of a
    new
    table
    ("pipelinerun"), w
    hich
    stores
    parsets
    and
    other configuration
    inform
    ation. Each
    dataset m
    ay
    consist of one
    or m
    ore
    pipeline
    runs. Each
    im
    age
    is
    uniquely
    associated
    w
    ith
    a
    pipeline
    run.
    W
    e
    store
    everything, but
    from
    here
    on
    w
    e
    are
    only
    using
    m
    easurem
    ents
    w
    hich
    assed
    the
    Q
    C
    step.
    Im
    age
    Stack
    R
    etrieve
    expected
    running
    catalogue
    sources
    Formal Specification and Design
    The Master Plan
    The basic functionality we are

    iterating towards in the future

    View full-size slide

  4. Release Planning
    Even numbers 㱺 Science functionality

    Odd numbers 㱺 Technical capabilities
    •Release 0: May 2013
    •In support of Cycle 0 data processing
    •10 new features; 49 bugs fixed
    •Release 1: November 2013
    •14 new features; 78 bugs fixed
    •Release 2: “Early” 2014
    •16 new features & 21 bugs currently targeted
    •Aiming for higher release cadence

    View full-size slide

  5. Release 1 Highlights
    •Provide the same science capability as Cycle 0:
    •Persistence; QC; Source Finding; Null Detections; Source
    Association; Transient Detection.
    •No automatic classification or response.
    •Transition to a new pipeline architecture (based on Celery).
    •Multi-frequency source association.
    •Support wider range of image formats.
    •Revised & updated documentation.
    •Provide proper positional uncertainties.
    •More robust against database failures.
    •Many, many bug fixes.

    View full-size slide

  6. Banana
    http://banana.transientskp.org/1/

    View full-size slide

  7. Documentation
    http://docs.transientskp.org/

    View full-size slide

  8. Standalone Sourcefinder: PySE

    View full-size slide

  9. Coming in Release 2
    (Or: the most egregious issues with the current release)
    •Handle “single-epoch” transients…
    …without an avalanche of false detections.
    •Simpler/faster/better handling of “null detections”.
    •Rework “monitoring list” for correct behaviour.
    •Dynamic (“query time”) definition of a transient.
    •Proper storage of meta- and logging data.
    •Developer documentation.
    •Open source!

    View full-size slide

  10. The (near) future
    •Science:
    •Association with catalogue sources.
    •Improved (per-image) quality control.
    •Technical:
    •Database interface cleanup (SQLAlchemy, etc?).
    •Sourcefinder run-time performance improvements.
    •Logging system improvements.
    •Robustness.

    View full-size slide

  11. The (further) future
    •Science:
    •Improved transient detection metrics.
    •Source classification.
    •Automatic response.
    •Archival database hosting & data mining interface.
    •Improved (per-source) quality control.
    •False Detection Rate algorithm support.
    •Full Stokes.
    •Image differencing.
    •Technical:
    •Streaming mode.
    •Further performance improvements.

    View full-size slide

  12. Development Process
    Focus on
    Quality
    Planning
    Testing
    Reviewing
    Documenting
    Communicating

    View full-size slide

  13. Issue Tracking: Redmine
    https://support.astron.nl/lofar_issuetracker/

    View full-size slide

  14. Communication: HipChat

    View full-size slide

  15. Source Control & Code Review: GitHub
    http://github.com/transientskp/tkp/

    View full-size slide

  16. Continuous Integration: Jenkins
    http://jenkins.transientskp.org/

    View full-size slide

  17. Conclusions
    •Trap Release 1 is now available: use it for science today.
    •Trap Release 2 is coming “soon”: it will be even more
    awesome!
    •We have lots of plans for the future…
    •…but appreciate your input.
    •…and your manpower!
    •Tightened development procedures with a focus on
    delivering quality, bug-free, documented software.

    View full-size slide