Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Update on the status of the transients detection pipeline

Update on the status of the transients detection pipeline

Evert Rol
LOFAR TKP Meeting, Amsterdam, June 2011

transientskp

June 17, 2012
Tweet

More Decks by transientskp

Other Decks in Science

Transcript

  1. TRAP: an end to end pipeline • Relatively simple, but

    we can go from raw data to classified transients • For practical purposes, it also incorporates the imaging steps (flagging, calibration, imaging), but does not have to. • So far, only Martin Bell and I have run it... • Documentation online: http://docs.transientskp.org/trap
  2. Pipeline • flagging • calibration • imaging • source extraction

    • database storage • source association • transient detection • feature extraction • classification Uses the default LOFAR pipeline framework Steps: Imaging pipeline TKP Python library
  3. TKP library • General code clean up • Improved independence

    between subpackages (eg, the sourcefinder can now more easily be run standalone) • Updated and extended (unit) tests • Nightly build on local (A'dam) machines, CEP1 and CEP2 (includes automated tests) • Updated and extended documentation: http:// docs.transientskp.org/tkp
  4. From images to transients pipeline Severe disconnect in the form

    of lacking metadata: • Images produced by the imager come with only very, very minimal header information. Missing are eg observation times. source extraction images imager imaging pipeline transients pipeline
  5. From images to transients pipeline Severe disconnect in the form

    of lacking metadata: • Images produced by the imager come with only very, very minimal header information. Missing are eg observation times. Hack: source extraction images imager MS set FITS update header find original data imaging pipeline transients pipeline
  6. From images to transients pipeline Severe disconnect in the form

    of lacking metadata: • Images produced by the imager come with only very, very minimal header information. Missing are eg observation times. Looking forward to HDF5! source extraction images imager imaging pipeline transients pipeline
  7. Source extraction • Has not changed over the past time

    • Being looked at improvements • Does rely on a decent image quality from the imaging pipeline
  8. Database storage, source association and transient detection • All this

    happens inside the database itself (SQL). See also Bart Scheers' talk • No matching with existing catalogues • No cross-frequency matching • Transient detection may fail when transient goes below background
  9. Database storage, source association and transient • Transient detection may

    fail when transient goes below background • As recently tested by Martin Bell 0 5 10 0 0.5 1 1.5 Time Flux Real Lightcurve Detection level (5m) Current Lightcurve 0 5 10 0 0.5 1 1.5 Time Flux Real Lightcurve Detection level (5m) Current Lightcurve 0 5 10 0 0.5 1 1.5 Time Flux Real Lightcurve Detection level (5m) Current Lightcurve 0 5 10 0 0.5 1 1.5 Time Flux Real Lightcurve Detection level (5m) Current Lightcurve
  10. Classification • Feature extraction • Default statistics • No periodicity

    search yet • Needs more: what to look for? • Classification • Manual decision tree exists. Lack of decisions • Clustering algorithms being implemented. Lack of training sample • Tests with existing data sets (optical)
  11. Future • Near: • Documentation update • Source association and

    transient detection improvement • Classification training set, and decision tree update • Far: • Different classification algorithms • Use of multi-frequency data (association, classification) • Speed tests • Response to and from other event sources
  12. Manual decision tree implementation class SlowTransient(ClassifiedTransient): """Slow transient""" def test_duration(self):

    if self.duration > 1e6: return 0.9 def test_variability(self): if self.variability > 1e4: return 0.9 class MainBranch(Branch): """Starting point for manual classification tree""" def test1(self): return [SubBranch1] def test2(self): return [SubBranch2] class SubBranch1(Branch): def test1(self): return [SlowTransient]