Update on the status of the transients detection pipeline

Transcript

1. An update on the transients detection and classification pipeline Evert

   Rol University of Amsterdam

2. TRAP: an end to end pipeline • Relatively simple, but

   we can go from raw data to classified transients • For practical purposes, it also incorporates the imaging steps (flagging, calibration, imaging), but does not have to. • So far, only Martin Bell and I have run it... • Documentation online: http://docs.transientskp.org/trap

3. Pipeline • flagging • calibration • imaging • source extraction

   • database storage • source association • transient detection • feature extraction • classification Uses the default LOFAR pipeline framework Steps: Imaging pipeline TKP Python library

4. TKP library • General code clean up • Improved independence

   between subpackages (eg, the sourcefinder can now more easily be run standalone) • Updated and extended (unit) tests • Nightly build on local (A'dam) machines, CEP1 and CEP2 (includes automated tests) • Updated and extended documentation: http:// docs.transientskp.org/tkp

5. From images to transients pipeline Severe disconnect in the form

   of lacking metadata: • Images produced by the imager come with only very, very minimal header information. Missing are eg observation times. source extraction images imager imaging pipeline transients pipeline

6. From images to transients pipeline Severe disconnect in the form

   of lacking metadata: • Images produced by the imager come with only very, very minimal header information. Missing are eg observation times. Hack: source extraction images imager MS set FITS update header find original data imaging pipeline transients pipeline

7. From images to transients pipeline Severe disconnect in the form

   of lacking metadata: • Images produced by the imager come with only very, very minimal header information. Missing are eg observation times. Looking forward to HDF5! source extraction images imager imaging pipeline transients pipeline

8. Source extraction • Has not changed over the past time

   • Being looked at improvements • Does rely on a decent image quality from the imaging pipeline

9. Database storage, source association and transient detection • All this

   happens inside the database itself (SQL). See also Bart Scheers' talk • No matching with existing catalogues • No cross-frequency matching • Transient detection may fail when transient goes below background

10. Database storage, source association and transient • Transient detection may

    fail when transient goes below background • As recently tested by Martin Bell 0 5 10 0 0.5 1 1.5 Time Flux Real Lightcurve Detection level (5m) Current Lightcurve 0 5 10 0 0.5 1 1.5 Time Flux Real Lightcurve Detection level (5m) Current Lightcurve 0 5 10 0 0.5 1 1.5 Time Flux Real Lightcurve Detection level (5m) Current Lightcurve 0 5 10 0 0.5 1 1.5 Time Flux Real Lightcurve Detection level (5m) Current Lightcurve

11. Classification • Feature extraction • Default statistics • No periodicity

    search yet • Needs more: what to look for? • Classification • Manual decision tree exists. Lack of decisions • Clustering algorithms being implemented. Lack of training sample • Tests with existing data sets (optical)

12. Classification

13. Classification

14. Future • Near: • Documentation update • Source association and

    transient detection improvement • Classification training set, and decision tree update • Far: • Different classification algorithms • Use of multi-frequency data (association, classification) • Speed tests • Response to and from other event sources

15. Classification

16. Manual decision tree implementation class SlowTransient(ClassifiedTransient): """Slow transient""" def test_duration(self):

    if self.duration > 1e6: return 0.9 def test_variability(self): if self.variability > 1e4: return 0.9 class MainBranch(Branch): """Starting point for manual classification tree""" def test1(self): return [SubBranch1] def test2(self): return [SubBranch2] class SubBranch1(Branch): def test1(self): return [SlowTransient]