Towards a Dynamic LOFAR Catalogue

Bart Scheers
LOFAR TKP Meeting, Amsterdam, June 2011

Transcript

1. Towards a Dynamic LOFAR Catalogue Bart Scheers Astronomical Institute Anton Pannekoek, University of Amsterdam Centre for Mathematics & Informatics (CWI) LOFAR TKP Meeting, API, 2011–06–29
2. 1 LOFAR TKP Meeting – 2011-06-29 LOFAR Catalogue ▸ List of all sources detected by LOFAR ▸ Multiple observations per source ▹ Adding time domain → dynamic ▹ Light-curves ▸ Keep track of 'meta-data' ▹ image properties ▹ observation characteristics ▸ Make available for data mining/discovery ▸ Scalable
3. Data Stream of Images 2 LOFAR TKP Meeting – 2011-06-29
4. What can we expect? 3 LOFAR TKP Meeting – 2011-06-29 Scheers, PhD Thesis, 2011
5. What can we expect? ▸ Full operation: 50 TB/yr ▸ Peaks: 10,000 sources per second ▸ Distinct sources: ~107 ▸ Source is about 300 Bytes in DB 3 LOFAR TKP Meeting – 2011-06-29
6. How fast will it grow? 4 LOFAR TKP Meeting – 2011-06-29 Full-NL Zenith Monitor
7. Inside the Database ▸ Appending all detected sources ▸ Associate detected sources ▹ running catalogue ◦ i.e. LOFAR sources ▹ external catalogues ◦ VLSS, WENSS, NVSS, exoplanets, ... ▸ Run variability checks ▸ Move observations to light-curve database 5 LOFAR TKP Meeting – 2011-06-29
8. Inside the Database 5 LOFAR TKP Meeting – 2011-06-29
9. Source Association 6 LOFAR TKP Meeting – 2011-06-29
10. Source Association 7 LOFAR TKP Meeting – 2011-06-29 (1) distance on sky (2) dimensionless distance (3) likelyhood ratio
11. Association Parameter, r 8 LOFAR TKP Meeting – 2011-06-29
12. Source Variability 9 LOFAR TKP Meeting – 2011-06-29 (1) absolute flux change (2) weighted flux change Maintain 6 parameters
13. Association & Variability Probabilities ▸ Rayleigh distribution ▸ η ν behaves as chi square probability 10 LOFAR TKP Meeting – 2011-06-29
14. Simulated Data 11 LOFAR TKP Meeting – 2011-06-29 Non-variable sources
15. WSRT GRB030329 Observations 12 LOFAR TKP Meeting – 2011-06-29
16. WSRT GRB030329 Observations 12 LOFAR TKP Meeting – 2011-06-29
17. More Light-curve Analysis ▸ Detecting trends ▹ n sequential data points mσ above average ▸ Systematic structure of light curve ▹ Ratio of the mean square successive difference to the sample variance ▸ Array Query Language: SciQL 13 LOFAR TKP Meeting – 2011-06-29
18. Data mining Light-curve Database ▸ Extending SQL → SciQL 14 LOFAR TKP Meeting – 2011-06-29
19. SciLens Experimentation Platform 15 LOFAR TKP Meeting – 2011-06-29 ▸ Computational top tier (1 node) ▸ High-end tier (16 nodes) ▸ Cloud-oriented tier (64 nodes) ▸ Energy-conservative tier (256 nodes)
20. Distributed Light-curve Database 16 LOFAR TKP Meeting – 2011-06-29