Sharded Light-curve Database

Sharded Light-curve Database

Bart Scheers
LOFAR Transients Key Project Meeting, Meudon, December 2011

Ab44292d7d6f032baf342a98230a6654?s=128

transientskp

June 23, 2012
Tweet

Transcript

  1. Sharded Light-curve Database Bart Scheers Astronomical Institute Anton Pannekoek, University

    of Amsterdam Centre for Mathematics & Informatics (CWI) LOFAR TKP Meeting, Meudon, 2011–12–14
  2. 1 LOFAR TKP Meeting – 2011-12-14 Bart Scheers LOFAR Catalogue

    (of Light Curves) ▸ List of all sources detected at least once by LOFAR ▸ Multiple observations per source ▹ Light curves ▹ Adding time domain → dynamic catalogue ▸ Keep track of 'meta-data' ▹ image properties (noise) ▹ observation characteristics ▸ Make available for data mining/discovery ▹ Scalable
  3. What do we expect? ▸ Full operation: 50 – 100

    TB/yr ▸ Peaks: 10,000 sources per second ▸ Distinct sources: ~107 – 108 ▹ which are revisted many, many, many times ▸ These numbers call for bulk-processing ▹ maintain statistical representations of data ▹ spread data over multiple nodes 2 LOFAR TKP Meeting – 2011-12-14 Bart Scheers
  4. Source Association 3 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

  5. Source Association 3 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

    Dynamic (updated after every image) Static (but updated after every db instantiation)
  6. TKP Data(base) flow 4 LOFAR TKP Meeting – 2011-12-14 Bart

    Scheers
  7. TKP Data(base) flow 4 LOFAR TKP Meeting – 2011-12-14 Bart

    Scheers Light-curve Database, Long-term Archive, 50 – 100 TB/yr TRAP Database, during observations, ≲ 500 GB
  8. 5 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

  9. Source Association 6 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

    (1) distance on sky (2) dimensionless distance (3) likelyhood ratio
  10. Source Association 6 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

    (1) distance on sky (2) dimensionless distance (3) likelyhood ratio
  11. Source Variability 7 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

    (1) absolute flux change (2) weighted flux change Maintain 6 properties
  12. Association & Variability Probabilities ▸ Rayleigh distribution ▸ η ν

    behaves as chi square probability 8 LOFAR TKP Meeting – 2011-12-14 Bart Scheers
  13. WSRT Data 9 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

  14. WSRT Data 9 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

  15. WSRT Data 9 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

  16. 10 LOFAR TKP Meeting – 2011-12-14 Bart Scheers Global Sky

    Model
  17. 11 LOFAR TKP Meeting – 2011-12-14 Bart Scheers Global Sky

    Model ▸ Get all VLSS sources within the field of view ▸ Find (none or) counterpart in WENSS and NVSS catalogues ▸ Fit spectral index, curvature and higher order curvature order terms ▸ Create source-list file ▸ Wanted: No VLSS in FoV, use WENSS as base
  18. 12 LOFAR TKP Meeting – 2011-12-14 Bart Scheers Global Sky

    Model gsm.expected_fluxes_in_fov(conn, ra_c, decl_c, fov_radius, assoc_theta, 'bbs.skymodel.test', storespectraplots=True)
  19. 13 LOFAR TKP Meeting – 2011-12-14 Bart Scheers

  20. We want to mine more... ▸ Detecting trends ▹ n

    sequential data points mσ above average ▸ Systematic structure of light curve ▹ Ratio of the mean square successive difference to the sample variance ▸ FTs, cross- & auto-correlations; all work with (varying) window sizes → SciQL 14 LOFAR TKP Meeting – 2011-12-14 Bart Scheers
  21. SciQL – Cross-Correlation Example ▸ Extend SQL2003 → SciQL 15

    LOFAR TKP Meeting – 2011-12-14 Bart Scheers
  22. SciLens Platform (original) 16 LOFAR TKP Meeting – 2011-12-14 Bart

    Scheers ▸ Computational top tier (1 node) ▸ High-end tier (16 nodes) ▸ Cloud-oriented tier (64 nodes) ▸ Energy-conservative tier (256 nodes)
  23. SciLens Platform (current) 17 LOFAR TKP Meeting – 2011-12-14 Bart

    Scheers ▸ Top and high-end tier not built yet ▸ Cloud-oriented tier ▹ 144 Rocks, single quad cores, 16 GB, 0.5 or 1 TB SSD, 1 × 2 TB, Infiniband (40Gb/s) ▸ Bottom tier ▹ 144 Pebbles AMD- Bobcat, 8GB, 5 × 2TB, ethernet (1Gb/s)
  24. SciLens Platform (current) 17 LOFAR TKP Meeting – 2011-12-14 Bart

    Scheers ▸ Top and high-end tier not built yet ▸ Cloud-oriented tier ▹ 144 Rocks, single quad cores, 16 GB, 0.5 or 1 TB SSD, 1 × 2 TB, Infiniband (40Gb/s) ▸ Bottom tier ▹ 144 Pebbles AMD- Bobcat, 8GB, 5 × 2TB, ethernet (1Gb/s)
  25. SciLens Platform (current) 17 LOFAR TKP Meeting – 2011-12-14 Bart

    Scheers ▸ Top and high-end tier not built yet ▸ Cloud-oriented tier ▹ 144 Rocks, single quad cores, 16 GB, 0.5 or 1 TB SSD, 1 × 2 TB, Infiniband (40Gb/s) ▸ Bottom tier ▹ 144 Pebbles AMD- Bobcat, 8GB, 5 × 2TB, ethernet (1Gb/s)
  26. SciLens Platform (current) 17 LOFAR TKP Meeting – 2011-12-14 Bart

    Scheers ▸ Top and high-end tier not built yet ▸ Cloud-oriented tier ▹ 144 Rocks, single quad cores, 16 GB, 0.5 or 1 TB SSD, 1 × 2 TB, Infiniband (40Gb/s) ▸ Bottom tier ▹ 144 Pebbles AMD- Bobcat, 8GB, 5 × 2TB, ethernet (1Gb/s)
  27. 18 LOFAR TKP Meeting – 2011-12-14 Bart Scheers Conclusions ▸

    Statistical representation of full LOFAR catalogue relaxes source association ▸ Sharded database reduces replication ▸ Together with SciLens the infrastructure is scalable ▸ SciQL extends data mining opertunities