Automating AMI: Working towards intelligent fast-response transient followup

Tim Staley
LOFAR Transients Key Project Meeting, Meudon, December 2011

Transcript

1. Automating Ami Prototyping rapid transient followup Tim Staley 4 Pi Sky (University of Southampton) LOFAR TKP Meeting, Meudon December 2011
2. Outline 1 LOFAR as a virtual observatory 2 VOEvents — recap and current status 3 How to write a VOEvent trigger system in 2 weeks? 4 Future work Tim Staley (Southampton) Automating Ami TKP Dec 2011 2 / 26
3. Outline 1 LOFAR as a virtual observatory 2 VOEvents — recap and current status 3 How to write a VOEvent trigger system in 2 weeks? 4 Future work Tim Staley (Southampton) Automating Ami TKP Dec 2011 3 / 26
4. LOFAR as a VO — First detections LOFAR could see exotic, rare events e.g. Lorimer burst? Integration of a LOFAR “stream” into the VOEvents network is a priority if we want rapid followup Key issue is deciding what is worthy of triggering an alert Tim Staley (Southampton) Automating Ami TKP Dec 2011 4 / 26
5. LOFAR as a VO — Follow up Often, transient events are observed with poor localization (e.g. Fermi, Swift) LOFAR’s huge field of view is very helpful here Transient alert systems are getting faster all the time Raises the possibility of cross-facility observations on short timescales LOFAR’s sub-second "digital slew" makes it very suited to this sort of ultra-rapid followup Perhaps we will see new phenomena in familiar transients at these timescales? Tim Staley (Southampton) Automating Ami TKP Dec 2011 5 / 26
6. Outline 1 LOFAR as a virtual observatory 2 VOEvents — recap and current status 3 How to write a VOEvent trigger system in 2 weeks? 4 Future work Tim Staley (Southampton) Automating Ami TKP Dec 2011 6 / 26
7. The VOEvents project Defines a standard alert format Easily machine readable (contrast with Astronomical Telegrams) Brings together software efforts Tim Staley (Southampton) Automating Ami TKP Dec 2011 7 / 26
8. VOEvent jargon — A bluffer’s guide VOEvent , VOEvent packet A single packet of information transmitted in one go Must haves: IVORN, Who, What, How, etc. Optional: Tables, hyperlinked images, . . . Formatted as XML Tim Staley (Southampton) Automating Ami TKP Dec 2011 8 / 26
9. Tim Staley (Southampton) Automating Ami TKP Dec 2011 9 / 26
10. VOEvent jargon — A bluffer’s guide Brokers: Nodes on the network Subscribers: “Leaf nodes” i.e. end users Stream: A series of VOEvents originating from a particular facility Portfolio: A group of VOEvents known to relate to the same astronomical source Tim Staley (Southampton) Automating Ami TKP Dec 2011 10 / 26
11. VOEvent resources Tim Staley (Southampton) Automating Ami TKP Dec 2011 11 / 26
12. Outline 1 LOFAR as a virtual observatory 2 VOEvents — recap and current status 3 How to write a VOEvent trigger system in 2 weeks? 4 Future work Tim Staley (Southampton) Automating Ami TKP Dec 2011 12 / 26
13. The Arcminute Microkelvin Imager
14. The AMI response system Automated email system Credit goes to MRAO — David Titterington, Keith Grainge et al Tim Staley (Southampton) Automating Ami TKP Dec 2011 14 / 26
15. The AMI response system import smtplib to = ’mkyong2002@yahoo.com’ gmail_user = ’mkyong2002@gmail.com’ gmail_pwd = ’yourpassword’ smtpserver = smtplib.SMTP("smtp.gmail.com",587) smtpserver.ehlo() smtpserver.starttls() smtpserver.ehlo smtpserver.login(gmail_user, gmail_pwd) header = ’To:’ + to + ’\n’ + ’From: ’ + gmail_use print header msg = header + ’\n this is test msg from mkyong.c smtpserver.sendmail(gmail_user, to, msg) print ’done!’ smtpserver.close() Tim Staley (Southampton) Automating Ami TKP Dec 2011 15 / 26
16. Pulling in VOEvents Skyalert ATOM feed, + python ATOM reader library Slow? Tim Staley (Southampton) Automating Ami TKP Dec 2011 16 / 26
17. Pulling in VOEvents Skyalert ATOM feed, + python ATOM reader library Slow? Skyalert Jabber feed Undocumented (broken?) Tim Staley (Southampton) Automating Ami TKP Dec 2011 16 / 26
18. Pulling in VOEvents Skyalert ATOM feed, + python ATOM reader library Slow? Skyalert Jabber feed Undocumented (broken?) DC3.com VOEvent broker Fast Working Dakotatools software (Bob Denny), or ‘Twisted’ python library applet (John Swinbank) But: broker is privately run Tim Staley (Southampton) Automating Ami TKP Dec 2011 16 / 26
19. Processing VOEvents — Building Blocks Dakotatools ‘broker’ as the feed Plus the VOEventLib python library for parsing Mostly just works Tim Staley (Southampton) Automating Ami TKP Dec 2011 17 / 26
20. Processing VOEvents — New code Very simple ‘Portfolio’ class List of VOEvent packet references Standard fields as direct members (Who / When / IVORN) Dictionary of key / value pairs for everything else Tim Staley (Southampton) Automating Ami TKP Dec 2011 18 / 26
21. Processing VOEvents — New code Now simple to implement evaluation criteria Reevaluate a portfolio whenever created / updated Just a simple decision tree for now. Implemented in a separate module (easy to edit) Tim Staley (Southampton) Automating Ami TKP Dec 2011 19 / 26
22. Processing VOEvents — New code Skyalert archive scraping tools Tim Staley (Southampton) Automating Ami TKP Dec 2011 20 / 26
23. Outline 1 LOFAR as a virtual observatory 2 VOEvents — recap and current status 3 How to write a VOEvent trigger system in 2 weeks? 4 Future work Tim Staley (Southampton) Automating Ami TKP Dec 2011 21 / 26
24. Future work (1: Testing) Explore further VOEvent streams Further testing with AMI Tim Staley (Southampton) Automating Ami TKP Dec 2011 22 / 26
25. Future work (2: ???) Learn from optical transient research programs Palomar Transients Factory (Ashish Mahabal) Exeter eStar project (Alasdair Allan) Gaia science alerts (Lukasz Wyrzykowski) Many different approaches ‘Agents’ — multiple decision trees working together (eStar) ‘Expert’ modules — Bayesian, neural nets, GPR (PTF) Tim Staley (Southampton) Automating Ami TKP Dec 2011 23 / 26
26. Aside: Implementing intelligence Different methods depending upon the strength of your convictions Decision trees: Deterministic Bayesian methods: Designed probabilistic Neural networks: “Blackbox” probabilistic Self-organizing maps: Investigation (no priors) Tim Staley (Southampton) Automating Ami TKP Dec 2011 24 / 26
27. Future work (3: Profit!) Integration with LOFAR Followup Policies to decide allocation of ToO time, etc. Implementation may be significantly harder than for AMI? First detections Intrinsically linked to the Trap Real time pipeline please! Training data Tim Staley (Southampton) Automating Ami TKP Dec 2011 25 / 26
28. Summary AMI response system very nearly live It’s fairly easy to get stuck in with VOEvents The hard part is machine intelligence Tim Staley (Southampton) Automating Ami TKP Dec 2011 26 / 26