PyConZA 2012: "Control and Monitoring of the Karoo Telescope Arrays using Python" by Neilen Marias and Charles de Villiers

MeerKAT is a 64-dish array radio telescope that is currently under development. When MeerKAT is built in the Karoo it will be a world-class instrument, designed to do groundbreaking science. MeerKAT is scheduled to enter full operation in 2016 and it will be the largest and most sensitive radio telescope in the southern hemisphere until the completion of the Square Kilometre Array (SKA) around 2024. Via MeerKAT, South Africa is playing a key role in design and technology developments in the SKA. KAT-7 is an engineering prototype for the development of MeerKAT. This 7-dish array has already been built in the Karoo and it is currently being commissioned and operated 24x7, producing real scientific results. This talk is about the current Control and Monitoring (CAM) sub-system of KAT-7 and our plans to expand it for MeerKAT. The CAM is a distributed system, running on Linux, and integrates the KAT-7 equipment such as antennas, receivers and ancillary devices, monitoring hardware and software for health, status and alarms, and controlling them as required to perform observations and to provide a human interface for operations. Most of the KAT-7 CAM software is developed in Python.

* Overview (SKA vs MeerKAT vs KAT-7)
* Description of the problem domain (what we monitor & control, data volumes)
* Architecture of the CAM solution (KATcp, proxies, monitors)
* Overview of the iPython user interface
* Overview of the KAT GUI
* Real radio images produced by KAT-7
* Live demo (via webcam) of remote control of the KAT-7 dishes using the GUI

Transcript

1. Control and Monitoring of the Karoo Telescope Arrays using Python Neilen Marais Charles de Villiers Senior Software Engineers nmarais@ska.ac.za charles@ska.ac.za
2. Why Radio Telescopes? See a different part of the spectrum 2
3. What is a Radio Telescope? 3
4. What is a Radio Telescope? 4
5. What is a Radio Telescope? 5
6. Control and Monitoring (CAM) Telescope is a distributed system CAM is the central nervous system Control: Make it move Monitoring: Sensory feedback Designed for remote operation 6
7. What are we dealing with now? KAT-7 (Karoo Array Telescope 7) 7
8. What are we working towards? MeerKAT
9. KAT-7 Receptor Detail
10. KAT-7 Receptor Detail  Receiver Horn Antenna  Stirling Cryo cooler with Ion Pump  RF Low Noise Amplifier (LNA)  RF Noise diode coupler  RF Amplifier
11. KAT-7 Receptor Detail  Antenna positioner control unit  RF amplifier/attenuator  RF to optical transducer  Pedestal chiller  Building Management  Weather station
12. KAT-7 Array
13. KAT-7 Compute Container
14. KAT-7 Compute Container Detail  Antenna positioner control unit  RF amplifier/attenuator  RF to optical transducer  Pedestal chiller  Optical to RF transducers  RF Down-conversion and conditioning  FPGA based Digital Back End  Data capture server  Time/Frequency reference  CAM Servers  BMS  Chiller
15. (Meer)KAT CAM Principles Everything speaks the same language (KATCP) Self describing, i.e. introspectable Soft Real-time Proxy layer between devices and users 15
16. KAT Control Protocol (KATCP) Ethernet as fieldbus Simple text-line oriented protocol over TCP sockets Request/Reply for RPC Device introspection part of the standard Clients Subscribe to sensors 16
17. Simple KATCP Device <Source> <Demo> 17
18. KATCP Proxies Shields devices from users Implement higher level functionality Aggregate multiple devices Use ?help and ?sensor-list to introspect devices 18
19. Some CAM design principles Device control through Proxy layer Low-level / command line control through libraries & scripting interface Remote operations 19
20. Why monitor? Monitoring Provides essential context for the observation data Locates immediate problems Indicates trends in the equipment and the environment Assists with fault-finding Detects dangerous conditions and raises alarms 20
21. What do we monitor?  KAT-7 monitoring (sensors only, excluding logs and alarms)  2 CAM servers with 10-20 processes each  ± 260 sensors per antenna (x 7 antennas)  ± 4500 total sensors  ± 650 samples per second  ± 450MB per day (compressed)  MeerKAT monitoring (estimates)  6 CAM servers with 10-30 processes each  ± 500 sensors per antenna (x 64 antennas)  ± 100 000 total sensors  ± 12000 samples per second  ± 10 GB per day (compressed) 21
22. Stormy weather? 22
23. Monitoring the telescopes Sensors Basic data-gatherers Managed by a proxy Follow the Observer pattern Because Sensors are software-defined, we can have aggregate sensors, averaging sensors, etc 23
24. How many sensors do we need? Currently (KAT-7) we have about 4 500 sensors This will scale > linearly with the number of receptors, so we can expect to have (at least) 100 000 sensors for MeerKAT (64 dishes) And SKA will have about 3000 dishes... 24
25. How much data does that produce? Data arriving via KATcp is raw text. We need to store all that data indefinitely, but we can compress it. KAT-7: about 3.4Gb/day (uncompressed), 500 Mb/day (compressed) MeerKAT : about 10 times that (say 5Gb/day, compressed) Relational databases don't work well with this We currently use a hybrid system involving compressed files and an index stored in a relational database But we are thinking about other approaches... 25
26. Managing the telescope 26
27. Managing the telescope (2) Enter the Observation Framework! Built around Schedule Blocks (SBs), of various types: oObservation (scripted in Python) oManual (interactively controlled via iPython) oMaintenance (just reserve the hardware)
28. Managing the telescope (3) Observation Framework Observation Framework components KatCoreLib Controller/Resource Manager Scheduler Executor
29. • IPython demo Observation Framework
30. 30 30 Thank you! http://www.ska.ac.za Questions? nmarais@ska.ac.za charles@ska.ac.za
31. 1 Control and Monitoring of the Karoo Telescope Arrays using Python Neilen Marais Charles de Villiers Senior Software Engineers nmarais@ska.ac.za charles@ska.ac.za
32. Why Radio Telescopes? See a different part of the spectrum 2 • Optical image on the left, on the right a zoomed in copy with a false-colour radio image (Generated by VLA in the US) of the interstellar hydrogen gas overlayed. • For more details look up the excellent talk given by simon ratcliffe yesterday
33. What is a Radio Telescope? 3 An optical telescope looks like this Light is collected by a large mirror and focussed onto an optical sensor Like your digital camera, only better!
34. What is a Radio Telescope? 4 A minimal radio telescope looks like this Radio waves are collected by a large conductive parabolic reflector and are focussed onto a receiving antenna with a Low Noise Amplifier Like your DSTV dish, only better! A single dish is rather limited, almost like a single pixel camera
35. What is a Radio Telescope? 5 Use multiple dishes. A typical modern radio telescope looks like this Radio waves from multiple dishes are mathematically combined through correlation Allowing the radio-frequency sky to be imaged
36. Control and Monitoring (CAM) Telescope is a distributed system CAM is the central nervous system Control: Make it move Monitoring: Sensory feedback Designed for remote operation 6 A radio telescope consists of a large number of distributed independent devices CAM is the nervous system that pulls them together Control is getting them all to work together Monitoring is receiving sensory feedback: Identify unsafe or damaging situations, and take action as needed Monitor parameters that influence data quality Store telemetry information (e.g. antenna positions, etc) needed to process the scientific data and for debugging Remote operations – network transparency. Keep RFI polluting humans far away
37. What are we dealing with now? KAT-7 (Karoo Array Telescope 7) 7 KAT-7 is a seven dish interferometer array Conventional prime-focus design, unusual Stirling cryo- cooler and composite construction. Construction started in 2008 Currently installed in the Karoo and in daily use Primarily a risk-reduction engineering prototype for MeerKAT – Hardware equivalent of release early and often Also used for science
38. What are we working towards? MeerKAT MeerKAT (more KAT) will be a 64 dish interferometer array Unusual Offset Gregorian antenna design, standard GM cryo-cooler and metal construction. Multi-band receiver carousel, allows the instrument to work over a much wider range of frequencies Will be the largest radio telescope in the southern hemisphere upon completion. Preliminary construction underway as we speak MeerKAT expected to grow into SKA phase one, ~300 dishes
39. KAT-7 Receptor Detail 9 Let's look at a single KAT-7 receptor. A receptor is the combination of a reflector, receiving antenna, physical motion control and a whole bunch of RF components needed to receive a signal
40. KAT-7 Receptor Detail 10  Receiver Horn Antenna  Stirling Cryo cooler with Ion Pump  RF Low Noise Amplifier (LNA)  RF Noise diode coupler  RF Amplifier At the focus: cryo cooler with ion pump <- NOT FROM STAR ion pump <- NOT FROM STAR TREK TREK RF LNA, RF Noise Diode RF LNA, RF Noise Diode
41. KAT-7 Receptor Detail 11  Antenna positioner control unit  RF amplifier/attenuator  RF to optical transducer  Pedestal chiller  Building Management  Weather station In the pedestal: BMS: door open, fire alarm. Door open important for RFI Actually only one weather station on a pole near the receptors
42. 12 KAT-7 Array Seven receptors, so Seven times everything you saw before. And yes, you've seen this picture before
43. KAT-7 Compute Container 13 Signals are all fed via optical fibre to the Compute Signals are all fed via optical fibre to the Compute container container Provides an RFI shielded enclosure with 'air lock' doors Provides an RFI shielded enclosure with 'air lock' doors to prevent RFI from computers and other equipment to prevent RFI from computers and other equipment from contaminating measurements from contaminating measurements
44. KAT-7 Compute Container Detail 14  Antenna positioner control unit  RF amplifier/attenuator  RF to optical transducer  Pedestal chiller  Optical to RF transducers  RF Down-conversion and conditioning  FPGA based Digital Back End  Data capture server  Time/Frequency reference  CAM Servers  BMS  Chiller Just talk about the things one by one Time: Accurate time is absolutely critical to the operation of a radio telescope. The GPS disciplined atomic clock is possibly my favourite piece of equipment in KAT-7. It is the basis of our NTP server's timing, and is used by this signal generator to create the 800 MHz sampling clock for the ADCs
45. (Meer)KAT CAM Principles Everything speaks the same language (KATCP) Self describing, i.e. introspectable Soft Real-time Proxy layer between devices and users 15 For time critical commands we use soft realtime: Hard realtime operation is left to simple 'low-level' devices. Soft realtime involves CAM sending those devices a future-dated timestamp with every time-critical command. The commands are applied by the hard-realtime systems at the right time. This simplifies CAM development considerably since only small (usually outsourced) parts of the system have to deal with the difficulty of implementing hard real-time software.
46. KAT Control Protocol (KATCP) Ethernet as fieldbus Simple text-line oriented protocol over TCP sockets Request/Reply for RPC Device introspection part of the standard Clients Subscribe to sensors 16 • Karoo. Arr... T C P at heart of the telescope control system • We specify KATCP interfaces for all devices. Not possible? Add them ourselves. – allows us to use standard mechanisms to CAM everything, e.g. Ganglia, server running out of disk space can use the same mechanism as the 'antenna on fire' error to send an SMS. • Home grown – experimented with SNMP before. • KATCP concepts for the basis of higher level functionality
47. Simple KATCP Device <Source> <Demo> 17 • telnet localhost 8000 • ?help divide • ?sensor-list • ?sensor-sampling no-divides event • ?divide 1 2 • ?divide 2 3 • •
48. KATCP Proxies Shields devices from users Implement higher level functionality Aggregate multiple devices Use ?help and ?sensor-list to introspect devices 18 • Shielding: Keep load off sensitive devices • Many allow only one concurrent connection • Hide/wrap certain device functionalities. E.g. some DBE has 16 inputs, we only use 14. Sensors relating to the other two are hidden by the proxy to prevent false alarms • Higher level: E.g. antenna device only knows how to steer to azimuth-elevation coordinates relative to the antenna position. • Proxy knows know to continuously track a named target, sends position updates to the antenna every 50ms • Antenna proxy (should really be called Receptor proxy) connects to all the devices in the receptor and consolidates them as one larger device.
49. Some CAM design principles Device control through Proxy layer Low-level / command line control through libraries & scripting interface Remote operations 19 Our proxy layer protects the hardware and adds a layer of fucntionality. The iPython shell allows us to control and monitor interactively, permitting exploratory development The system was designed from the start for remote operations – allows better infrastructure & support
50. Why monitor? Monitoring Provides essential context for the observation data Locates immediate problems Indicates trends in the equipment and the environment Assists with fault-finding Detects dangerous conditions and raises alarms 20 The radio-telescope collects RF signal data. But that data is meaningless without its context. Signals from multiple dishes must be combined and correlated, so we need to know exactly where they were pointing, and when. Many corrections must be applied – geometric (based on antenna layout) load (based on antenna orientation and wind), temperature,... Raw data is augmented with some of this context, and the science data processing depends on it. We also need to be aware of equipment failures and trends, weather patterns. etc For instance – if the wind speed exceeds a certain threshold, the antennas must be stowed (describe) If pedestal or container doors are left open, the operators must be notified If design-basis temperatures are exceeded, equipment must be shut down If RFE power levels are too high or too low, signal quality is degraded etc...
51. 21 What do we monitor?  KAT-7 monitoring (sensors only, excluding logs and alarms)  2 CAM servers with 10-20 processes each  ± 260 sensors per antenna (x 7 antennas)  ± 4500 total sensors  ± 650 samples per second  ± 450MB per day (compressed)  MeerKAT monitoring (estimates)  6 CAM servers with 10-30 processes each  ± 500 sensors per antenna (x 64 antennas)  ± 100 000 total sensors  ± 12000 samples per second  ± 10 GB per day (compressed) 21 KAT-7 ± 300 sensors sampled at ~ ms rate ± 100 sensors sampled at ~ second rate Rest sampled with default rate of 10s or on event (depending on sensor type) but can be configured from ms up to minutes  MeerKAT •± 2500 sensors sampled at ~ ms rate •± 100 sensors sampled at ~ second rate •Rest sampled with default rate of 10s or on event •
52. 22 Stormy weather? 22 In view of these evolving requirements, we need an archiving design that is scalable and efficient.
53. Monitoring the telescopes Sensors Basic data-gatherers Managed by a proxy Follow the Observer pattern Because Sensors are software-defined, we can have aggregate sensors, averaging sensors, etc 23 Monitoring data is gathered by sensors. A sensor is a Python object managed by a proxy, that has its own data type and strategy. It is not meant to be polled – it implements the Observer pattern so that clients can 'register an interest'. Sensors communicate via KATcp. - The type may be float (eg temperature, azimuth, elevation), integer (process ids, memory usage) discrete (system and device states, logging level), boolean (ok flags) or string (URLs, script arguments). Because a sensor is a software object, we can also define aggregate and derived sensors, averaging sensors, trend sensors... We have a LOT of sensors in KAT7, and we're going to need a lot more for MeerKAT!
54. How many sensors do we need? Currently (KAT-7) we have about 4 500 sensors This will scale > linearly with the number of receptors, so we can expect to have (at least) 100 000 sensors for MeerKAT (64 dishes) And SKA will have about 3000 dishes... 24 Kat-7 has about 8000 sensors, with new ones being invented from time to time. MeerKAT will have at least ten times this number. Managing all this could be a huge configuration management problem. Fortunately the proxies and sensors are self-describing, and so the system can configure itself dynamically – just tell it where to find the proxies, and the proxies will announce their available commands and sensors. Python introspection to the rescue!
55. How much data does that produce? Data arriving via KATcp is raw text. We need to store all that data indefinitely, but we can compress it. KAT-7: about 3.4Gb/day (uncompressed), 500 Mb/day (compressed) MeerKAT : about 10 times that (say 5Gb/day, compressed) Relational databases don't work well with this We currently use a hybrid system involving compressed files and an index stored in a relational database But we are thinking about other approaches... 25 We are dealing with large, but not 'astronomical' quantities of data. At the moment we store this in compressed format, along with an indexing system in a regular relational DB. Clients (augment task, iPython users. Sensorgraph users, interested statisticians) can query the historical data, filtering it by sensor names and times. Query performance is OK for now, but we would like it to be better. Both the data storage requirements and the query bandwidth requirements can only increase from here. We are looking at other storage solutions, including PyTables (HDF5 format)
56. 26 Managing the telescope 26 KAT-7 is already producing science data MeerKAT will be in heavy demand, 24/7/365.25... With 64 dishes, several observations/ activities may be running simultaneously We need a framework that controls access and allows concurrent activities while avoiding resource conflicts
57. Managing the telescope (2) Enter the Observation Framework! Built around Schedule Blocks (SBs), of various types: o Observation (scripted in Python) o Manual (interactively controlled via iPython) o Maintenance (just reserve the hardware) 27 •Astronomers are accustomed to running Python scripts that just grab resources and perform actions •SBs regulate this by specifying in advance the resource and time requirements of an observation •They will also feature in the planned Proposal Management and Observation Planning tools for MeerKAT •Sbs define a unit of work for the telescope.
58. Managing the telescope (3) Observation Framework 28 Observation Framework components KatCoreLib Controller/Resource Manager Scheduler Executor •KatCoreLib •Contains the routines allowing one to build a KAT host •high-level container for proxies. Etc •We'll build one of these in a minute •Controller/Resource Manager •Starts/stops the proxies •Validates schedule blocks •Allocates resources on request from the Scheduler •Scheduler •Manages a queue of schedule blocks •Submits them to the Executor •Executor •Runs observation scripts and collects output
59. • IPython demo Observation Framework 29 configure_cam('Fri Oct 5 2012','Fri Oct f cabc') : target = kat.sources['PKS 1610-60'] : kat.ant1.req.target(target.description) : kat.ant1.req.mode("POINT") : kat.ant1.wait("lock",True,300) kat.ant1.req.mode(“STOW”) : kat.disconnect()
60. 30 30 Thank you! http://www.ska.ac.za Questions? nmarais@ska.ac.za charles@ska.ac.za