"Dolosse: Distributed Physics Data Acquisition System" by Bertram Losper & Sehlabaka Qhobosheane

In this talk we will introduce key concepts and techniques of triggering and nuclear physics data acquisition. We will then discuss the development of a new distributed data acquisition software system developed for the subatomic and materials research experimental facilities at iThemba LABS, using python and open source streaming tools.

The discussion will show how we use Apache kafka to modernise our DAQ systems. Apache Kafka is a distributed messaging system which we use to build a DAQ system that has a multi-producer, multi-consumer model. This discussion will focus on how we use kafka to configure experimental runs, monitor and control the run and the data acquisition of real-time physics events.

We ingest time-series scientific data from dedicated spectroscopy hardware modules, filtering to determine if we have event coincidence between these different modules, build an interesting physics event from this data and exposing it for easy analysis and visualisation through the python ecosystem. By using kafka the need for heavy processing servers/PC’s are eliminated.

iThemba Laboratory for Accelerator-Based Sciences (LABS) is a national infrastructure platform of the National Research Foundation (NRF) specialising in particle accelerator-based sciences and engineering, with offices in Cape Town and Johannesburg. It has a number of particle accelerators used for basic research in material sciences, subatomic physics as well as production of radio-pharmaceuticals for local and international markets

Transcript

1. Dolosse: A Distributed Physics Data Acquisition System
   Bertram Losper & Sehlabaka Qhobosheane
   PyConZA2020
   

   View Slide

2. Outline
   ● NRF/iThemba LABS
   ○ who we are
   ○ what we do
   ● Trigger & Data Acquisition (TDAQ) overview
   ○ Trigger management
   ○ Data readout
   ○ Event building and data storage
   ○ System control and monitoring
   ● Dolosse DAQ
   ○ Requirements
   ○ Design
   ○ Implementation
   

   View Slide

3. NRF/iThemba LABS; Who we are
   CPT
   JHB
   

   View Slide

4. NRF/iThemba LABS; What we do
   ● Multidisciplinary Research Facility of the NRF
   ○ Nuclear Physics
   ○ Materials Research
   ○ Production of Radiopharmaceuticals (local & international markets)
   ○ Training & Development of students
   ● > 4 Accelerators
   

   View Slide

5. K8 Injector Cyclotron 1
   K8 Injector Cyclotron 2 K200 Separated Sector Cyclotron
   (SSC)
   3MV EN Tandetron K11 Cyclotron - FDG PET Isotopes
   6MV EN Tandem AMS
   

   View Slide

6. K600 FEE
   K600 Beamline
   

   View Slide

7. Outline
   ● NRF/iThemba LABS
   ○ who we are
   ○ what we do
   ● Trigger & Data Acquisition (TDAQ) overview
   ○ Trigger management
   ○ Data readout
   ○ Event building and data storage
   ○ System control and monitoring
   ● Dolosse DAQ
   ○ Requirements
   ○ Design
   ○ Implementation
   

   View Slide

8. ● Overall the main role of a DAQ is to process the signals generated in
   a detector and saving the interesting information on some permanent
   storage.
   Nuclear Physics DAQ
   

   View Slide

9. 9
   May 2019 | iTL-SAAO Meeting
   ● Trigger:
   ○ Either selects interesting events or rejects boring ones, in real
   time
   ○ i.e. with minimal controlled latency
   ■ time it takes to form and distribute its decision
   ● DAQ:
   ○ Gathers data produced by detectors: Readout
   ○ Forms complete events:
   Data Collection and Event Building
   ○ Stores event data: Data Logging
   ○ Provides Run Control, Configuration, Monitoring
   deterministic
   Trigger & DAQ (TDAQ)
   

   View Slide

10. Trigger, DAQ & Controls
    Dracarys!
    Monitoring, Run Control & Configuration
    Detector Channels
    Trigger
    Front End Electronics
    Readout Network / Event Building
    Storage
    Processing/
    Filtering
    DAQ
    

    View Slide

11. Trigger management
    ● Determining when data is available..
    ○ Interrupt
    ■ An interrupt is sent by a hardware device
    ■ The interrupt is
    ● Transformed into a software signal
    ● Caught by a data acquisition program
    ○ Undetermined latency is a potential problem!
    ○ Data readout starts
    ○ Polling
    ■ Some register in a module is continuously read out
    ■ Data readout happens when register “signals” new data
    

    View Slide

12. Data readout
    Physical
    system/quantity
    Signal
    conditioning
    Analog/Digital
    Converter
    Computer
    transducer Field wiring
    ● Data digitization (analog-to-digital conversion) by Frontend Electronics
    e.g. VME modules (ADC,TDC, QDC etc)
    ● Trigger signal received by a trigger module
    ○ I/O register or interrupt generator
    ● Data read-out by a Single Board Computer (SBC)
    ○ Either events are buffered or de-randomized in the FEE
    ■ Performance is usually improved by DMA readout
    ● Readout topology: bus or network
    

    View Slide

13. Event building and data storage
    ● Evt framing, buffering and data transmission:
    ○ Why buffering?
    ■ Triggers are uncorrelated
    ■ Create internal de-randomizers (to make system more
    deterministic)
    ○ Minimize deadtime thus improve efficiency
    ○ Optimize the usage of output channels
    ■ Disk
    ■ Network
    

    View Slide

14. Event building and data storage
    ● Necessary to define an event format
    ○ Identify every chunk of data, w/ a source id
    ■ Both during data taking and offline
    ○ To correctly read back data from files
    ■ i.e. the application should be able to identify each full event
    structure and navigate among all its fragments
    ○ Must be easily extendable
    ■ e.g.: adding sub-detectors
    ○ Keep track of the event format version number
    ○ NB: event format is the core of your experiment
    ■ Used both online and offline
    ●
    

    View Slide

15. System control and monitoring
    ● Two main aspects to system monitoring:
    ○ System operational monitoring
    ■ Sharing variables through the system
    ○ Data monitoring
    ■ Sampling data for monitoring processes
    ■ Sharing histogram through the system
    ■ Histogram browsing
    

    View Slide

16. System control and monitoring
    ● System control
    ○ Each DAQ component must have
    ■ A set of well defined states
    ■ A set of rules to pass from one state to another
    => Finite State Machine
    ○ A central process controls the system
    ■ Run control
    ● Implements the state machine
    ● Triggers state changes and takes track of
    components’ states
    ○ A GUI interfaces the user to the Run control
    ■ …and various system services…
    ●
    

    View Slide

17. Outline
    ● NRF/iThemba LABS
    ○ who we are
    ○ what we do
    ● Trigger & Data Acquisition (TDAQ) overview
    ○ Trigger management
    ○ Data readout
    ○ Event building and data storage
    ○ System control and monitoring
    ● Dolosse DAQ
    ○ Requirements
    ○ Design
    ○ Implementation
    

    View Slide

18. DAQ Requirements
    ● Transmitting data from the Detector (interesting event) to Storage
    ● The DAQ should be configurable i.e the configuration of Front-end
    electronics, trigger processing and run control
    ● Monitor data flow
    ● Monitor hardware ( front-end Electronics)
    ● Feedback messages from different processes/modules to inform the user
    ● Experiments should be able to and can run for days/weeks/months.
    

    View Slide

19. Dolosse DAQ
    Collaboration between Project Science and iThemba LABS
    Modernizing physics data processing
    • Extensible, scalable, robust DAQ that integrates multiple systems
    • Data storage to replace binary formats (raw format)
    • Data analysis that doesn’t limit concurrent operations
    • What existing frameworks to allow for our workflow?
    

    View Slide

20. Main Components of Dolosse
    DAQ Visualization
    Data Analysis
    

    View Slide

21. Apache Kafka
    Kafka is a messaging framework that allows us to
    manage communication between all of our different data
    sources. It allows for a multi-producer, multi-consumer
    model. We can collect and analyze in real-time.
    ● Fault-tolerant, replicated data streams
    ● Input : 1 M msg / sec | Output: 2 M msg / sec
    ● Huge amount of community support
    ● Could be used as data storage
    

    View Slide

22. Producers
    Using Kafka as a central messaging system,
    we combine previously distinct data sources.
    We can aggregate from any number of
    auxiliary systems with the same timestamp.
    Horizontally scales our workload, don’t need
    beefy DAQ machines.
    

    View Slide

23. Consumers
    We again scale horizontally. No
    monster analysis machines (though
    they help).
    Can provide visualization and
    monitoring via industry-standard
    applications (Plotly).
    Researchers take their data home,
    Parquet /Avro files offer ideal
    storage.
    

    View Slide

24. Putting it all together
    

    View Slide

25. Data Flow
    

    View Slide

26. Event Builder
    Data source 1
    (Producer)
    Data source 2
    Data source 3
    kafka
    Raw binary
    [0..255] 0..255...
    

    View Slide

27. Event Builder
    Data source 2
    Data source 3
    kafka
    Collated Event
    Builder
    consumer/Producer
    Raw binary
    [0..255] 0..255...
    The event-builder follows the algorithm: t1 =
    F1(s1), t2 =(t1, F2(s2..sX)), t3 = (t2,
    F3(sX+1..sN)
    for i in c:
    msg = i.poll( 0.1)
    Data source 1
    (Producer)
    Data source 1
    (Producer)
    

    View Slide

28. Event Builder
    Data source 1
    (Producer)
    Data source
    2
    Data source
    3
    kafka
    Collated Event
    Builder
    consumer/Producer
    Raw binary
    [0..255] 0..255...
    data=list(unpack('I' * (len(raw_dat) // 4), raw_dat))
    event_data = [
    hex(x) for x in data]
    evt_data[evt_name].append(
    {'name':msg.topic(),'data': event_data})
    Data source 2
    Data source 3
    Data source 1
    (Producer)
    

    View Slide

29. Event Builder
    Data source 1
    (Producer)
    Data source 2
    Data source 3
    kafka
    Collated Event
    Builder
    consumer/Producer
    Raw binary
    Human readable event
    [0..255] 0..255...
    The event-builder follows the algorithm: t1 =
    F1(s1), t2 =(t1, F2(s2..sX)), t3 = (t2,
    F3(sX+1..sN)
    

    View Slide

30. Visualization
    Visualization
    Plotly | VueJs
    The framework we use for visualization is Plotly.js
    within Vue.js or ROOT. Most visualization
    frameworks are really going to be application
    dependent.
    Plotly:
    ● It is a powerful graphing library
    ● Plotly creates interactive plots for example:
    Histograms, Heatmap, etc.
    Plotly can be used in Jupyter
    

    View Slide

31. Examples
    

    View Slide

32. Potential Pitfalls
    Event builder would have event “misalignment”(readout misalignment), if the datasource started producing data before event
    builder was started (discovered during testing).
    {
    'category': 'measurement',
    'technique': 'k600',
    'run number': 0, 'Event number': 1456,
    'Event': [{'name': 'adc_0', 'data': ['0xfa002000', '0xf8004002', ','0xfc0003b1']}, {'name': 'adc_0', 'data': ['0xfa002000', '0xf8004002', '0xfc0003b1']}, {{'name': 'adc_0', 'data':
    ['0xfa002000', '0xf8004002', '0xfc0003b1']}]
    }
    This was due to the nature of how partitions are fetched in kafka. The issue was resolved by creating a consumer object for each
    topic (Event Fragment) to be read out and reading out each fragment sequentially.
    {
    'category': 'measurement',
    'technique': 'k600',
    'run number': 0, 'Event number': 1456, 'Event': [{'name': 'adc_0', 'data': ['0xfa002000', '0xf8004002', '0xf8104073', '0xf8014076', '0xf811404b', '0xf8024045', '0xf812405f',
    '0xf803404f','0xfc0003b1']}, {'name': 'tdc_0', 'data': ['0x4000763f', ' '0x4000763f'}, {'name': 'qdc_0', 'data': ['0xfa001000', '0xf800425f', '0xf810406f', '0xf80242b5',
    '0xf8124084', '0xf8044275', '0xf814406f', '0xf8064259', '0xf8164077', '0xf8084063', '0xf818406e', '0xf80a4067', '0xf81a408a', '0xf80c407a', '0xf81c4083', '0xf80e406e',
    '0xf81e408f', '0xfc0003b2']}]
    }
    

    View Slide

33. Development Environment(s)
    ● Internal Fork
    ○ GitLab
    ■ CI/CD
    ● Public Repo
    ○ GitHub
    ■ Travis
    ○ ZenHub
    

    View Slide

34. Concluding Remarks
    ● Next milestone is to release the k600 Subsystem module - 2 weeks
    ● Release other subsystem (9 total) modules - every 1-2 months
    ● Biggest Challenges/Risks:
    ○ Limited developer time due to juggling between other projects &
    supporting experiments
    ● Links:
    ○ https://tlabs.ac.za
    ○ https://gitlab.tlabs.ac.za/init/dolosse
    ○ https://github.com/dolosse/dolosse
    

    View Slide

35. Baie Dankie.
    Rea leboha.
    Enkosi.
    Thank you.
    blosper@tlabs.ac.za qhobosheanesb@tlabs.ac.za
    

    View Slide