BLA Measurement Improving on the art Take-away Spectrometry, Data Acquisition, and 2D Image Processing Bruce Ravel Synchrotron Methods Group Materials Measurement Laboratory National Institute of Standards and Technology ISS Weekly Meeting, 11 April, 2012 1 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Acknowledgements Jeremy Kropf, Argonne Jeremy developed and built the spectrometer and contributed to the image processing development Gordon Southam, Chris Omelan, Jeremiah Shuster, University of Western Ontario Sample prep and help running the experiment Matt Newville, University of Chicago Epics help and author of Epics Channel Access for Python Everything shown here happened at MRCAT, 10ID, APS. 2 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away A common XAS science problem ... Tracking the evolution of a chemical transformation is a common sort of problem for an XAS beamline. In this case, we see the reduction of a gold chloride solution to metallic gold over the course of many hours. Questions What is the rate constant? Is there an intermediate species? 3 / 20 Spectrometry, Data Acquisition, and 2D Image Processing M. Lengke et el., Mechanisms of Gold Bioaccumulation by Filamentous Cyanobacteria from Gold(III)-Chloride Complex, Environ. Sci. Technol. 40(20) p. 6304-6309. (2006), DOI: 10.1021/es061040r

BLA Measurement Improving on the art Take-away ... and its solution We solve this by testing linear combinations of measured standards against the data. The linear combination that best fits the data is assumed to be the constituents of the data. In this case, we can model the data as a combination of gold metal, gold chloride solution, and a species with a gold-sulphur bond. The problem Within the statistical limitations of the data, we could not distinguish between the various sulfur species. 4 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away XANES at higher energy resolution A likely solution to the sulphur species problem is high energy resolution fluorescence detection, or “HERFD”. (Yes, it really is called HERFD....) In HERFD, we use some kind of crystal spectrometer to measure energy resolution below the core-hole lifetime, which is the physical effect dominating the energy resolution in a conventional measurement. We hope that the enhancement of the features enables better distinction between similar chemical species. 5 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The bent Laue spectrometer The analyzer is a thin silicon wafer bent over an aluminum frame milled into an approximation of a log-spiral. This is the right shape for preserving the divergence of the beam while using diffraction from the crystal planes for wavelength discrimination. The Pilatus is used for energy resolution – photons with different wavelengths hit different places on the Pilatus. 6 / 20 Spectrometry, Data Acquisition, and 2D Image Processing The romesco and the plot are public domain images taken from the Wikipedia page “Logorithmic Spiral”. The shell: “This Wikipedia and Wikimedia Commons image is from the user Chris73 and is freely available at http://commons.wikimedia.org/wiki/... File:NautilusCutawayLogarithmicSpiral.jpg under the Creative Commons cc-by-sa 3.0 license.”

BLA Measurement Improving on the art Take-away Why bent Laue spectrometry? The main reason Jeremy has this spectrometer. Pros Fairly efficient for emission energies above 9 keV One wafer can cover a large energy range (by changing the angle) Good solution for high energy edges Very easy to align Can be multiplexed for better throughput Except for the Pilatus, fairly inexpensive Cons Energy resolution limited to about 1.5 eV to 2 eV, at best Not suitable for transition metals which have core-hole lifetimes smaller than the intrinsic resolution of the instrument Fairly small solid angle subtended per analyzer Each analyzer needs its own Pilatus 7 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The HERFD measurement Optics This is an XAS scan, so we moved the monochromator with coordinated motion of the undulator Focusing We used the functional equivalent of a KB mirror pair, which was a non-standard beamline component Motors Sample stage + three degrees of freedom for the spectrometer (linear rowards sample, crystal rotation, rotation of Pilatus arm) Sample Solution samples in plastic frames, enclosed with Kapton tape Scan file File containing columns of I0 , It , ROI from the SDD, some areal ROIs from camserver, & a few miscellaneous columns. Also a column of metadata! Images One image for each energy pointin the XANES scan + 10-15 images for energy calibration 8 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The HERFD measurement Optics This is an XAS scan, so we moved the monochromator with coordinated motion of the undulator Focusing We used the functional equivalent of a KB mirror pair, which was a non-standard beamline component Motors Sample stage + three degrees of freedom for the spectrometer (linear rowards sample, crystal rotation, rotation of Pilatus arm) Sample Solution samples in plastic frames, enclosed with Kapton tape Scan file File containing columns of I0 , It , ROI from the SDD, some areal ROIs from camserver, & a few miscellaneous columns. Also a column of metadata! Images One image for each energy pointin the XANES scan + 10-15 images for energy calibration Surprise! Severe radiation damage required moving the sample stage in ˆ z synchronously during image exposure 8 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away One energy point At each energy, the Pilatus sees something that looks like this: This image: 1 is a tiff file 2 has 487 × 195 pixels, each containing a signed 32 bit integer 3 has several bad pixels 4 is almost 400 kB 5 somehow represents µ(E) for that value of E 6 and is in need of further processing How do we turn that image into a data point? 9 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Energy calibration Take a sequence of snapshots with the incident energy in a range around the Au Kα1 peak (9713 eV). 9697 eV 9705 eV 9713 eV 9699 eV 9707 eV 9715 eV 9701 eV 9709 eV 9717 eV 9703 eV 9711 eV 9719 eV Energy disperses across the face of the Pilatus This allows us to assign particular emission energies to particular pixels. This gives us good energy resolution. 10 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Mask creation Here’s another view of the energy dispersion. ⇓ I developed a simple algorithm for removing bad pixels, scrubbing stray pixels, and filling in the gaps. This results in a simple “AND” mask. The scalar-multiplication of the mask and the image is summed over all pixels. This sum is the value of µ at that energy. So, = 1,292,800 counts (in 20 sec). 11 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Putting the pieces together A scan, then, consists of 1 A column data file with scalars and metadata 2 1 tiff image at each energy point – we were measuring about 100 energy points with 20 second integration and about 3 seconds of overhead per point, for about 40 minutes per scan. 3 12 tiff images at incident energies around the emission peak, again with 20 second integrations, for about 3 more minutes per sample. 4 In total, each measurement required 47 Mb (uncompressed – gzip compresses by 4.48 to 10.5 Mb) of diskspace and 45 minutes of time. 5 Various efficiencies in spectrometer design could multiply diskspace by N and divide time by N to 2N (N = number of analyzers). 12 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Data and metadata The column data file, Aufoil1.001 # HERFD scan on Aufoil1 # ---------------------------------- # energy time ring_current i0 it ifl ir roi1 roi2 roi3 roi4 tif 11850.000 20 98.6063252127 1387833 721744 466592 1420285 18 49 16 51 1 11860.000 20 98.5954416527 1372787 715385 473045 1404234 26 40 19 69 2 11870.000 20 98.5859850127 1370129 714683 480828 1402733 32 38 19 62 3 ... etc ... energy point tiff files Aufoil1 00001.tif through Aufoil1 00112.tif elastic tiff files Aufoil1 elastic 9697 00001.tif through Aufoil1 elastic 9719 00001.tif 13 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Data and metadata The column data file, Aufoil1.001 # HERFD scan on Aufoil1 # ---------------------------------- # energy time ring_current i0 it ifl ir roi1 roi2 roi3 roi4 tif 11850.000 20 98.6063252127 1387833 721744 466592 1420285 18 49 16 51 1 11860.000 20 98.5954416527 1372787 715385 473045 1404234 26 40 19 69 2 11870.000 20 98.5859850127 1370129 714683 480828 1402733 32 38 19 62 3 ... etc ... energy point tiff files Aufoil1 00001.tif through Aufoil1 00112.tif elastic tiff files Aufoil1 elastic 9697 00001.tif through Aufoil1 elastic 9719 00001.tif 1 All files have a common stub (Aufoil1) 2 The final column in the data file identifies the energy point tiff file (and is a number) 3 Processing the energy point tiff file requires a particular elastic tiff file 13 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Data and metadata The column data file, Aufoil1.001 # HERFD scan on Aufoil1 # ---------------------------------- # energy time ring_current i0 it ifl ir roi1 roi2 roi3 roi4 tif 11850.000 20 98.6063252127 1387833 721744 466592 1420285 18 49 16 51 1 11860.000 20 98.5954416527 1372787 715385 473045 1404234 26 40 19 69 2 11870.000 20 98.5859850127 1370129 714683 480828 1402733 32 38 19 62 3 ... etc ... energy point tiff files Aufoil1 00001.tif through Aufoil1 00112.tif elastic tiff files Aufoil1 elastic 9697 00001.tif through Aufoil1 elastic 9719 00001.tif 1 All files have a common stub (Aufoil1) 2 The final column in the data file identifies the energy point tiff file (and is a number) 3 Processing the energy point tiff file requires a particular elastic tiff file This is ridonkulously fragile! 13 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Post-processing the data Mask creation + processing 113 data points takes 45 seconds on my 4 year old Dell. In 3 days, we measured 31 possibly-useful samples. 14 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away In-line image processing Processing the whole measurement – 45 seconds bla herfd -e=9713 -c=config.ini Aufoil1 15 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away In-line image processing Processing the whole measurement – 45 seconds bla herfd -e=9713 -c=config.ini Aufoil1 Processing just the mask – 15 seconds bla mask -e=9713 -c=config.ini Aufoil1 Each elastic mask can be computed synchronously during the following exposure. 15 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away In-line image processing Processing the whole measurement – 45 seconds bla herfd -e=9713 -c=config.ini Aufoil1 Processing just the mask – 15 seconds bla mask -e=9713 -c=config.ini Aufoil1 Each elastic mask can be computed synchronously during the following exposure. Processing just one point, given a mask – 1 second bla point -e=9713 -c=config.ini -p=17 Aufoil1 Each data point is computed fast enough for in-scan display. 15 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away In-line image processing Processing the whole measurement – 45 seconds bla herfd -e=9713 -c=config.ini Aufoil1 Processing just the mask – 15 seconds bla mask -e=9713 -c=config.ini Aufoil1 Each elastic mask can be computed synchronously during the following exposure. Processing just one point, given a mask – 1 second bla point -e=9713 -c=config.ini -p=17 Aufoil1 Each data point is computed fast enough for in-scan display. Post-processing is only necessary if the user wants to experiment with the mask creation algorithm. 15 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The wonder of MX MRCAT uses a little-used control system called MX. MX runs most, but not quite all, of the beamline. 16 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The wonder of MX 16 / 20 Spectrometry, Data Acquisition, and 2D Image Processing It has ... ummm ... quirks and some missing features. For example, no one present knew how to make MX run a motor synchronously quite in the way we needed.

BLA Measurement Improving on the art Take-away The wonder of MX The fundamental problem with MX is the same as with spec and some other control solutions. It’s quirky and suffers from fundamentally poor language design. It’s not open source. It’s incomplete and adding new features requires interaction with a person whose priorities probably do not match those of the beamline scientist. Thus MX is a rather hostile environment for the clever, creative, motivated beamline scientist. 16 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The wonder of MX The fundamental problem with MX is the same as with spec and some other control solutions. It’s quirky and suffers from fundamentally poor language design. It’s not open source. It’s incomplete and adding new features requires interaction with a person whose priorities probably do not match those of the beamline scientist. Thus MX is a rather hostile environment for the clever, creative, motivated beamline scientist. Our solution We wrote a python script. The python Mx bindings moved the mono and read the scalars. Matt Newville’s EPICS bindings talked to the Pilatus and controlled the synchronous motor motion. 16 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Tackling the metadata problem Our solution to munging an ASCII file and ∼ 125 tiff images into a XANES spectrum involved rigid assumptions about file names. One innocuous change to the DAQ python script or one case change in a file name and the postprocessing fails. The fundamental problem is the reliance on file names and the file system. Fortunately better technology exists: Hierarchical file HDF5 is a popular solution in some circles. Even a zip file would be a defensible solution (consider Open Document Foundation formats). HDF5 uses tags to encode the relationships. A zip file would require a combination of convention and encapsolated documentation. Database Data, images, and metadata can be stored in a relational database. Reconstuction of the data depends upon a documented table layout. 17 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away Tackling the metadata problem (2) In either case, images should be 1 collected from the server 2 compressed (integer tiffs are very compressable) 3 stored directly in the hierarchical file or the database An argument for a database would be for better consideration of questions that cross-cut many measurements. In this example, what is the cause of the glitch at about 12045 that appears in most data sets? 18 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The s32 tiff problem The extended tiff standard allows for a mind-boggling variety of data types stored at each pixel. Signed 32 bit integers are a perfectly reasonable data type that virtually no existing software supports. ImageJ reads s32 by not using libtiff. Consequently everyone using a Pilatus also uses ImageJ. Libtiff, out of the box, does not do the s32. Any viewing application that uses libtiff does not read Pilatus images. The Windows desktop image viewer does not read s32 tiffs. ImageMagick and GraphicsMagick both require a special build to handle s32. I was able to compile IM to do so on linux only, but not GM. Imager, a Perl extension, flattens s32 numbers into an u24 format. Because our pixel values are small and positive definate, this is acceptible. My solution I use Imager (I also played around with ImageMagick) to import the image and immediately stuff it into a Perl Data Language object. All subsequent work is done with PDL. 19 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The take away messages The control system must allow for synchronous motion of arbitrary motors. We lacked a control framework that is complete, well supported, linguistically robust, and accessible to the motivated beamline scientist or expert user. Prototyping control and acquisition applications at the beamline is important. Standard, maintained, supported libraries for whatever wonky image format the detectors spew out are essential. It must be easy to compile and/or install on all the platforms. We lacked an image processing framework that can be plugged into a measurement loop in interesting ways and used to post-process data. Hierarchical files or well-structured databases are superior vehicles to named files strewn about the network on various file systems. Open Source is good for science. Closed source is bad for science. 20 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The take away messages The control system must allow for synchronous motion of arbitrary motors. We lacked a control framework that is complete, well supported, linguistically robust, and accessible to the motivated beamline scientist or expert user. Prototyping control and acquisition applications at the beamline is important. Standard, maintained, supported libraries for whatever wonky image format the detectors spew out are essential. It must be easy to compile and/or install on all the platforms. We lacked an image processing framework that can be plugged into a measurement loop in interesting ways and used to post-process data. Hierarchical files or well-structured databases are superior vehicles to named files strewn about the network on various file systems. Open Source is good for science. Closed source is bad for science. 20 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The take away messages The control system must allow for synchronous motion of arbitrary motors. We lacked a control framework that is complete, well supported, linguistically robust, and accessible to the motivated beamline scientist or expert user. Prototyping control and acquisition applications at the beamline is important. Standard, maintained, supported libraries for whatever wonky image format the detectors spew out are essential. It must be easy to compile and/or install on all the platforms. We lacked an image processing framework that can be plugged into a measurement loop in interesting ways and used to post-process data. Hierarchical files or well-structured databases are superior vehicles to named files strewn about the network on various file systems. Open Source is good for science. Closed source is bad for science. 20 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The take away messages The control system must allow for synchronous motion of arbitrary motors. We lacked a control framework that is complete, well supported, linguistically robust, and accessible to the motivated beamline scientist or expert user. Prototyping control and acquisition applications at the beamline is important. Standard, maintained, supported libraries for whatever wonky image format the detectors spew out are essential. It must be easy to compile and/or install on all the platforms. We lacked an image processing framework that can be plugged into a measurement loop in interesting ways and used to post-process data. Hierarchical files or well-structured databases are superior vehicles to named files strewn about the network on various file systems. Open Source is good for science. Closed source is bad for science. 20 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The take away messages The control system must allow for synchronous motion of arbitrary motors. We lacked a control framework that is complete, well supported, linguistically robust, and accessible to the motivated beamline scientist or expert user. Prototyping control and acquisition applications at the beamline is important. Standard, maintained, supported libraries for whatever wonky image format the detectors spew out are essential. It must be easy to compile and/or install on all the platforms. We lacked an image processing framework that can be plugged into a measurement loop in interesting ways and used to post-process data. Hierarchical files or well-structured databases are superior vehicles to named files strewn about the network on various file systems. Open Source is good for science. Closed source is bad for science. 20 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The take away messages The control system must allow for synchronous motion of arbitrary motors. We lacked a control framework that is complete, well supported, linguistically robust, and accessible to the motivated beamline scientist or expert user. Prototyping control and acquisition applications at the beamline is important. Standard, maintained, supported libraries for whatever wonky image format the detectors spew out are essential. It must be easy to compile and/or install on all the platforms. We lacked an image processing framework that can be plugged into a measurement loop in interesting ways and used to post-process data. Hierarchical files or well-structured databases are superior vehicles to named files strewn about the network on various file systems. Open Source is good for science. Closed source is bad for science. 20 / 20 Spectrometry, Data Acquisition, and 2D Image Processing

BLA Measurement Improving on the art Take-away The take away messages The control system must allow for synchronous motion of arbitrary motors. We lacked a control framework that is complete, well supported, linguistically robust, and accessible to the motivated beamline scientist or expert user. Prototyping control and acquisition applications at the beamline is important. Standard, maintained, supported libraries for whatever wonky image format the detectors spew out are essential. It must be easy to compile and/or install on all the platforms. We lacked an image processing framework that can be plugged into a measurement loop in interesting ways and used to post-process data. Hierarchical files or well-structured databases are superior vehicles to named files strewn about the network on various file systems. Open Source is good for science. Closed source is bad for science. 20 / 20 Spectrometry, Data Acquisition, and 2D Image Processing