Total Nitrogen Estimation in Agricultural Soils via Aerial Multispectral Imaging and LIBS

Measuring soil health indicators (SHIs), particularly soil total nitrogen (TN), is an important and challenging task that affects farmers' decisions on timing, placement, and quantity of fertilizers applied in the farms. Most existing methods to measure SHIs are in-lab wet chemistry or spectroscopy-based methods, which require significant human input and effort, time-consuming, costly, and are low-throughput in nature. To address this challenge, we develop an artificial intelligence (AI)-driven near real-time unmanned aerial vehicle (UAV)-based multispectral sensing solution (UMS) to estimate soil TN in an agricultural farm. TN is an important macro-nutrient or SHI that directly affects the crop health. Accurate prediction of soil TN can significantly increase crop yield through informed decision making on the timing of seed planting, and fertilizer quantity and timing. The ground-truth data required to train the AI approaches is generated via laser-induced breakdown spectroscopy (LIBS), which can be readily used to characterize soil samples, providing rapid chemical analysis of the samples and their constituents (e.g., nitrogen, potassium, phosphorus, calcium). Although LIBS was previously applied for soil nutrient detection, there is no existing study on the integration of LIBS with UAV multispectral imaging and AI. We train two machine learning (ML) models including multi-layer perceptron regression and support vector regression to predict the soil nitrogen using a suite of data classes including multispectral characteristics of the soil and crops in red (R), near-infrared (NIR), and green (G) spectral bands, computed vegetation indices (NDVI), and environmental variables including air temperature and relative humidity (RH). To generate the ground-truth data or the training data for the machine learning models, we determine the N spectrum of the soil samples (collected from a farm) using LIBS and develop a calibration model using the correlation between actual TN of the soil samples and the maximum intensity of N spectrum. In addition, we extract the features from the multispectral images captured while the UAV follows an autonomous flight plan, at different growth stages of the crops. The ML model's performance is tested on a fixed configuration space for the hyper-parameters using various hyper-parameter optimization (HPO) techniques at three different wavelengths of the N spectrum.

Transcript

1. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Total Nitrogen Estimation in Agricultural Soils via Aerial Multispectral Imaging and LIBS Md Abir Hossen M.S. Thesis Defense Department of Electrical Engineering South Dakota School of Mines and Technology April 5, 2021 Abir Hossen UAV-based Multispectral Sensing Solution 1 / 72
2. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion 1 Motivation Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS 2 Feature Extraction and Dataset Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis 3 Machine Learning-based Predictive Models Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization 4 Conclusion Abir Hossen UAV-based Multispectral Sensing Solution 2 / 72
3. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Soil Health Indicators Courtesy: United States Department of Agriculture Soil health indicators are a composite set of measurable physical, chemical and biological properties, which can be used to determine soil health status Abir Hossen UAV-based Multispectral Sensing Solution 3 / 72
4. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Effects of N in Crop Production Nitrogen (N) is the most limiting nutrient in many of the world’s agricultural areas Insufficient use of N causes economic loss, in contrast, excessive use of N implies wasting fertilizer, causes nitrate pollution and increases the cost Nitrogen treatment can account for up to 30% of the total production cost Abir Hossen UAV-based Multispectral Sensing Solution 4 / 72
5. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS N Fertilizer Efficiency Farmers apply high Nrates (amount of N applied during the growing season) at early stages to avoid under-fertilization risk. However, approximately one-third of total N uptake occurs after pollination Global N recovery in crops is usually less than 50%, affecting N fertilizer efficiency and increasing pollution This problem led farmers, scientists and politicians to explore ways to improve N output, reduce N inputs, and prevent water and soil contamination from crop production Accurate prediction of soil total nitrogen (TN) can significantly increase crop yield through informed decision making on the timing of seed planting, and fertilizer quantity and timing. Abir Hossen UAV-based Multispectral Sensing Solution 5 / 72
6. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Traditional Methods to Estimate Soil Nitrogen Courtesy: Soil Sampling Tools Sample where the crop will be planted Avoid unusual areas Avoid contaminating the sample Take the soil sample to the correct depth Courtesy: PennState College of Agricultural Sciences Abir Hossen UAV-based Multispectral Sensing Solution 6 / 72
7. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Traditional Methods to Estimate Soil Nitrogen Courtesy: LaMotte STH-7 Courtesy: U.S. Department of Agriculture Abir Hossen UAV-based Multispectral Sensing Solution 7 / 72
8. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Ground-based Measurements- Chlorophyll meter (CM) Courtesy: Vegetation Measurements Chlorophyll meter (CM) measures the chlorophyll content to estimate N nutrition status However, CM-based methods fail to capture the spatial variability that is often present within the field. Abir Hossen UAV-based Multispectral Sensing Solution 8 / 72
9. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Satellite-based Remote Sensing Courtesy: OpenWeather Satellite-based techniques utilize images at the spectral level for crop growth monitoring and real-time management Vegetation indices (VIs), evaluated using the data obtained from satellite-based multispectral sensors, have been used to detect the N stress at V4-V7 (4-7 leaves with visible leaf collar) stages Abir Hossen UAV-based Multispectral Sensing Solution 9 / 72
10. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Satellite-based Remote Sensing Limitations Satellite-based sensing suffers from lower spatial and temporal resolution Sensing disruption may occur during image acquisition in some areas because of cloud cover and/or sprinkler irrigation Farmers’ adoption of the system is still limited The high cost of obtaining satellite images for relatively small areas is a significant drawback Abir Hossen UAV-based Multispectral Sensing Solution 10 / 72
11. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Advantage of using UAV for Multispectral Imaging Courtesy: 3DINSIDER UAVs can be deployed rapidly and frequently Reduced costs Greater flexibility in terms of data resolution and mission timing Abir Hossen UAV-based Multispectral Sensing Solution 11 / 72
12. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Laser-Induced Breakdown Spectroscopy (LIBS) Courtesy: WWU MUNSTER LIBS is an analytical method for qualitative and quantitative elemental detection With appropriate calibration, the LIBS analysis can provide quantitative measurement for most soil elements including Carbon, Nitrogen, Potassium, Sulfur, and Phosphorus Abir Hossen UAV-based Multispectral Sensing Solution 12 / 72
13. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Our Approach UAV-based Multispectral Sensing Solution and LIBS We develop an artificial intelligence(AI)-driven near real-time unmanned aerial vehicle (UAV)-based multispectral sensing (UMS) solution to estimate total nitrogen (TN) of the soil Abir Hossen UAV-based Multispectral Sensing Solution 13 / 72
14. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS Key Contributions We implement machine learning-based predictive models to estimate the total nitrogen of soil using multispectral imaging and LIBS We perform empirical studies to generate the dataset for ML models We determine the N nutrients in soil using LIBS and construct a calibration model We perform a numerical study to quantify the impact of various hyper-parameter optimization techniques on the performance of ML models. One of the key contributions of this thesis is to formulate a greedy search algorithm for hyper-parameter optimization 2.7x faster and achieved prediction accuracy comparable to standard HPO techniques Abir Hossen UAV-based Multispectral Sensing Solution 14 / 72
15. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Outline 1 Motivation Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS 2 Feature Extraction and Dataset Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis 3 Machine Learning-based Predictive Models Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization 4 Conclusion Abir Hossen UAV-based Multispectral Sensing Solution 15 / 72
16. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Generating Dataset Collect the multispectral images and soil samples from a farmland Generate the NDVI zones across the patches of the farmland Analyse soil samples using LIBS and determine the nitrogen lines among other nutrients Develop a calibration model using the correlation between actual TN (acquired from lab) and the N spectrum’s maximum intensity (acquired from LIBS) Abir Hossen UAV-based Multispectral Sensing Solution 16 / 72
17. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Data Collection- Multispectral Images Multispectral camera mounted on a Mavic 2 Pro UAV Raster scan pattern and photo dots The agricultural farm used for data collection is located in Sturgis, South Dakota, U.S.A. (44◦ 25 27 N; 103◦ 22 34 W) We created an autonomous UAV flight plan for minimal passes similar to a raster scan pattern using the coordinates of the four corners of the field We captured 865 multispectral images while the UAV was following the raster scan pattern Abir Hossen UAV-based Multispectral Sensing Solution 17 / 72
18. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Considerations While Flying the UAV The UAV was flown when the sun was highest in the sky for more accurate data Data is best when sky conditions are consistent, ideally 100% sunny or 100% cloudy Flying with a mix of sun and clouds causes inconsistency in brightness and contrast while stitching images resulting in an inaccurate NDVI value Abir Hossen UAV-based Multispectral Sensing Solution 18 / 72
19. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Data Collection- Soil Samples Captured multispectral images and soil sample locations Soil samples were collected at an 8-inch depth in each of the 6 patches at the V4, V8 and V12 stages using a hydraulic probe We avoided sampling from the areas where conditions were different from the rest of the field (e.g., former manure piles, fertilizer bands, or fence lines) Abir Hossen UAV-based Multispectral Sensing Solution 19 / 72
20. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Red (R), Green(G), and Near-infrared (NIR) Bands The multispectral images are composed of three channels, channel-1: R, channel-2: G and channel-3: NIR Channel-1 contains both R and NIR light (from the datasheet). Therefore, the NIR light needed to be removed to isolate R and compute NDVI Courtesy: Sentera Abir Hossen UAV-based Multispectral Sensing Solution 20 / 72
21. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Isolating R and NIR The equations for R and NIR light are, R = 1.0 ∗ DNch1 − 1.012 ∗ DNch3 (1) NIR = 9.605 ∗ DNch3 − 0.6182 ∗ DNch1 (2) where DNch1 is the Digital Number (pixel value) of channel one, and DNch3 is the Digital Number (pixel value) of channel three. The coefficients of DN were provided in the datasheet. Abir Hossen UAV-based Multispectral Sensing Solution 21 / 72
22. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Separated Bands (R, NIR, and G) Using Equation 1, and Equation 2, band separation was performed to extract the pixel values from each of the bands Separated bands We generate the dataset using the mean pixel values of NIR, R and G bands from individual zones Abir Hossen UAV-based Multispectral Sensing Solution 22 / 72
23. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis NIR and R Dataset We generate the zones by averaging the nearest R and NIR values of the individual pixels and perform the grouping using the zonal statistics package of QGIS Abir Hossen UAV-based Multispectral Sensing Solution 23 / 72
24. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Normalized Difference Vegetation Index (NDVI) Visible lights (Red, Blue) are strongly absorbed by the healthy vegetation (chlorophyll) compared to other wavelengths Courtesy: NASA NIR and Green lights are strongly reflected by the cellular composition of the leaves. The plants absorb more of the NIR light when the plant becomes dehydrated and sick. Abir Hossen UAV-based Multispectral Sensing Solution 24 / 72
25. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis NDVI Dataset Substitute for R and NIR, using Equation 1 and Equation 2, NDV I = 1.236 ∗ DNch3 − 0.188 ∗ DNch1 1.000 ∗ DNch3 − 0.044 ∗ DNch1 (3) We capture the multispectral images at V4, V8, and V12 stages and compute the NDVI for each stage using Equation 3 Computed NDVI pixels and zones for 6 patches Abir Hossen UAV-based Multispectral Sensing Solution 25 / 72
26. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis LIBS Emission Spectrum Ten laser pulses are shot on the soil samples to obtain averaged data on each measurement The focused laser on the soil surface forms a µm size of a sample into > 10, 000◦K plasma The spectrometer collects the unique emission spectrum as the plasma cools Emission lines of soil samples at the V4, V8, and V12 stages for six patches Abir Hossen UAV-based Multispectral Sensing Solution 26 / 72
27. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Determining N Lines using NIST Database Abir Hossen UAV-based Multispectral Sensing Solution 27 / 72
28. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Calibration From the soil samples, we obtain actual TN in ppm and analyze the samples in LIBS to determine the maximum intensity from the N spectrum for all the samples at V4, V8 and V12 stages Using the correlation between actual TN and the maximum intensity of N spectrum, we generate a calibration plot through linear regression Calibration plot for computing soil TN using the peak intensity of the nitrogen spectrum at 493.4 nm, 821.4 nm, and 868.1 nm Abir Hossen UAV-based Multispectral Sensing Solution 28 / 72
29. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Calibration Using the calibrated model, we converted the peak intensity of the N spectrum to TN (ppm) for all the samples Nitrogen spectrum at 493.4 nm of the soil samples for six patches at the V4, V8 and V12 stages Abir Hossen UAV-based Multispectral Sensing Solution 29 / 72
30. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Calibration- 821.4 nm Nitrogen spectrum at 821.4 nm of the soil samples for six patches at the V4, V8 and V12 stages Abir Hossen UAV-based Multispectral Sensing Solution 30 / 72
31. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis Calibration- 868.1 nm Nitrogen spectrum at 868.1 nm of the soil samples for six patches at the V4, V8 and V12 stages Abir Hossen UAV-based Multispectral Sensing Solution 31 / 72
32. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Outline 1 Motivation Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS 2 Feature Extraction and Dataset Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis 3 Machine Learning-based Predictive Models Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization 4 Conclusion Abir Hossen UAV-based Multispectral Sensing Solution 32 / 72
33. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Supervised Learning Algorithms In supervised learning, the goal is to obtain an optimal predictive model function f∗ based on the input x and the output y to minimize the cost function L(f(x), y) We particularly use Multi-layer Perceptron (MLP) and Support Vector Machine (SVM) which can be used for both classification and regression problems Abir Hossen UAV-based Multispectral Sensing Solution 33 / 72
34. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Multi-layer Perceptron (MLP) MLP learns a function f(.) : Rx → Ro by training on a dataset, where x is the number of input dimension and o is the number of output dimension For a given a set of features x = {R, NIR, G, NDV I, AirTemperature, RH} and target y = TN, f(.) : R6 → R1 where input layer ∈ R6, hidden layer ∈ R4, output layer ∈ R1 and n1, n2, n3, and n4 represent the number of perceptron in each hidden layer, respectively Abir Hossen UAV-based Multispectral Sensing Solution 34 / 72
35. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization MLP Hyper-parameter Configuration The hyper-parameter configuration was created using the solver type, activation function, learning rate, and hidden layer sizes MLP Regressor Hyper-parameter Abir Hossen UAV-based Multispectral Sensing Solution 35 / 72
36. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Support Vector Machine (SVM) SVM makes data points linearly separable by mapping them from low-dimensional to high-dimensional space The classification boundary creates a partition between the data points by generating a hyperplane For n data points, the objective function of SVM can be written as: arg min w n i=1 max{0, 1 − yi f(xi )} + CwT w (4) where w is a normalization vector; C is the penalty parameter of the error term Abir Hossen UAV-based Multispectral Sensing Solution 36 / 72
37. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization SVM Kernels SVM uses different types of kernel functions to measure the similarities between two data points xi and yi , can be set to different types in SVM models Common kernel types in SVM: Linear kernel Polynomial kernel Radial basis function (RBF) kernel Sigmoid kernel Abir Hossen UAV-based Multispectral Sensing Solution 37 / 72
38. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization SVM Hyper-parameter Configuration We create the hyper-parameter configuration using the kernel types, generalization parameter (C), and error (epsilon) of the loss function SVM Regressor Hyper-parameter Abir Hossen UAV-based Multispectral Sensing Solution 38 / 72
39. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Hyper-parameter Configuration Specifics of the configuration space for the hyper-parameters Abir Hossen UAV-based Multispectral Sensing Solution 39 / 72
40. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Hyper-parameter Optimization The process of determining the optimal hyper-parameter configuration is known as hyper-parameter optimization Courtesy: Towards data science We want to find the model hyper-parameters that yield the best score on the validation set metric Abir Hossen UAV-based Multispectral Sensing Solution 40 / 72
41. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Hyper-parameter Optimization Hyper-parameter optimization is represented in equation form as: λ∗ = argmin λ∈A Ex∼Gx [L(x; Aλ (X(train)))] (5) The objective of a learning algorithm A is to find a function f that minimizes expected loss L(x; f) over samples x from a natural distribution Gx A learning algorithm A maps a dataset X(train) to a function f The actual learning algorithm is determined after choosing λ, which can be denoted Aλ and f = Aλ (X(train)) for trainig set X(train) We need to choose λ in a way to minimize generalization error Ex∼Gx [L(x; Aλ (X(train)))] Our goal is to identify a good value for hyper-parameters λ Hyper-parameter optimization is minimization of λ∗. Abir Hossen UAV-based Multispectral Sensing Solution 41 / 72
42. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization HPO Techniques Grid Search Random Search Genetic Algorithm Greedy Search Abir Hossen UAV-based Multispectral Sensing Solution 42 / 72
43. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Grid Search (GS) In GS, we specify a finite set of hyper-parameter combinations in the grid configuration, then the algorithm exhaustively evaluates all the combinations in the hyper-parameter configuration space Flowchart for identifying the global optimums in GS Abir Hossen UAV-based Multispectral Sensing Solution 43 / 72
44. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Disadvantage of Grid Search GS hyper-parameter configuration example Grid search is inefficient in high-dimensionality hyper-parameter configuration space, since the number of evaluations grows exponentially with the number of hyper-parameters This exponential growth is referred to as the curse of dimensionality Abir Hossen UAV-based Multispectral Sensing Solution 44 / 72
45. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Random Search (RS) In RS, we define a budget (i.e., time) and the upper and lower boundary of the hyper-parameter values RS randomly selects the values from the pre-defined boundary and trains until the budget is exhausted If the configuration space is wide enough, RS is able to detect the global optimums Abir Hossen UAV-based Multispectral Sensing Solution 45 / 72
46. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Random Search vs Grid Search Let’s assume two hyper-parameters, Hyperparameter-1 and Hyperparameter-2 In GS, the hyper-parameters are fixed in grid combinations. Thus the chances of falling into the white region are less compared to RS Abir Hossen UAV-based Multispectral Sensing Solution 46 / 72
47. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Random Search vs Grid Search Hyperparameter-1 and Hyperparameter-2 are arbitrary hyper-parameters that aren’t necessarily equal in importance Assuming Hyperparameter-1 is more important than Hyperparameter-2, which means a small change in Hyperparameter-1 is more important than a change in Hyperparameter-2 Abir Hossen UAV-based Multispectral Sensing Solution 47 / 72
48. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Genetic Algorithm (GA) GA randomly initializes the population and chromosomes Genes represent the entire search space, hyper-parameters, and hyper-parameter values GA uses a fitness function to evaluate each individual’s performance in the current generation To produce a new generation, GA performs selection, crossover, and mutation operations on the chromosomes involving the next hyper-parameter configurations to be evaluated Abir Hossen UAV-based Multispectral Sensing Solution 48 / 72
49. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Genetic Algorithm (GA) Flowchart for identifying the global optimums in GA Abir Hossen UAV-based Multispectral Sensing Solution 49 / 72
50. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Pseudo-code of the Genetic Algorithm Abir Hossen UAV-based Multispectral Sensing Solution 50 / 72
51. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Greedy Search We obtain the RMSE locally for each hyper-parameter Each iteration adds a new hyper-parameter value Hv (Hv = {v1 , ..., vn }) from the hyper-parameter configuration space The greedy evaluation function attempts to improve RMSE in an iterative fashion, by successively replacing the current Hv (higher RMSE) by a better Hv (lower RMSE) in a neighborhood of the current Hv A global lowest RMSE can be reached by choosing the lowest RMSE from each hyper-parameter Abir Hossen UAV-based Multispectral Sensing Solution 51 / 72
52. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Greedy Search Visual Representation Example of finding Hv locally that yields the lowest RMSE using greedy search Abir Hossen UAV-based Multispectral Sensing Solution 52 / 72
53. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Pseudo-code of Greedy Search Abir Hossen UAV-based Multispectral Sensing Solution 53 / 72
54. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Hyper-parameter Tuning Processes Abir Hossen UAV-based Multispectral Sensing Solution 54 / 72
55. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Performance comparison of Different HPO Techniques We measured Computational Time (CT) as a model efficiency metric The optimal hyper-parameter configuration was determined based on the lowest RMSE We specify the same hyper-parameter configuration space to fairly compare the GS, RS and GA HPO techniques Performance comparison for different HPO algorithms at different wavelengths Abir Hossen UAV-based Multispectral Sensing Solution 55 / 72
56. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Prediction on the Test-dataset Abir Hossen UAV-based Multispectral Sensing Solution 56 / 72
57. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Prediction on the Test-dataset Abir Hossen UAV-based Multispectral Sensing Solution 57 / 72
58. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Prediction on the Test-dataset Abir Hossen UAV-based Multispectral Sensing Solution 58 / 72
59. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Estimation error for Predicting Total Nitrogen of Soil The estimation error of predicting soil TN is lowest in GA compared to GS and RS for both MLP and SVM, where µ is the mean and σ is the standard deviation While training the models, we split our dataset into train and test, where we use 80% of the data for training and 20% for testing Abir Hossen UAV-based Multispectral Sensing Solution 59 / 72
60. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Greedy Search vs Genetic Algorithm We apply our greedy search algorithm on SVM model (SVM greedy) and compare it with SVM GA that yielded the lowest estimation error for predicting soil TN in our dataset The SVM greedy is 2.7 times faster in CT and has only 2.5% additional root mean square percent error (RMSPE) Abir Hossen UAV-based Multispectral Sensing Solution 60 / 72
61. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Greedy Search Submodularity Check Submodular functions satisfy the property of diminising marginal returns. That is, as a set becomes larger, the contribution of any factor X to the overall value of the set decreases. Nemhauser, Wolsey, and Fisher proved that if an objective function has submodular property then the solution from greedy algorithm is guaranteed to be at least (1 − 1\e) ≈ 0.63-approximation of the optimal objective value We perform a numerical test to check the submodularity property of the SVM greedy search trying different combinations of the SVM hyper-parameters For bench-marking, we use the RMSE of SVM GA (RMSE = 94.14) and compare it with the RMSE of our greedy search algorithm Abir Hossen UAV-based Multispectral Sensing Solution 61 / 72
62. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization Greedy Search Subsidiarity Check For all the hyper-parameter combinations, the SVM greedy search reached 89.1% − 92.79% of global optimum Different combinations of SVM hyper-parameters to check submodularity of SVM greedy search Abir Hossen UAV-based Multispectral Sensing Solution 62 / 72
63. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Outline 1 Motivation Soil Health Indicators Effects of Nitrogen (N) in Crop Production Estimation of N Nutrient UAV-based Multispectral Sensing Solution and LIBS 2 Feature Extraction and Dataset Experimental Design Multispectral Characteristics Extraction Laser-Induced Breakdown Spectroscopy (LIBS) Analysis 3 Machine Learning-based Predictive Models Hyper-parameters in Supervised Learning Algorithms Hyper-parameter Optimization (HPO) HPO Techniques Applying Hyper-parameter Optimization 4 Conclusion Abir Hossen UAV-based Multispectral Sensing Solution 63 / 72
64. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Conclusions We have demonstrated the ability of a UAV-based multispectral sensing solution to estimate the total nitrogen of soil We implemented two machine learning models multilayer perceptron and support vector machine to predict soil total nitrogen using a suite of data classes including UAV-based imaging data in red, near infrared, and green spectral bands, normalized difference vegetation indices (computed using the multispectral images), air temperature, and relative humidity To generate the training data for the ML models, we determine the N spectrum of the soil samples (collected from a farm) using LIBS and develop a calibration model using the correlation between actual TN of the soil samples and maximum intensity of N spectrum. Abir Hossen UAV-based Multispectral Sensing Solution 64 / 72
65. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Conclusions In addition, we extract the features from the multispectral images captured while the UAV was following an autonomous flight plan, at different growth stages of the crops. We performed hyperparameter optimization methods to tune the models for prediction performance Overall, our numerical studies confirm that our machine learning-based predictive models can estimate total nitrogen of the soil with a root mean square percent error (RMSPE) of 10.8% Our greedy search algorithm for hyper-parameter optimization is 2.7x faster and achieved prediction accuracy comparable to its counterpart, genetic algorithm Abir Hossen UAV-based Multispectral Sensing Solution 65 / 72
66. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Limitations Depending on the type of soil and crops the model needs to be re-calibrated The actual TN of soil should be obtained from the subset of the samples to calibrate the N spectrum’s intensity after determining the N lines using LIBS N lines that falls around the 500 nm region should be avoided in sea sand because of interferences with Titanium (Ti) lines Abir Hossen UAV-based Multispectral Sensing Solution 66 / 72
67. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Future Scope Our future work will consider other environmental factors such as ground moisture, soil humidity, and estimation of additional soil nutrients such as potassium (K), and phosphorus (P) We will apply advanced hyper-parameter optimization techniques (e.g., Bayesian Optimization, Particle swarm optimization (PSO)) We will also explore the submodularity property of the SVM greedy search algorithm through a mathematical proof Abir Hossen UAV-based Multispectral Sensing Solution 67 / 72
68. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Deployment The UMS can be deployed to the field to predict the total nitrogen of soil in near real-time However, due to the limited computational power on the UAV’s on-board microcontroller it is impossible to perform the stitching task of the multispectral images using the standalone UAV To overcome the limitation, distributed neural network can be used along with quantization techniques Abir Hossen UAV-based Multispectral Sensing Solution 68 / 72
69. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Deployment UMS adapts machine learning services to device-to-cloud continuum to enable perception Abir Hossen UAV-based Multispectral Sensing Solution 69 / 72
70. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Deployment Inference of TN using quantization and distributed neural network Abir Hossen UAV-based Multispectral Sensing Solution 70 / 72
71. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion MS Publications Md Abir Hossen, Prasoon K Diwakar, Shankarachary Ragi, “Total Nitrogen Estimation in Agricultural Soils via Aerial Multispectral Imaging and LIBS”. Scientific Reports (2021). (revision stage) Abir Hossen UAV-based Multispectral Sensing Solution 71 / 72
72. Motivation Feature Extraction and Dataset Machine Learning-based Predictive Models Conclusion Questions? Abir Hossen UAV-based Multispectral Sensing Solution 72 / 72