SuperSmartLab.pdf

Transcript

1. SUPERCHARGING SCIENCE IN THE SMART LABORATORY KEITH T. BUTLER 1 https://tinyurl.com/y6lbtap3 https://speakerdeck.com/keeeto/ktb-ssi
2. SOLID-STATE IONICS 2019 PYEONGCHANG SCIML/STFC/RAL ▸ Scientific machine learning group ▸ Interact with Facilities/Academia/Industry 2
3. SINGAPORE 2019 ACADEMIA / INDUSTRY IN AI ▸ These companies all have many, very large, private datasets that they will never make publicly available ▸ Each of these companies employs many hundreds of computer scientists with PhDs in Machine Learning and AI ▸ Their researchers and developers have essentially unlimited computing power at their disposal 3
4. SOLID-STATE IONICS 2019 PYEONGCHANG DATA COMES FROM FACILITIES ▸ National facilities are data rich ▸ Eg single time-resolved tomographic experiment = 100 TB data 4 Diamond Light Source ISIS Neutron and Muon Central laser facility Electron microscopy facility PP Data Tier 1 JASMIN environmental data
5. SINGAPORE 2019 THE ALAN TURING INSTITUTE 5
6. TEXT NVIDIA DGX-2
7. SINGAPORE 2019 5 KEY QUESTIONS BEFORE GOING ML ▸ What do I want to achieve? ▸ How much data do I have/can I get? ▸ What kind of data do I have? ▸ Do I care more about prediction or inference? ▸ What kind of hardware do I have? 7
8. SINGAPORE 2019 OVERVIEW ▸ Some useful traditional ML approaches ▸ Deep networks for materials science ▸ SciML @ RAL projects 8
9. SINGAPORE 2019 CLASSICAL/DEEP MACHINE LEARNING ▸ Example decision tree (classical); neural network (deep) 9 Robustness Scaling Interpretability Simplicity Speed Accuracy ANN DT Traditional ML Deep NN Performance Data
10. SINGAPORE 2019 DECISION TREES ▸ Has a band gap Y/N 10 At each node, the space is split such that samples with similar labels are grouped together
11. SINGAPORE 2019 DECISION TREES 11 For a trial split at node j… The impurity of j is calculated for a trial split using an impurity function H()… And are chosen in a greedy fashion…
12. SINGAPORE 2019 ENSEMBLE LEARNER ▸ Decision trees are weak learners ▸ A group of trees can overcome limitations ▸ Can optimise the group or chose randomly ▸ Random forest ▸ Gradient boosted 12
13. SINGAPORE 2019 EXAMPLE: GRADIENT BOOSTED 13 + + Root mean squared error (RMSE) Sub model of errors Learning rate …
14. SINGAPORE 2019 SUPPORT VECTOR MACHINES (CLASSIFICATION) ▸ SVMs seek to separate classes of observation ▸ Additional constraint of maximum margins ▸ Use a hyper-plane (a plane with one dimension less than the feature space) 14 https://towardsdatascience.com/support-vector-machine-simply-explained-fee28eba5496
15. SINGAPORE 2019 SVMS IN NON-LINEAR SEPARATIONS ▸ Classes not linearly separable in the feature space ▸ Soft margins ▸ Kernel trick 15 https://towardsdatascience.com/support-vector-machine-simply-explained-fee28eba5496
16. SINGAPORE 2019 SVMS WITH SOFT MARGINS ▸ Tolerate a certain number of mis-classifications to maximise the margin ▸ Trade-off between mis-classification and margin width ▸ Tolerance hyper-parameter determines the balance 16 Classification is more important than margin Margin is more important than classification
17. SINGAPORE 2019 SVMS WITH THE KERNEL TRICK ▸ Combine and manipulate existing parameters to create new parameters ▸ Move the objects to a new dimensional space ▸ See if the classes are linearly separable in the new space 17 Not separable in standard space Apply polynomial kernel => separable
18. SINGAPORE 2019 CLASSICAL MODELS SUMMARY 18
19. SINGAPORE 2019 NEURAL NETWORKS ▸ A history of neural nets ▸ Rise-Fall-Rise-Fall-Rise-? ▸ The elements of a network ▸ Neurons, connections, optimisers ▸ Modern networks: CNNs ▸ Image recognition, feature detection etc 19
20. SINGAPORE 2019 THE PERCEPTRON ▸ Originally a device ▸ Intended for binary classification ▸ Produces a single output from a matrix of inputs, weights and biases 20
21. SINGAPORE 2019 THE FIRST FALL OF NEURAL NETWORKS ▸ Single layer ▸ Minsky and Papert showed they could not solve non-linear classification 21
22. SINGAPORE 2019 THE NEXT WAVE OF NEURAL NETS: 1980S ▸ Back propagation ▸ Now gradients could be used to minimise error ▸ Modifications back propagate through the network using the chain rule 22
23. SINGAPORE 2019 THE NEXT WAVE OF NEURAL NETS: 1980S ▸ Multi-layer perceptrons (MLPs) ▸ Can now solve non-linear problems 23
24. SINGAPORE 2019 THE ELEMENTS OF A NEURAL NETWORK ▸ Input layer ▸ Hidden layers ▸ Output layer 24
25. SINGAPORE 2019 THE ELEMENTS OF A NEURAL NETWORK ▸ Input layer ▸ Hidden layers ▸ Output layer 25 ‣ Data structure ‣ Features of the data
26. SINGAPORE 2019 THE ELEMENTS OF A NEURAL NETWORK ▸ Input layer ▸ Hidden layers ▸ Output layer 26 ‣ Regression ‣ Single unit (usually) ‣ Classification ‣ Multiple units ‣ One-hot encoding
27. SINGAPORE 2019 HIDDEN LAYERS ▸ Neurons ▸ Connections 27 Outputs from previous Matrix of weights Bias Activation function Signal to next layer
28. SINGAPORE 2019 BACK PROPAGATION ▸ Learn from the loss 28 LOSS(L) y[1] y[2] y[l] dy[l] Notation: dL/dy[l] => dy[l] dy[1] dy[2] dw[1] db[1] dw[2] db[2] dw[l] db[l] y[0]
29. SINGAPORE 2019 TRAINING ▸ Run back-prop until a criterion is met ▸ Loss functions ▸ Cross-entropy (categorisation) ▸ Mean average error (regression) ▸ Optimisers ▸ Stochastic gradient descent ▸ ADAM 29 Validation Training Accuracy Epoch
30. SINGAPORE 2019 MLPS STRUGGLE WITH IMAGE RECOGNITION ▸ For a computer, literally these do not match ▸ The MLP has no real concept of the spatial relations ▸ Also, dense connections lead to parametric explosions for many pixel images 30
31. SINGAPORE 2019 CONVOLUTIONAL NEURAL NETS (CNNS) ▸ Uses filters to pick out important features ▸ Compresses image information ▸ Is finally connected to a typical NN layer ▸ Successful CNNs are often very deep 31
32. SINGAPORE 2019 HOW CNNS WORK ▸ Filters work to pick out features in the image 32 0.8 1 1 0.5 1 1 0.8 =
33. SINGAPORE 2019 HOW CNNS WORK ▸ Filter is a matrix ▸ Filter dot product with image to produce scalar ▸ A number of filters are added at each layer 33 5x5 Filter 32x32 image 28x28 image
34. SINGAPORE 2019 LOOKING AT THE FILTERS ▸ Visualise trained filters to see what they detect ▸ Example form the AlexNet network ▸ Filters pick up edges and colours 34
35. SINGAPORE 2019 LOW DATA LEARNING OF CNNS ▸ Often existing feature maps will work for a new problem ▸ Can load existing models and weights ▸ Retrain on a small labelled dataset ▸ Transfer learning 35 Performance Data From scratch Transfer
36. SOLID-STATE IONICS 22 PYEONGCHANG NEURAL NETWORKS FOR TIME SERIES DATA ▸ Often algorithms are desired for predicting the next event based on a series of previous events ▸ Eg Pressure/temperature evolution, speech prediction … ▸ In this case standard NNs are not very useful due to a lack of ‘memory’ 36 Feed forward network Information never touches a node twice
37. SOLID-STATE IONICS 22 PYEONGCHANG RECURRENT NEURAL NETS ▸ Recurrent networks re-apply a representation of the state from the previous step ▸ This is combined with the new information to influence the outcome of the present step ▸ This gives the network memory - but only for one step 37 Recurrent network Information is fed back to the node at the next step http://colah.github.io/posts/2015-08-Understanding-LSTMs/
38. SOLID-STATE IONICS 22 PYEONGCHANG LONG SHORT TERM MEMORY NETWORKS ▸ LSTMs store representations in separate memory units ▸ These have three gates ▸ Input - decides if a state should enter memory ▸ Output - decides if memory should affect the current state ▸ Forget - decides if memory should be dumped ▸ Very effective for time series problems 38 http://colah.github.io/posts/2015-08-Understanding-LSTMs/ LSTMs have concurrent memory and processing streams
39. SOLID-STATE IONICS 22 PYEONGCHANG LSTMS FOR MATERIALS CHARACTERISATION ▸ LSTM can predict the likelihood of a structural transition during operando measurement of a material ▸ Allows for optimisation of experiment and identification of the region of interest 39 https://doi.org/10.1145/3217197.3217204
40. SINGAPORE 2019 THREE WAYS TO USE A CNN ▸ Regression ▸ Classification ▸ Segmentation 40
41. SINGAPORE 2019 THREE WAYS WE USE CNNS TO ACCELERATE SCIENCE ▸ Regression ▸ Classification ▸ Segmentation 41
42. SINGAPORE 2019 CNNS FOR ANALYSIS OF SPIN WAVE MATERIALS ▸ Rb2MnF4 ▸ Traditional approach - take many slices of data, use slices to refine fitting with Hamiltonians ▸ Detailed analysis one experiment = full paper to analyse 42
43. SINGAPORE 2019 CNNS FOR ANALYSIS OF SPIN WAVE MATERIALS 43 ▸ Rb2MnF4 ▸ ML approach - train a network on examples from simulations - infer coupling constants directly from data ▸ Data generation and training done in advance - analysis takes minutes
44. SINGAPORE 2019 THREE WAYS TO USE A CNN ▸ Regression ▸ Classification ▸ Segmentation 44
45. SINGAPORE 2019 CNNS FOR ANALYSIS OF MORPHOTROPIC PHASE BOUNDARIES ▸ Images are compressed by filters ▸ Filters are updated to learn the important features of the image 45 Feature maps 32@486x194 3x3 kernel Feature maps 64@242x96 3x3 kernel Fully connected Layers 16 nodes 8 nodes Identify lattices present Butler, Proc. Royal Soc. A - Under Review
46. SINGAPORE 2019 THREE WAYS TO USE A CNN ▸ Regression ▸ Classification ▸ Segmentation 46 Micros. Microanal. 25, 21, 2019
47. SINGAPORE 2019 OUR VISION: A SMART LABORATORY, SUPERCHARGING SCIENCE 47 ACCELERATED DISCOVERY ACCELERATED ANALYSIS ACCELERATED CHARACTERISATION DATA KNOWLEDGE
48. SINGAPORE 2019 ACKNOWLEDGMENTS ▸ Tony Hey, Jeyan Thiyagalingam, Rebecca Mackenzie, Sam Jackson (SciML) ▸ Toby Perring, Duc Le (ISIS Neutron and Muon Source) ▸ Gareth Nisbet, Steve Collins (Diamond Light Source) ▸ Alex Leung, Peter Lee (Research Complex at Harwell, UCL) 48
49. SINGAPORE 2019 THANK YOU 49 NATURE, 2018, 559, 547. @keeeto2000 @ml_sci keeeto.github.io www.scd.stfc.ac.uk/ Pages/Scientific- Machine-Learning.aspx