Bayesian Neural Networks

Johanna Fleckner, Project Manager & Data Scientist at Blue-Yonder at Data Science London @ds_ldn meetup March 28th, 20123

Transcript

1. The Physics of Everyday Life Bayesian Neural Networks in Business Applications Data Science London – March 2013 Dr. J. Fleckner, Blue Yonder GmbH & Co. KG, Karlsruhe, Germany Johanna.Fleckner@blue-yonder.com
2. What does the future hold? Page 2 What would happen if... ... a large supermarket chain knew precisely how much fresh fruit it will sell? 28.03.2013 Bayesian Neural Networks in Business Applications
3. Parietal Cortex Frontal Lobe Motor Cortex Temporal Lobe Brain Stem Occipital Lobe Cerebellum Neural networks Self learning procedures, copied from nature 3 Bayesian Neural Networks in Business Applications One way to construct a one dimensional test statistic from multidimensional input (a MVA- method): 28.03.2013
4. Artificial Neural Networks: The NeuroBayes Algorithm 4 Bayesian Neural Networks in Business Applications The NeuroBayes classification core is based on a simple feed forward neural network. The information (the knowledge, the expertise) is coded in the connections between the neurons. Each neuron performs fuzzy decisions. A neural network can learn from examples. Human brain: about 100 billion ( 1011 ) neurons about 100 trillion ( 1014 ) connections NeuroBayes: 10 to few 100 neurons 28.03.2013
5. Bayes‘ Theorem 28.03.2013 Seite 5 Bayesian Neural Networks in Business Applications ►  Posterior: Probability that the theory is true given the data ►  N.B.:In a lot of cases a flat prior is assumed (Likelihood-fit) in order to express the lack of knowledge about the true distribution ►  In most cases the flat prior is wrong! Posterior Evidence Likelihood Prior
6. Bayes‘ Theorem and NeuroBayes 6 Bayesian Neural Networks in Business Applications 28.03.2013 Posterior Evidence Likelihood Prior ►  NeuroBayes internally uses Bayesian arguments for regularisation ►  NeuroBayes automatically makes Bayesian posterior statements
7. Mode I: Classifications 28.03.2013 Seite 7 Bayesian Neural Networks in Business Applications Classification: Binary targets: Each single outcome will be “yes“ or “no“ NeuroBayes output is the Bayesian posterior probability that answer is “yes“ (given that inclusive rates are the same in training and test sample, otherwise simple transformation necessary). Examples: ►  This elementary particle is a K meson. ►  This jet is a b-jet. ►  Customer Meier will cancel his contract next year.
8. Probability density for real valued targets: For each possible (real) value a probability (density) is given. From that all statistical quantities like mean value, median, mode, standard deviation, etc. can be deduced. Mode II: Conditional probability densities 8 Bayesian Neural Networks in Business Applications Examples: ►  Energy of an elementary particle (e.g a semileptonically decaying B meson with missing neutrino) ►  Lifetime of a decay ►  Turnaround of an article next year (very important in industrial applications) 28.03.2013
9. NeuroBayes classifications 9 Bayesian Neural Networks in Business Applications NeuroBayes Teacher: Learning of complex relationships from existing data bases (e.g. Monte Carlo) NeuroBayes Expert: Prognosis for unknown data Output Input Significance control Postprocessing Preprocessing 28.03.2013
10. Neural Network: basic functions 10 Bayesian Neural Networks in Business Applications 28.03.2013
11. Neural network transfer functions 11 Bayesian Neural Networks in Business Applications 28.03.2013
12. Historic or simulated data Data set a = ... b = ... c = ... .... t = …! NeuroBayes® Teacher NeuroBayes® Expert Actual (new real) data Data set a = ... b = ... c = ... .... t = ? Expertise Expert system f t Probability that hypothesis is correct (classification) or probability density for variable t t How it works: training and application 12 Bayesian Neural Networks in Business Applications 28.03.2013
13. Backpropagation (Rumelhardt et al. 1986): Calculate gradient backwards by applying chain rule Optimise using gradient descent method. Step size?? Neural network training 13 Bayesian Neural Networks in Business Applications 28.03.2013
14. Easy to find the next local minimum... but globally... ...impossible! è needs good preconditioning Neural network training: the challange 14 Bayesian Neural Networks in Business Applications Difficulty: find global minimum of highly non-linear function in high (~ >100) dimensional space. Imagine task to find deepest valley in the Alps (just 2 dimensions) 28.03.2013
15. NeuroBayes – key features 15 Bayesian Neural Networks in Business Applications ►  is based on 2nd generation neural network algorithms, Bayesian regularisation, optimised preprocessing with non-linear transformations and decorrelation of input variables and linear correlation to output. ►  learns extremely fast due to 2nd order BFGS methods and even faster with 0-iteration mode. ►  produces small expertise files. ►  is extremely robust against outliers in input data. ►  is immune against learning by heart statistical noise. ►  tells you if there is nothing relevant to be learned. ►  delivers sensible prognoses already with small statistics. ►  can handle weighted events, even negative weights. ►  has advanced boost and cross validation features. ►  is steadily further developed professionally. 28.03.2013
16. Strategy Summary 16 Bayesian Neural Networks in Business Applications 28.03.2013
17. Application of NeuroBayes in fundamental research at the forefront of science »  Extensive research experience in particle physics »  More than 350 man-years spent to develop NeuroBayes machine learning solution. »  NeuroBayes used at all major particle accelerators: »  CERN (Geneva, Switzerland): LEP, LHC »  Fermilab (Chicago, USA): CDF »  KEK (Tsukuba/Tokai, Japan) : Belle, Belle II CERN (CH) KEK (Japan) Fermilab (USA) Page 17 28.03.2013 Bayesian Neural Networks in Business Applications
18. Blue Yonder – forward looking, forward thinking Page 18 Now about 100 employees of which most are post-docs, mainly from HEP. Doubling our numbers in 2012, 2013 is looking good… 3 Offices: Karlsruhe, Hamburg (Germany) London (UK) Started as a spin-off from the University of Karlsruhe, Germany supported by the Federal Ministry for Education and Research. 28.03.2013 Bayesian Neural Networks in Business Applications
19. Automated replenishment In supermarkets Fondsmanagement Insurance: Risk prediction Fashion: Sales prediction Media Churn Management Artikelabsatz P E(X)=413 Stock Exchange Order Placement System SAP NeuroBayes® NeuroBayes from Science to Industry Predictive Analytics is the key to many industries Page 19 28.03.2013 Bayesian Neural Networks in Business Applications
20. Use all available and relevant information as input, e.g. measurements from the various sub-detectors, … NeuroBayes will extract statistically significant patterns in the data to derive the prediction. Prediction will return the best estimator for a measurement including a statistically sound estimation of the expected spread. 100 Energy Momentum Direction Type 50 90 Sub-Detector Distance 200 Calo Kaon ... propabilityP Particle Property E(X) NeuroBayes from Science to Industry Predictive Analytics in High Energy Physics Page 20 28.03.2013 Bayesian Neural Networks in Business Applications
21. Use all available and relevant information as input, e.g. article properties, previous sales, etc NeuroBayes will extract statistically significant patterns in the data to derive the prediction. Prediction will return e.g. the most probable sales rate including a statistically sound estimation of the expected spread. Article size Picture size colour Previous sales M 21% red brand price 19,9 171 24 ... propabilityP Prediction sales E(X) NeuroBayes from Science to Industry Predictive Analytics in industry E.g. Retail NeuroBayes allows data-driven analysis and forecasts – both in science and industry Page 21 28.03.2013 Bayesian Neural Networks in Business Applications
22. »  Most conventional methods: Forecast is a single number »  No estimate how precise this number is »  Does not allow to handle asymmetric distribution of probabilities »  NeuroBayes: Prediction of a full probability density distribution Asymmetric probability density distribution X1 : most probable value (n.b. all other values may still occur) P (x) quantity(x) x1 Optimal estimate for your use-case X2 : Median: 50% of all values are smaller, 50% larger than this x2 NeuroBayes from Science to Industry NeuroBayes core technology Page 22 28.03.2013 Bayesian Neural Networks in Business Applications
23. Grocery Chain: Auto Replenishment Predictions from Blue Yonder vs. In-House solution Blue Yonder Forecasted (Actual) In-House Solution Forecast (Actual) Pecentage of write-offs CW 06 CW 07 CW 08 CW 09 CW 10 CW 11 CW 12 CW 13 CW 14 Overconfidence and gut feeling produced up to 40% higher write-offs in stores not fully automated by Blue Yonder
24. Conclusion Seite 24 Exploiting “Big data” is the next big challenge to advance industry à Predictive Analytics ►  “In this war for customers, the ammunition is data — and lots of it […]” (G. Hawkings, Harvard Business Review, Sep. 2012) ►  Data-driven business instead of models and assumptions. ►  Peta-bytes of data and machine learning techniques allow statistically sound analyses The NeuroBayes machine learning solution ►  is based on 2nd generation neural network algorithms and Bayesian regularisation ►  Very fast and robust ►  Allows to optimise a wide range of business cases Blue Yonder: From “Big Science” to “Big Business” ►  Background in High Energy Physics: Crossing the bridge from understanding the behaviour of the fundamental particles at the origin of the universe to the “Big Bang” in sales forecast, risk analysis, churn management, etc. 28.03.2013 Bayesian Neural Networks in Business Applications
25. Thank you very much For your attention!
26. Disclaimer This Presentation (the Presentation) has been prepared by Blue Yonder GmbH & Co KG (collectively, with any officer, director, employee, advisor or agent of any of them, the Preparers) for the purpose of setting out certain confidential information in respect of Blue Yonder’s business activities and strategy. References to the “Presentation” includes any information which has been or may be supplied in writing or orally in connection with the Presentation or in connection with any further inquiries in respect of the Presentation. This Presentation is for the exclusive use of the recipients to whom it is addressed. This Presentation and the information contained herein is confidential. In addition to the terms of any confidentiality undertaking that a recipient may have entered into with Blue Yonder, by its acceptance of the Presentation, each recipient agrees that it will not, and it will procure that each of its agents, representatives, advisors, directors or employees (collectively, Representatives), will not, and will not permit any third party to, copy, reproduce or distribute to others this Presentation, in whole or in part, at any time without the prior written consent of Blue Yonder, and that it will keep confidential all information contained herein not already in the public domain and will use this Presentation for the sole purpose of setting out [familiarizing itself with] certain limited background information concerning Blue Yonder and its business strategy and activities. The foregoing confidentiality obligation shall be legally binding for the recipient infinitely. This Presentation is not intended to serve as basis for any investment decision. If a recipient has signed a confidentiality undertaking with Blue Yonder, this Presentation also constitutes Confidential Information for the purposes of such undertaking. While the information contained in this Presentation is believed to be accurate, the Preparers have not conducted any investigation with respect to such information. The Preparers expressly disclaim any and all liability for representations or warranties, expressed or implied, contained in, or for omissions from, this Presentation or any other written or oral communication transmitted to any interested party in connection with this Presentation so far as is permitted by law. In particular, but without limitation, no representation or warranty is given as to the achievement or reasonableness of, and no reliance should be placed on, any projections, estimates, forecasts, analyses or forward looking statements contained in this Presentation which involve by their nature a number of risks, uncertainties or assumptions that could cause actual results or events to differ materially from those expressed or implied in this Presentation. Only those particular representations and warranties which may be made in a definitive written agreement, when and if one is executed, and subject to such limitations and restrictions as may be specified in such agreement, shall have any legal effect. By its acceptance hereof, each recipient agrees that none of the Preparers nor any of their respective Representatives shall be liable for any direct, indirect or consequential loss or damages suffered by any person as a result of relying on any statement in or omission from this Presentation, along with other information furnished in connection therewith, and any such liability is expressly disclaimed. Except to the extent otherwise indicated, this Presentation presents information as of the date hereof. The delivery of this Presentation shall not, under any circumstances, create any implication that there will be no change in the affairs of Blue Yonder after the date hereof. In furnishing this Presentation, the Preparers reserve the right to amend or replace this Presentation at any time and undertake no obligation to update any of the information contained in the Presentation or to correct any inaccuracies that may become apparent. This Presentation shall remain the property of Blue Yonder. Blue Yonder may, at any time, request any recipient, or its Representatives, shall promptly deliver to Blue Yonder or, if directed in writing by Blue Yonder, destroy all confidential information relating to this Presentation received in written, electronic or other tangible form whatsoever, including without limitation all copies, reproductions, computer diskettes or written materials which contain such confidential information. At such time, all other notes, analyses or compilations constituting or containing confidential information in the recipient’s, or their Representatives’, possession shall be destroyed. Such destruction shall be certified to Blue Yonder by the recipient in writing. Neither the dissemination of this Presentation nor any part of its contents is to be taken as any form of commitment on the part of the Preparers or any of their respective affiliates to enter any contract or otherwise create any legally binding obligation or commitment. The Preparers expressly reserve the right, in their absolute discretion, without prior notice and without any liability to any recipient to terminate discussions with any recipient or any other parties. The distribution of this Presentation in certain jurisdictions may be restricted by law and, accordingly, recipients of this Presentation represent that they are able to receive this Presentation without contravention of any unfulfilled registration requirements or other legal restrictions in the jurisdiction in which they reside or conduct business.