Build your own Neural Network, with PHP! @ IPC Berlin 2019

Build your own Neural Network, with PHP! @ IPC Berlin 2019

Curious about all the hype around Machine Learning and Artificial Intelligence? Heard of "Neural Networks" and "Deep Learning" but confused about what it really means?

In this talk, you'll see what Artificial Neural Networks (ANN) look like and how they can "learn". And along the way, you'll discover how you can build your own ANN, with PHP of course!

475493db5519a08372c9367d879ef9ad?s=128

Vítor Brandão

June 05, 2019
Tweet

Transcript

  1. None
  2. None
  3. None
  4. None
  5. Disclaimers

  6. Hint: he's not a "real" doctor

  7. None
  8. None
  9. AI? Machine Learning? Neural Networks? PHP?!

  10. None
  11. None
  12. h"ps://twi"er.com/mhbergen/status/966876932313763841

  13. "I thought that it might be be.er to use a

    familiar language to learn something unfamiliar like ML." h"ps://tech.kartenmacherei.de/why-i-used-php-to-teach-myself- machine-learning-10ed90af8996
  14. h"ps://www.offerzen.com/blog/how-to-build-a-content-based-recommender-system-for-your-product

  15. None
  16. h"ps://www.theatlan.c.com/magazine/archive/2013/03/the-robot-will-see-you-now/309216/

  17. h"ps://www.geek.com/tech/ai-beats-human-lawyers-at-their-own-game-1732154/

  18. h"ps://blog.floydhub.com/turning-design-mockups-into-code-with-deep-learning/

  19. h"ps://www.cnet.com/news/machine-learning-algorithm-ai-nicolas-cage-movies-indiana-jones/

  20. None
  21. Artificial Neural Networks

  22. Coming up next · Why are they called "Neural Networks"?

    · How do they "learn"? · How can I write one (with PHP)? · What the hell is "Deep Learning"?
  23. The Human Brain

  24. None
  25. ≈86,000,000,000 neurons

  26. Neuron

  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. 1943: First Ar(ficial Neuron (McCulloch and Pitts) 1957: Perceptron (Rosenblatt)

  35. None
  36. None
  37. None
  38. None
  39. Learnable Parameters

  40. Weights { } · Represents the synaptic strenght (influence of

    one neuron on another). Bias { } · Ensures the output best fits the incoming signal (allows the activation function to be shifted to the left or right).
  41. None
  42. None
  43. Neuron input Neuron output

  44. Activation Function · Models the firing rate of the neuron

    (frequency of the spikes along the axon). · Goal: add non-linearity into the network.
  45. Activation Function: Sigmoid

  46. None
  47. None
  48. None
  49. Supervised Machine Learning

  50. Supervised Machine Learning Given a set of inputs , learn

    a function mapping to some known output ; So that we can accurately predict a new output from unseen inputs.
  51. Adapted from h-ps://www.coursera.org/learn/introduc9on-tensorflow/lecture/PoOzi/a-primer-in-machine-learning

  52. Adapted from h-ps://www.coursera.org/learn/introduc9on-tensorflow/lecture/PoOzi/a-primer-in-machine-learning

  53. None
  54. None
  55. None
  56. None
  57. None
  58. None
  59. None
  60. "When the activation function is non- linear, then a two-layer

    neural network can be proven to be an universal func-on approximator"
  61. Training approach · Start with random weights. · Predict based

    on input data . · Compare with target output . · Adjust network parameters ( and ). · Repeat until we are "close enough"
  62. None
  63. Learning the XOR

  64. XOR x1 x2 y 0 0 0 0 1 1

    1 0 1 1 1 0
  65. None
  66. None
  67. None
  68. None
  69. None
  70. None
  71. Build Your Own Neural Network h"ps:/ /github.com/noiselabs/byonn

  72. // › src/NeuralNetwork.php class NeuralNetwork { public function train(array $inputs,

    array $targets) { } public function predict(array $input) { } }
  73. // › src/NeuralNetwork.php class NeuralNetwork { public function train(array $inputs,

    array $targets) { } public function predict(array $input) { } }
  74. // › src/NeuralNetwork.php class NeuralNetwork { public function train(array $inputs,

    array $targets) { } public function predict(array $input) { } }
  75. // › examples/xor.php $inputs = [ [0, 0], // 0

    [0, 1], // 1 [1, 0], // 1 [1, 1] // 0 ]; $targets = [0, 1, 1, 0]; $neuralNetwork = new NeuralNetwork();
  76. // › examples/xor.php $inputs = [ [0, 0], // 0

    [0, 1], // 1 [1, 0], // 1 [1, 1] // 0 ]; $targets = [0, 1, 1, 0]; $neuralNetwork = new NeuralNetwork();
  77. Network Topology

  78. // › src/NeuralNetwork.php class NeuralNetwork { const INPUTS = 2;

    const HIDDEN_NEURONS = 2; const LAYERS = 2; const OUTPUTS = 1; // ... }
  79. Parameters

  80. // › src/Parameters.php class Parameters { /** @var array Weights

    */ public $w = []; /** @var array Biases */ public $b = []; /** @var array The input of the activation function */ public $z = []; /** @var array The neuron output, after applying an activation function */ public $a = []; }
  81. class NeuralNetwork { public function __construct() { $this->p = new

    Parameters(); } }
  82. None
  83. Hidden Layer

  84. Output Layer

  85. › h#ps:/ /github.com/markrogoyski/math-php

  86. use MathPHP\LinearAlgebra\Matrix; use MathPHP\LinearAlgebra\MatrixFactory; $matrix = [ [1, 2, 3],

    [4, 5, 6], [7, 8, 9], ]; $A = MatrixFactory::create($matrix); $B = MatrixFactory::create($matrix); $A+B = $A->add($B); $AB = $A->multiply($B); $A∘B = $A->hadamardProduct($B); $C = $A->map(function($x) { return $x * 2; });
  87. use MathPHP\LinearAlgebra\Matrix; use MathPHP\LinearAlgebra\MatrixFactory; $matrix = [ [1, 2, 3],

    [4, 5, 6], [7, 8, 9], ]; $A = MatrixFactory::create($matrix); $B = MatrixFactory::create($matrix); $A+B = $A->add($B); $AB = $A->multiply($B); $A∘B = $A->hadamardProduct($B); $C = $A->map(function($x) { return $x * 2; });
  88. use MathPHP\LinearAlgebra\Matrix; use MathPHP\LinearAlgebra\MatrixFactory; $matrix = [ [1, 2, 3],

    [4, 5, 6], [7, 8, 9], ]; $A = MatrixFactory::create($matrix); $B = MatrixFactory::create($matrix); $A+B = $A->add($B); $AB = $A->multiply($B); $A∘B = $A->hadamardProduct($B); $C = $A->map(function($x) { return $x * 2; });
  89. use MathPHP\LinearAlgebra\Matrix; use MathPHP\LinearAlgebra\MatrixFactory; $matrix = [ [1, 2, 3],

    [4, 5, 6], [7, 8, 9], ]; $A = MatrixFactory::create($matrix); $B = MatrixFactory::create($matrix); $A+B = $A->add($B); $AB = $A->multiply($B); $A∘B = $A->hadamardProduct($B); $C = $A->map(function($x) { return $x * 2; });
  90. Vectorization?

  91. › h#ps:/ /github.com/phpsci/phpsci

  92. Activation Function

  93. None
  94. None
  95. // › src/Activation/Sigmoid.php namespace Activation; use MathPHP\LinearAlgebra\Matrix; class Sigmoid {

    public function compute(Matrix $m): Matrix { return $values->map(function($value) { return $this->sigmoid($value); }); } private function sigmoid($t) { return 1 / (1 + exp(-$t)); } }
  96. // › src/Activation/Sigmoid.php namespace Activation; use MathPHP\LinearAlgebra\Matrix; class Sigmoid {

    public function compute(Matrix $m): Matrix { return $values->map(function($value) { return $this->sigmoid($value); }); } private function sigmoid($t) { return 1 / (1 + exp(-$t)); } }
  97. // › src/Activation/Sigmoid.php namespace Activation; use MathPHP\LinearAlgebra\Matrix; class Sigmoid {

    public function compute(Matrix $m): Matrix { return $values->map(function($value) { return $this->sigmoid($value); }); } private function sigmoid($t) { return 1 / (1 + exp(-$t)); } }
  98. // › src/Activation/Sigmoid.php namespace Activation; use MathPHP\LinearAlgebra\Matrix; class Sigmoid {

    public function compute(Matrix $m): Matrix { return $values->map(function($value) { return $this->sigmoid($value); }); } private function sigmoid($t) { return 1 / (1 + exp(-$t)); } }
  99. class NeuralNetwork { public function __construct( ActivationFunction\Sigmoid $activationFunction ) {

    $this->activationFunction = $activationFunction; $this->p = new Parameters(); } }
  100. None
  101. Algorithm for training initialise_weights_and_biases() # 1. while i < n_iterations

    or error > max_error: for m in training_examples: forward_pass() # 2. compute_cost() # 3. backpropagation() # 4. adjust_weights_and_biases() # 5.
  102. Algorithm for training initialise_weights_and_biases() # 1. while i < n_iterations

    or error > max_error: for m in training_examples: forward_pass() # 2. compute_cost() # 3. backpropagation() # 4. adjust_weights_and_biases() # 5.
  103. Algorithm for training initialise_weights_and_biases() # 1. while i < n_iterations

    or error > max_error: for m in training_examples: forward_pass() # 2. compute_cost() # 3. backpropagation() # 4. adjust_weights_and_biases() # 5.
  104. Algorithm for training initialise_weights_and_biases() # 1. while i < n_iterations

    or error > max_error: for m in training_examples: forward_pass() # 2. compute_cost() # 3. backpropagation() # 4. adjust_weights_and_biases() # 5.
  105. › Training 1. Initialise Parameters

  106. class NeuralNetwork { // ... private function initializeParameters(): void {

    // Hidden layer $this->p->b[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, 1); $this->p->w[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, self::INPUTS) $this->p->w[1]->map(function($v) { return random_int(1, 1000) / 1000; }); // Output layer $this->p->b[2] = MatrixFactory::zero(self::OUTPUTS, 1); $this->p->w[2] = MatrixFactory::zero(self::OUTPUTS, self::HIDDEN_NEURONS); $this->p->w[2]->map(function($v) { return random_int(1, 1000) / 1000; }); }
  107. class NeuralNetwork { // ... private function initializeParameters(): void {

    // Hidden layer $this->p->b[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, 1); $this->p->w[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, self::INPUTS) $this->p->w[1]->map(function($v) { return random_int(1, 1000) / 1000; }); // Output layer $this->p->b[2] = MatrixFactory::zero(self::OUTPUTS, 1); $this->p->w[2] = MatrixFactory::zero(self::OUTPUTS, self::HIDDEN_NEURONS); $this->p->w[2]->map(function($v) { return random_int(1, 1000) / 1000; }); }
  108. class NeuralNetwork { // ... private function initializeParameters(): void {

    // Hidden layer $this->p->b[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, 1); $this->p->w[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, self::INPUTS) $this->p->w[1]->map(function($v) { return random_int(1, 1000) / 1000; }); // Output layer $this->p->b[2] = MatrixFactory::zero(self::OUTPUTS, 1); $this->p->w[2] = MatrixFactory::zero(self::OUTPUTS, self::HIDDEN_NEURONS); $this->p->w[2]->map(function($v) { return random_int(1, 1000) / 1000; }); }
  109. class NeuralNetwork { // ... private function initializeParameters(): void {

    // Hidden layer $this->p->b[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, 1); $this->p->w[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, self::INPUTS) $this->p->w[1]->map(function($v) { return random_int(1, 1000) / 1000; }); // Output layer $this->p->b[2] = MatrixFactory::zero(self::OUTPUTS, 1); $this->p->w[2] = MatrixFactory::zero(self::OUTPUTS, self::HIDDEN_NEURONS); $this->p->w[2]->map(function($v) { return random_int(1, 1000) / 1000; }); }
  110. class NeuralNetwork { // ... private function initializeParameters(): void {

    // Hidden layer $this->p->b[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, 1); $this->p->w[1] = MatrixFactory::zero(self::HIDDEN_NEURONS, self::INPUTS) $this->p->w[1]->map(function($v) { return random_int(1, 1000) / 1000; }); // Output layer $this->p->b[2] = MatrixFactory::zero(self::OUTPUTS, 1); $this->p->w[2] = MatrixFactory::zero(self::OUTPUTS, self::HIDDEN_NEURONS); $this->p->w[2]->map(function($v) { return random_int(1, 1000) / 1000; }); }
  111. class NeuralNetwork { public function __construct( ActivationFunction\Sigmoid $activationFunction ) {

    $this->activationFunction = $activationFunction; $this->p = new Parameters(); $this->initializeParameters(); } }
  112. None
  113. class NeuralNetwork { // ... public function train(array $inputs, array

    $targets) { $inputs = $this->toMatrix($inputs); $targets = $this->toMatrix($targets); // ... } }
  114. class NeuralNetwork { // ... public function train(array $inputs, array

    $targets) { $inputs = $this->toMatrix($inputs); $targets = $this->toMatrix($targets); // ... } }
  115. class NeuralNetwork { public function train(array $inputs, array $targets) {

    // ... $maxTrainingIterations = 20000; $maxError = 0.001; $iteration = 0; $error = INF; }
  116. class NeuralNetwork { public function train(array $inputs, array $targets) {

    // ... $maxTrainingIterations = 20000; $maxError = 0.01; $iteration = 0; $error = INF; while ($iteration < $maxTrainingIterations && $error > $maxError) { $iteration++; $costs = []; for ($i = 0; $i < count($inputs); $i++) { // 1. doForwardPropagation() // 2. $costs = computeCost() // 3. doBackPropagation() // 4. updateParameters() } $error = array_sum($costs) / count($costs); } }
  117. › Training 2. Forward propagation

  118. Hidden Layer: Output Layer:

  119. Hidden Layer: Output Layer:

  120. Hidden Layer: Output Layer:

  121. class NeuralNetwork { private function doForwardPropagation(Matrix $input): array { //

    To ease calculations do: "Layer-0" activations = inputs $this->p->a[0] = $input; for ($l = 1; $l <= self::LAYERS; $l++) { // Z[l] = W[l]·A[l-1] + b[l] $this->p->z[$l] = $this->p->w[$l] ->multiply($this->p->a[$l-1])) ->add($this->p->b[$l]); $this->p->a[$l] = $this->activation->compute($this->p->z[$l]); } // Prediction: return $this->toArray($this->p->a[self::LAYERS]); }
  122. class NeuralNetwork { private function doForwardPropagation(Matrix $input): array { //

    To ease calculations do: "Layer-0" activations = inputs $this->p->a[0] = $input; for ($l = 1; $l <= self::LAYERS; $l++) { // Z[l] = W[l]·A[l-1] + b[l] $this->p->z[$l] = $this->p->w[$l] ->multiply($this->p->a[$l-1])) ->add($this->p->b[$l]); $this->p->a[$l] = $this->activation->compute($this->p->z[$l]); } // Prediction: return $this->toArray($this->p->a[self::LAYERS]); }
  123. class NeuralNetwork { private function doForwardPropagation(Matrix $input): array { //

    To ease calculations do: "Layer-0" activations = inputs $this->p->a[0] = $input; for ($l = 1; $l <= self::LAYERS; $l++) { // Z[l] = W[l]·A[l-1] + b[l] $this->p->z[$l] = $this->p->w[$l] ->multiply($this->p->a[$l-1])) ->add($this->p->b[$l]); $this->p->a[$l] = $this->activation->compute($this->p->z[$l]); } // Prediction: return $this->toArray($this->p->a[self::LAYERS]); }
  124. class NeuralNetwork { private function doForwardPropagation(Matrix $input): array { //

    To ease calculations do: "Layer-0" activations = inputs $this->p->a[0] = $input; for ($l = 1; $l <= self::LAYERS; $l++) { // Z[l] = W[l]·A[l-1] + b[l] $this->p->z[$l] = $this->p->w[$l] ->multiply($this->p->a[$l-1])) ->add($this->p->b[$l]); $this->p->a[$l] = $this->activation->compute($this->p->z[$l]); } // Prediction: return $this->toArray($this->p->a[self::LAYERS]); }
  125. class NeuralNetwork { private function doForwardPropagation(Matrix $input): array { //

    To ease calculations do: "Layer-0" activations = inputs $this->p->a[0] = $input; for ($l = 1; $l <= self::LAYERS; $l++) { // Z[l] = W[l]·A[l-1] + b[l] $this->p->z[$l] = $this->p->w[$l] ->multiply($this->p->a[$l-1])) ->add($this->p->b[$l]); $this->p->a[$l] = $this->activation->compute($this->p->z[$l]); } // Prediction: return $this->toArray($this->p->a[self::LAYERS]); }
  126. class NeuralNetwork { private function doForwardPropagation(Matrix $input): array { //

    To ease calculations do: "Layer-0" activations = inputs $this->p->a[0] = $input; for ($l = 1; $l <= self::LAYERS; $l++) { // Z[l] = W[l]·A[l-1] + b[l] $this->p->z[$l] = $this->p->w[$l] ->multiply($this->p->a[$l-1])) ->add($this->p->b[$l]); $this->p->a[$l] = $this->activation->compute($this->p->z[$l]); } // Prediction: return $this->toArray($this->p->a[self::LAYERS]); }
  127. class NeuralNetwork { public function train(array $inputs, array $targets) {

    // ... while ($iteration < $maxTrainingIterations && $error > $maxError) { $costs = []; for ($i = 0; $i < count($inputs); $i++) { $prediction = $this->doForwardPropagation($inputs[$i]); } } }
  128. None
  129. › Training 3. Compute cost

  130. Cost function: Mean Squared Error

  131. // › src/CostFunction/MeanSquaredError.php namespace CostFunction; use MathPHP\LinearAlgebra\Vector; class MeanSquaredError {

    public function compute(Matrix $prediction, Matrix $target): float { return ($target[0] - $prediction[0]) ** 2; } }
  132. class NeuralNetwork { public function __construct( ActivationFunction\Sigmoid $activationFunction, CostFunction\MeanSquaredError $costFunction

    ) { $this->activationFunction = $activationFunction; $this->costFunction = $costFunction; $this->p = new Parameters(); }
  133. class NeuralNetwork { // ... private function computeCost(Matrix $prediction, Matrix

    $target): float { return $this->costFunction->compute($prediction, $target); }
  134. class NeuralNetwork { public function train(array $inputs, array $targets) {

    // ... while ($iteration < $maxTrainingIterations && $error > $maxError) { $costs = []; for ($i = 0; $i < count($inputs); $i++) { $prediction = $this->doForwardPropagation($inputs[$i]); $costs[$i] = $this->computeCost($prediction, $targets[$i]); } } }
  135. None
  136. Goal: minimise the wrongness

  137. › Training 4. Backpropagation

  138. Backpropagation · A supervised learning method for multilayer feed-forward networks.

    · Backward propaga,on of errors using gradient descent (calculates the gradient of the error function with respect to the neural network's weights).
  139. Adapted from h-ps://www.khanacademy.org/math/mul:variable-calculus/mul:variable-deriva:ves/par:al-deriva:ve-and-gradient-ar:cles/ a/introduc:on-to-par:al-deriva:ves

  140. Output Layer: Hidden Layer:

  141. Output Layer: Hidden Layer:

  142. // › src/Parameters.php class Parameters { // ... /** *

    Gradient of the cost with respect to `w`. * * @var array|Matrix[] */ public $dw = []; /** * Gradient of the cost with respect to `b`. * * @var array|Matrix[] */ public $db = []; }
  143. Mean Squared Error derivative

  144. Mean Squared Error derivative

  145. namespace CostFunction; use MathPHP\LinearAlgebra\Matrix; use MathPHP\LinearAlgebra\Vector; class MeanSquaredError { public

    function compute(Matrix $prediction, Matrix $target): float { return ($target[0] - $prediction[0]) ** 2; } public function differentiate(Matrix $prediction, Matrix $target): Matrix { // ∂L = 2·(Y -Ŷ) return ($target->subtract($prediction))->scalarMultiply(2); } }
  146. Sigmoid derivative

  147. Sigmoid derivative

  148. namespace Activation; use MathPHP\LinearAlgebra\Matrix; class Sigmoid { public function compute(Matrix

    $m): Matrix { return $values->map(function($value) { return $this->sigmoid($value); }); } public function differentiate(Matrix $values): Matrix { // ∂σ = σ·(1 - σ) return $values->map(function($value) { $computedValue = $this->sigmoid(value); return $computedValue * (1 - $computedValue); }); } private function sigmoid($t) { return 1 / (1 + exp(-$t)); } }
  149. class NeuralNetwork { private function doBackPropagation(Matrix $target): void { $l

    = self::LAYERS; // Output Layer $da[$l] = $this->costFunction->differentiate($this->p->a[$l], $target); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; // Hidden Layer(s) for ($l = (self::LAYERS - 1); $l >= 1; $l--) { $da[$l] = $this->p->w[$l+1]->transpose()->multiply($dz[$l+1]); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; } }
  150. Output Layer: Hidden Layer:

  151. class NeuralNetwork { private function doBackPropagation(Matrix $target): void { $l

    = self::LAYERS; // Output Layer $da[$l] = $this->costFunction->differentiate($this->p->a[$l], $target); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; // Hidden Layer(s) for ($l = (self::LAYERS - 1); $l >= 1; $l--) { $da[$l] = $this->p->w[$l+1]->transpose()->multiply($dz[$l+1]); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; } }
  152. class NeuralNetwork { private function doBackPropagation(Matrix $target): void { $l

    = self::LAYERS; // Output Layer $da[$l] = $this->costFunction->differentiate($this->p->a[$l], $target); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; // Hidden Layer(s) for ($l = (self::LAYERS - 1); $l >= 1; $l--) { $da[$l] = $this->p->w[$l+1]->transpose()->multiply($dz[$l+1]); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; } }
  153. class NeuralNetwork { private function doBackPropagation(Matrix $target): void { $l

    = self::LAYERS; // Output Layer $da[$l] = $this->costFunction->differentiate($this->p->a[$l], $target); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; // Hidden Layer(s) for ($l = (self::LAYERS - 1); $l >= 1; $l--) { $da[$l] = $this->p->w[$l+1]->transpose()->multiply($dz[$l+1]); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; } }
  154. Output Layer: Hidden Layer:

  155. class NeuralNetwork { private function doBackPropagation(Matrix $target): void { $l

    = self::LAYERS; // Output Layer $da[$l] = $this->costFunction->differentiate($this->p->a[$l], $target); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; // Hidden Layer(s) for ($l = (self::LAYERS - 1); $l >= 1; $l--) { $da[$l] = $this->p->w[$l+1]->transpose()->multiply($dz[$l+1]); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; } }
  156. class NeuralNetwork { private function doBackPropagation(Matrix $target): void { $l

    = self::LAYERS; // Output Layer $da[$l] = $this->costFunction->differentiate($this->p->a[$l], $target); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; // Hidden Layer(s) for ($l = (self::LAYERS - 1); $l >= 1; $l--) { $da[$l] = $this->p->w[$l+1]->transpose()->multiply($dz[$l+1]); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; } }
  157. class NeuralNetwork { private function doBackPropagation(Matrix $target): void { $l

    = self::LAYERS; // Output Layer $da[$l] = $this->costFunction->differentiate($this->p->a[$l], $target); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; // Hidden Layer(s) for ($l = (self::LAYERS - 1); $l >= 1; $l--) { $da[$l] = $this->p->w[$l+1]->transpose()->multiply($dz[$l+1]); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; } }
  158. class NeuralNetwork { private function doBackPropagation(Matrix $target): void { $l

    = self::LAYERS; // Output Layer $da[$l] = $this->costFunction->differentiate($this->p->a[$l], $target); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; // Hidden Layer(s) for ($l = (self::LAYERS - 1); $l >= 1; $l--) { $da[$l] = $this->p->w[$l+1]->transpose()->multiply($dz[$l+1]); $dz[$l] = $da[$l]->hadamardProduct( $this->activations[$l]->differentiate($this->p->z[$l])); $this->p->dw[$l] = $dz[$l]->multiply($this->p->a[$l-1]->transpose()); $this->p->db[$l] = $dz[$l]; } }
  159. class NeuralNetwork { // ... public function train(array $inputs, array

    $targets) { // ... while ($iteration < $maxTrainingIterations && $error > $maxError) { $iteration++; $costs = []; for ($i = 0; $i < count($inputs); $i++) { $prediction = $this->doForwardPropagation($inputs[$i]); $costs[$i] = $this->computeCost($prediction, $targets[$i]); $this->doBackPropagation($targets[$i]); } } }
  160. › Training 5. Update Parameters

  161. Gradient Descent

  162. Gradient Descent

  163. Gradient Descent

  164. class NeuralNetwork { public function __construct( ActivationFunction\Sigmoid $activationFunction, CostFunction\MeanSquaredError $costFunction,

    float $learningRate = 0.1 ) { $this->activationFunction = $activationFunction; $this->costFunction = $costFunction; $this->learningRate = $learningRate; $this->p = new Parameters(); }
  165. class NeuralNetwork { // ... private function updateParameters(): void {

    for ($l = 1; $l <= self::LAYERS; $l++) { $this->p->w[$l] = $this->p->w[$l]->subtract( $this->p->dw[$l]->scalarMultiply($this->learningRate)); $this->p->b[$l] = $this->p->b[$l]->subtract( $this->p->db[$l]->scalarMultiply($this->learningRate)); } }
  166. class NeuralNetwork { // ... private function updateParameters(): void {

    for ($l = 1; $l <= self::LAYERS; $l++) { $this->p->w[$l] = $this->p->w[$l]->subtract( $this->p->dw[$l]->scalarMultiply($this->learningRate)); $this->p->b[$l] = $this->p->b[$l]->subtract( $this->p->db[$l]->scalarMultiply($this->learningRate)); } }
  167. class NeuralNetwork { // ... private function updateParameters(): void {

    for ($l = 1; $l <= self::LAYERS; $l++) { $this->p->w[$l] = $this->p->w[$l]->subtract( $this->p->dw[$l]->scalarMultiply($this->learningRate)); $this->p->b[$l] = $this->p->b[$l]->subtract( $this->p->db[$l]->scalarMultiply($this->learningRate)); } }
  168. class NeuralNetwork { // ... private function updateParameters(): void {

    for ($l = 1; $l <= self::LAYERS; $l++) { $this->p->w[$l] = $this->p->w[$l]->subtract( $this->p->dw[$l]->scalarMultiply($this->learningRate)); $this->p->b[$l] = $this->p->b[$l]->subtract( $this->p->db[$l]->scalarMultiply($this->learningRate)); } }
  169. class NeuralNetwork { // ... public function train(array $inputs, array

    $targets) { // ... while ($iteration < $maxTrainingIterations && $error > $maxError) { $iteration++; $costs = []; for ($i = 0; $i < count($inputs); $i++) { $prediction = $this->doForwardPropagation($inputs[$i]); $costs[$i] = $this->computeCost($prediction, $targets[$i]); $this->doBackPropagation($targets[$i]); $this->updateParameters(); } } }
  170. class NeuralNetwork { // ... public function train(array $inputs, array

    $targets) { // ... while ($iteration < $maxTrainingIterations && $error > $maxError) { $iteration++; $costs = []; for ($i = 0; $i < count($inputs); $i++) { $prediction = $this->doForwardPropagation($inputs[$i]); $costs[$i] = $this->computeCost($prediction, $targets[$i]); $this->doBackPropagation($targets[$i]); $this->updateParameters(); } $error = array_sum($costs) / count($costs); } }
  171. class NeuralNetwork { public function train(array $inputs, array $targets) {

    $inputs = $this->toMatrix($inputs); $targets = $this->toMatrix($targets); $maxTrainingIterations = 20000; $maxError = 0.001; $iteration = 0; $error = INF; while ($iteration < $maxTrainingIterations && $error > $maxError) { $iteration++; $costs = []; for ($i = 0; $i < count($inputs); $i++) { $prediction = $this->doForwardPropagation($inputs[$i]); $costs[$i] = $this->computeCost($prediction, $targets[$i]); $this->doBackPropagation($targets[$i]); $this->updateParameters(); } $error = array_sum($costs) / count($costs); } }
  172. None
  173. Algorithm for training initialise_weights_and_biases() # 1. while i < n_iterations

    or error > max_error: for m in training_examples: forward_pass() # 2. compute_cost() # 3. backpropagation() # 4. adjust_weights_and_biases() # 5.
  174. None
  175. $ php examples/xor.php Training for 20000 epochs or until the

    cost falls below 0.001... * Epoch: 1000, Error: 0.229587 * Epoch: 2000, Error: 0.062260 * Epoch: 3000, Error: 0.009333 * Epoch: 4000, Error: 0.004388 * Epoch: 5000, Error: 0.002788 * Epoch: 6000, Error: 0.002020 * Epoch: 7000, Error: 0.001575 * Epoch: 8000, Error: 0.001286 * Epoch: 9000, Error: 0.001084 * Epoch: 9500, Error: 0.001005 Predicting... * Input: [0, 0] Prediction: 0.0341 Target: 0 * Input: [0, 1] Prediction: 0.9697 Target: 1 * Input: [1, 0] Prediction: 0.9698 Target: 1 * Input: [1, 1] Prediction: 0.0317 Target: 0
  176. $ php examples/xor.php Training for 20000 epochs or until the

    cost falls below 0.001... * Epoch: 1000, Error: 0.229587 * Epoch: 2000, Error: 0.062260 * Epoch: 3000, Error: 0.009333 * Epoch: 4000, Error: 0.004388 * Epoch: 5000, Error: 0.002788 * Epoch: 6000, Error: 0.002020 * Epoch: 7000, Error: 0.001575 * Epoch: 8000, Error: 0.001286 * Epoch: 9000, Error: 0.001084 * Epoch: 9500, Error: 0.001005 Predicting... * Input: [0, 0] Prediction: 0.0341 Target: 0 * Input: [0, 1] Prediction: 0.9697 Target: 1 * Input: [1, 0] Prediction: 0.9698 Target: 1 * Input: [1, 1] Prediction: 0.0317 Target: 0
  177. $ php examples/xor.php Training for 20000 epochs or until the

    cost falls below 0.001... * Epoch: 1000, Error: 0.229587 * Epoch: 2000, Error: 0.062260 * Epoch: 3000, Error: 0.009333 * Epoch: 4000, Error: 0.004388 * Epoch: 5000, Error: 0.002788 * Epoch: 6000, Error: 0.002020 * Epoch: 7000, Error: 0.001575 * Epoch: 8000, Error: 0.001286 * Epoch: 9000, Error: 0.001084 * Epoch: 9500, Error: 0.001005 Predicting... * Input: [0, 0] Prediction: 0.0341 Target: 0 * Input: [0, 1] Prediction: 0.9697 Target: 1 * Input: [1, 0] Prediction: 0.9698 Target: 1 * Input: [1, 1] Prediction: 0.0317 Target: 0
  178. $ php examples/xor.php Training for 20000 epochs or until the

    cost falls below 0.001... * Epoch: 1000, Error: 0.229587 * Epoch: 2000, Error: 0.062260 * Epoch: 3000, Error: 0.009333 * Epoch: 4000, Error: 0.004388 * Epoch: 5000, Error: 0.002788 * Epoch: 6000, Error: 0.002020 * Epoch: 7000, Error: 0.001575 * Epoch: 8000, Error: 0.001286 * Epoch: 9000, Error: 0.001084 * Epoch: 9500, Error: 0.001005 Predicting... * Input: [0, 0] Prediction: 0.0341 Target: 0 * Input: [0, 1] Prediction: 0.9697 Target: 1 * Input: [1, 0] Prediction: 0.9698 Target: 1 * Input: [1, 1] Prediction: 0.0317 Target: 0
  179. None
  180. // › examples/xor.php $inputs = [ [0, 0], // 0

    [0, 1], // 1 [1, 0], // 1 [1, 1] // 0 ]; $targets = [0, 1, 1, 0]; $neuralNetwork = new NeuralNetwork( new Activation\Sigmoid(), new CostFunction\MeanSquaredError() ); $neuralNetwork->train($inputs, $targets); echo "Predicting...\n"; echo sprintf("* Input: [0,0] Prediction: %.2f Target: 0\n", $neuralNetwork->predict([0,0]])); echo sprintf("* Input: [0,1] Prediction: %.2f Target: 1\n", $neuralNetwork->predict([0,1]])); echo sprintf("* Input: [1,0] Prediction: %.2f Target: 1\n", $neuralNetwork->predict([1,0]])); echo sprintf("* Input: [1,1] Prediction: %.2f Target: 0\n", $neuralNetwork->predict([1,1]]));
  181. class NeuralNetwork { // ... public function predict(array $input): array

    { return $this->doForwardPropagation($this->toMatrix($input)); }
  182. $ php examples/xor.php Training for 20000 epochs or until the

    cost falls below 0.001... * Epoch: 1000, Error: 0.229587 * Epoch: 2000, Error: 0.062260 * Epoch: 3000, Error: 0.009333 * Epoch: 4000, Error: 0.004388 * Epoch: 5000, Error: 0.002788 * Epoch: 6000, Error: 0.002020 * Epoch: 7000, Error: 0.001575 * Epoch: 8000, Error: 0.001286 * Epoch: 9000, Error: 0.001084 * Epoch: 9500, Error: 0.001005 Predicting... * Input: [0, 0] Prediction: 0.0341 Target: 0 * Input: [0, 1] Prediction: 0.9697 Target: 1 * Input: [1, 0] Prediction: 0.9698 Target: 1 * Input: [1, 1] Prediction: 0.0317 Target: 0
  183. $ php examples/xor.php Training for 20000 epochs or until the

    cost falls below 0.001... * Epoch: 1000, Error: 0.229587 * Epoch: 2000, Error: 0.062260 * Epoch: 3000, Error: 0.009333 * Epoch: 4000, Error: 0.004388 * Epoch: 5000, Error: 0.002788 * Epoch: 6000, Error: 0.002020 * Epoch: 7000, Error: 0.001575 * Epoch: 8000, Error: 0.001286 * Epoch: 9000, Error: 0.001084 * Epoch: 9500, Error: 0.001005 Predicting... * Input: [0, 0] Prediction: 0.0341 Target: 0 * Input: [0, 1] Prediction: 0.9697 Target: 1 * Input: [1, 0] Prediction: 0.9698 Target: 1 * Input: [1, 1] Prediction: 0.0317 Target: 0
  184. None
  185. But the fun doesn't stop here! · Pick a more

    creative example · Experiment with different activation and cost functions, and learning rates (send me a PR?) · Try a a different topology (more hidden layers?) · Compete at Kaggle!
  186. Artificial Neural Networks Recap · Inspired by the human brain.

    · Today we saw a feed-forward network (there are other architectures). · Uses Supervised Machine Learning (learn from examples). · Error is minimised using Backpropagation and Gradient Descent. · Can approximate any function*.
  187. None
  188. None
  189. None
  190. Getting deeper...

  191. None
  192. "Deep Learning"

  193. None
  194. Neural Networks and Machine Learning resources · https://en.wikipedia.org/wiki/Artificial_neural_network · https://www.coursera.org/learn/machine-learning

    · https://www.coursera.org/specializations/deep-learning · https://www.cs.toronto.edu/~hinton/coursera_lectures.html · https://developers.google.com/machine-learning/crash-course/ · http://neuralnetworksanddeeplearning.com/
  195. None
  196. h"ps://youtu.be/fXOsFF95i5

  197. None
  198. The End

  199. Vielen Dank! › h#p:/ /bit.ly/2KCD1dx-byonn-ipc19