# Newton method

It's a simple and easy introduction of Newton method for machine learning with python written by machine-learning beginner and in only Japanese.
This presentation includes how to calculate square root, cubic root and n-th root of real number by using Newton method, and introduces simply that how do we use it on machine learning.

May 15, 2019

## Transcript

1. 1.

2. 2.

3. 3.

### ۙࣅख๏ͷྫ ɾ࿈෼਺Λ༻͍Δ ɾ/FXUPO๏ xn+1 ← xn − f(xn ) f′(xn

) ɾςΠϥʔల։ 2 = Σn k=0 f n(a) n! = 1 + 1 2 * 1! − 1 4 * 2! + 3 8 * 3! − 3 * 5 16 * 4! + . . . ɾஞ࣍ܭࢉ๏ xn+1 ← 2 xn + xn 2  2 = 1 + 1 2 + 1 2 + 1 2 + 2 − 1
4. 4.

### ۙࣅख๏ͷྫ ɾ࿈෼਺Λ༻͍Δ ɾ/FXUPO๏ 2 = 1 + 1 2 +

1 2 + 1 2 + 2 − 1 xn+1 ← xn − f(xn ) f′(xn ) ɾςΠϥʔల։ 2 = Σn k=0 f n(a) n! = 1 + 1 2 * 1! − 1 4 * 2! + 3 8 * 3! − 3 * 5 16 * 4! + . . . ɾஞ࣍ܭࢉ๏ xn+1 ← 2 xn + xn 2 ॳظ஋Y ʹΑͬͯ ऩଋͷ଎౓͕มΘΔ https://www.procrasist.com/entry/sqrt2 
5. 5.

### /FXUPO๏ಋग़ ઀ઢͷํఔ͔ࣜΒಋग़ͯ͠ΈΔʜ y − f(a) = f′(a)(x − a) y

= 0 ͱ͓͍ͨ࣌ͷ x = a − f(a) f′(a) ͸ ઀ઢͱY࣠ͷަ఺Λࣔ͢ɽ 
6. 6.

### /FXUPO๏ਤܗతղऍ ࠇ ੺ y = x2 − 2 y =

f′(2)(x − 2) + f(2) = 4x − 6 x = a − f(a) f′(a) = 2 − 2 4 = 3 2 ͜͜ͷY͸ ߋ৽͞ΕͨYͷ஋ʹ͓͚Δ઀ઢͷํఔࣜΛ࡞Γɼಉ༷ͷॲཧΛߦ͏ͱʜ y = f′( 3 2 )(x − 3 2 ) + f( 3 2 ) = 3x − 17 4 → x = 17 12 = 1.416... y = f′( 17 12 )(x − 17 12 ) + f( 17 12 ) = 17 6 x − 577 144 → x = 577 408 = 1.414... → 2 
7. 7.

### f(x) = x2 − a /FXUPO๏ೋ৐ࠜ f(x) = 0 ͷ࣌

x = a ͱ͓͚͹ ͕ٻ·ΔͷͰ xn+1 ← xn − f(xn ) f′(xn ) = xn − x2 n − a 2xn ͔Βೋ৐ࠜΛۙࣅతʹܭࢉͰ͖Δ 
8. 8.

### # Newton method # calculate square roots def sqrt(x): max_iter

= 10 a = x if x >= 0: for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x /FXUPO๏ೋ৐ࠜ NBY@JUFS NBY@JUFS NBY@JUFSͷ਺஋ʹΑͬͯਫ਼౓Λ޲্Ͱ͖Δ 
9. 9.

### /FXUPO๏ೋ৐ࠜ 2 = 1.41421356... 3 = 1.7320508... 5 = 2.2360679...

10000 = 100 12345654321 = 111111 NBY@JUFSͷݶք ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ Ұ໷Ұ໷ʹਓݟࠒ ਓฒΈʹᇋΕ΍ ෋࢜ࢁ࿢Φ΢Ϝ໐͘ 
10. 10.

### /FXUPO๏ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾNBY@JUFSʹ͢Δ ɾॳظ஋Λม͑Δ # Newton method # calculate square

roots def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x Yͷ஋ΛΑΓখ͘͞Ͱ͖ͨΒ ߋ৽ͷճ਺͕গͳͯ͘ྑͦ͞͏ ͲΜͳYΛઃఆ͢΂͖͔ʁ 
11. 11.

### /FXUPO๏ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾॳظ஋Λม͑Δ # Newton method # calculate square roots

def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x ͲΜͳYΛઃఆ͢΂͖͔ʁ 
12. 12.

### /FXUPO๏ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾॳظ஋Λม͑Δ # Newton method # calculate square roots

def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x ݁࿦͔Βݴ͏ͱ x0 = a 2 + a ͲΜͳYΛઃఆ͢΂͖͔ʁ ຊ຤స౗ʜ ࠓͷॴࢥ͍͔ͭͳ͍ͷͰ ܁Γฦ͠ͷճ਺Λ૿΍͢͜ͱͰਫ਼౓Λ্͛Δʜ 
13. 13.

14. 14.

### f(x) = x3 − a ͱ͓͚͹ٻΊΔ͜ͱ͕Ͱ͖Δʁ xn+1 ← xn −

f(xn ) f′(xn ) = xn − x3 n − a 3x2 n ͔Βࡾ৐ࠜΛܭࢉͯ͠ΈΔʜ /FXUPO๏ࡾ৐ࠜ 
15. 15.

### # calculate cubic roots def cbrt(x): max_iter = 100 a

= x # real number term for i in range(max_iter): x = x - (x**3 - a) / (3*x**2 + 1e-7 ) # imaginary number terms t1 = x*complex(-1/2, sqrt(3)/2) t2 = x*complex(-1/2, -sqrt(3)/2) return list([x, t1, t2]) Ͳ͏ͤͳͷͰɼx3 − a = 0 ͷࡾͭͷղΛग़ͤΔΑ͏ʹͯ͠Έͨɽ /FXUPO๏ࡾ৐ࠜ 
16. 16.

### /FXUPO๏ࡾ৐ࠜ 3 2 = 1.25992104989... 3 3 = 1.44224957031... 3

8 = 2 3 27 = 3 3 64 = 4 3 1000 = 10 3 1367631 = 111 3 1 = 1 
17. 17.

### ͷࡾͭͷࡾ৐ࠜ͸ෳૉฏ໘্ͷ୯Ґԁ্ʹ͋Γ ͦΕͧΕͷภ֯͸ʹͳΔͷͰʜ ࢒Γͷೋͭͷղ͸ʜ t1 = cos 2 3 π +

sin 2 3 π t2 = cos 4 3 π + sin 4 3 π /FXUPO๏ࡾ৐ࠜ 
18. 18.

19. 19.

### f(x) = xk − a ͱ͓͚͹ٻΊΔ͜ͱ͕Ͱ͖Δʁ xn+1 ← xn −

f(xn ) f′(xn ) = xn − xk n − a kxk−1 n ͔ΒL৐ࠜΛܭࢉͯ͠ΈΔʜ /FXUPO๏L৐ࠜ 
20. 20.

### # calculate n-th root def ndrt(x, n): # xͷn৐ࠜ max_iter

= 100 pi = np.pi a = x ans_list = [] # real for i in range(max_iter): x = x - (x**n - a) / (n*x**(n-1) + 1e-7) # others for angle in range(n): t = x*complex(np.cos(2*angle*pi/n), np.sin(2*angle*pi/n)) ans_list.append(t) return ans_list /FXUPO๏L৐ࠜ 
21. 21.

### /FXUPO๏L৐ࠜ 1 2 = 2 2 4 = 2 3

8 = 2 4 16 = 2 5 32 = 2 6 64 = 2 7 128 = 2 8 256 = 2 9 512 = 2 10 1024 = 2 ͷ΂͖৐ͷL৐ࠜ 
22. 22.

### ෳૉฏ໘্ʹඳըͯ͠Έͨʜ x10 = 1 ݸͷղ /FXUPO๏L৐ࠜ [(1+0j), (0.8090169943749475+0.5877852522924731j), (0.30901699437494745+0.9510565162951535j), (-0.30901699437494734+0.9510565162951536j),

(-0.8090169943749473+0.5877852522924732j), (-1+1.2246467991473532e-16j),  (-0.8090169943749475-0.587785252292473j),  (-0.30901699437494756-0.9510565162951535j),  (0.30901699437494723-0.9510565162951536j),  (0.8090169943749473-0.5877852522924732j)] 
23. 23.

### /FXUPO๏L৐ࠜ x10 = 1 ͷݸͷղ [(1+0j), (0.8090169943749475+0.5877852522924731j), (0.30901699437494745+0.9510565162951535j), (-0.30901699437494734+0.9510565162951536j), (-0.8090169943749473+0.5877852522924732j),

(-1+1.2246467991473532e-16j),  (-0.8090169943749475-0.587785252292473j),  (-0.30901699437494756-0.9510565162951535j),  (0.30901699437494723-0.9510565162951536j),  (0.8090169943749473-0.5877852522924732j)] 
24. 24.

### /FXUPO๏ػցֶश χϡʔτϯ๏ xn+1 ← xn − f(xn ) f′(xn )

xn+1 = xn − η ∂f(x) ∂x ޯ഑߱Լ๏ Zͷ࣌ͷYͷ஋Λߋ৽͢Δ ZΛ࠷খ஋ʹ͚ۙͮΔͨΊʹ  Z`ͷ࣌ͷYΛߋ৽͢Δ Z`ͷ࣌ͷYͷ஋Λߋ৽͢Δ 
25. 25.

### xn+1 ← xn − f(xn ) f′(xn ) Zͷ࣌ͷYͷ஋Λߋ৽͢Δ Z`ͷ࣌ͷYͷ஋Λߋ৽͢Δ

/FXUPO๏ػցֶश xn+1 ← xn − f′(xn ) f′′(xn ) ݩͷؔ਺ͱҰ֊ඍ෼ͷൺͰͳ͘ Ұ֊ඍ෼ͱೋ֊ඍ෼ͷൺΛ࢖͏ ޯ഑߱Լ๏ͱಉ͡Α͏ʹ࢖͑Δ χϡʔτϯ๏ ଟม਺ܥͰ͸ϔοηߦྻΛ࢖͑Δʜ xn+1 ← xn − H−1 ∇f 
26. 26.

### /FXUPO๏·ͱΊ ɾ/FXUPO๏ xn+1 ← xn − f(xn ) f′(xn )

ɾL৐ࠜ L  ʜ ͷۙࣅ஋ # calculate n-th root def ndrt(x, n): # xͷn৐ࠜ max_iter = 100 pi = np.pi a = x ans_list = [] # real for i in range(max_iter): x = x - (x**n - a) / (n*x**(n-1) + 1e-7) # others for angle in range(n): t = x*complex(np.cos(2*angle*pi/n), np.sin(2*angle*pi/n)) ans_list.append(t) return ans_list ɾػցֶशʹԠ༻Ͱ͖Δ xn+1 ← xn − H−1 ∇f xn+1 ← xn − f′(xn ) f′′(xn ) Ұม਺ ଟม਺ 
27. 27.