Newton method

It's a simple and easy introduction of Newton method for machine learning with python written by machine-learning beginner and in only Japanese.
This presentation includes how to calculate square root, cubic root and n-th root of real number by using Newton method, and introduces simply that how do we use it on machine learning.

Transcript

1. /FXUPO๏ ޻౻঑ ྩ࿨ݩ೥ޒ݄ेޒ೔
2. ೋ৐ࠜΛٻΊ͍ͨ x2 = 2 x = 2 = 1.41421356... 
3. ۙࣅख๏ͷྫ ɾ࿈෼਺Λ༻͍Δ ɾ/FXUPO๏ xn+1 ← xn − f(xn ) f′(xn ) ɾςΠϥʔల։ 2 = Σn k=0 f n(a) n! = 1 + 1 2 * 1! − 1 4 * 2! + 3 8 * 3! − 3 * 5 16 * 4! + . . . ɾஞ࣍ܭࢉ๏ xn+1 ← 2 xn + xn 2  2 = 1 + 1 2 + 1 2 + 1 2 + 2 − 1
4. ۙࣅख๏ͷྫ ɾ࿈෼਺Λ༻͍Δ ɾ/FXUPO๏ 2 = 1 + 1 2 + 1 2 + 1 2 + 2 − 1 xn+1 ← xn − f(xn ) f′(xn ) ɾςΠϥʔల։ 2 = Σn k=0 f n(a) n! = 1 + 1 2 * 1! − 1 4 * 2! + 3 8 * 3! − 3 * 5 16 * 4! + . . . ɾஞ࣍ܭࢉ๏ xn+1 ← 2 xn + xn 2 ॳظ஋Y ʹΑͬͯ
   ऩଋͷ଎౓͕มΘΔ https://www.procrasist.com/entry/sqrt2 
5. /FXUPO๏
   ಋग़ ઀ઢͷํఔ͔ࣜΒಋग़ͯ͠ΈΔʜ y − f(a) = f′(a)(x − a) y = 0 ͱ͓͍ͨ࣌ͷ x = a − f(a) f′(a) ͸
   ઀ઢͱY࣠ͷަ఺Λࣔ͢ɽ 
6. /FXUPO๏
   ਤܗతղऍ ࠇ
   ੺ y = x2 − 2 y = f′(2)(x − 2) + f(2) = 4x − 6 x = a − f(a) f′(a) = 2 − 2 4 = 3 2 ͜͜ͷY͸ ߋ৽͞ΕͨYͷ஋ʹ͓͚Δ઀ઢͷํఔࣜΛ࡞Γɼಉ༷ͷॲཧΛߦ͏ͱʜ y = f′( 3 2 )(x − 3 2 ) + f( 3 2 ) = 3x − 17 4 → x = 17 12 = 1.416... y = f′( 17 12 )(x − 17 12 ) + f( 17 12 ) = 17 6 x − 577 144 → x = 577 408 = 1.414... → 2 
7. f(x) = x2 − a /FXUPO๏
   ೋ৐ࠜ f(x) = 0 ͷ࣌ x = a ͱ͓͚͹ ͕ٻ·ΔͷͰ xn+1 ← xn − f(xn ) f′(xn ) = xn − x2 n − a 2xn ͔Βೋ৐ࠜΛۙࣅతʹܭࢉͰ͖Δ 
8. # Newton method # calculate square roots def sqrt(x): max_iter = 10 a = x if x >= 0: for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x /FXUPO๏
   ೋ৐ࠜ NBY@JUFS
   
    NBY@JUFS
   
    NBY@JUFSͷ਺஋ʹΑͬͯਫ਼౓Λ޲্Ͱ͖Δ 
9. /FXUPO๏
   ೋ৐ࠜ 2 = 1.41421356... 3 = 1.7320508... 5 = 2.2360679... 10000 = 100 12345654321 = 111111 NBY@JUFS
   
   ͷݶք ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ Ұ໷Ұ໷ʹਓݟࠒ ਓฒΈʹᇋΕ΍ ෋࢜ࢁ࿢Φ΢Ϝ໐͘ 
10. /FXUPO๏
    ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾNBY@JUFS
    
    ʹ͢Δ ɾॳظ஋Λม͑Δ # Newton method # calculate square roots def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x Yͷ஋ΛΑΓখ͘͞Ͱ͖ͨΒ
    ߋ৽ͷճ਺͕গͳͯ͘ྑͦ͞͏ ͲΜͳYΛઃఆ͢΂͖͔ʁ 
11. /FXUPO๏
    ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾॳظ஋Λม͑Δ # Newton method # calculate square roots def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x ͲΜͳYΛઃఆ͢΂͖͔ʁ 
12. /FXUPO๏
    ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾॳظ஋Λม͑Δ # Newton method # calculate square roots def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x ݁࿦͔Βݴ͏ͱ x0 = a 2 + a ͲΜͳYΛઃఆ͢΂͖͔ʁ ຊ຤స౗ʜ ࠓͷॴࢥ͍͔ͭͳ͍ͷͰ
    ܁Γฦ͠ͷճ਺Λ૿΍͢͜ͱͰਫ਼౓Λ্͛Δʜ 
13. ࡾ৐ࠜ΋͍͚Δʜ /FXUPO๏
    ࡾ৐ࠜ 
14. f(x) = x3 − a ͱ͓͚͹ٻΊΔ͜ͱ͕Ͱ͖Δʁ xn+1 ← xn − f(xn ) f′(xn ) = xn − x3 n − a 3x2 n ͔Βࡾ৐ࠜΛܭࢉͯ͠ΈΔʜ /FXUPO๏
    ࡾ৐ࠜ 
15. # calculate cubic roots def cbrt(x): max_iter = 100 a = x # real number term for i in range(max_iter): x = x - (x**3 - a) / (3*x**2 + 1e-7 ) # imaginary number terms t1 = x*complex(-1/2, sqrt(3)/2) t2 = x*complex(-1/2, -sqrt(3)/2) return list([x, t1, t2]) Ͳ͏ͤͳͷͰɼx3 − a = 0 ͷࡾͭͷղΛग़ͤΔΑ͏ʹͯ͠Έͨɽ /FXUPO๏
    ࡾ৐ࠜ 
16. /FXUPO๏
    ࡾ৐ࠜ 3 2 = 1.25992104989... 3 3 = 1.44224957031... 3 8 = 2 3 27 = 3 3 64 = 4 3 1000 = 10 3 1367631 = 111 3 1 = 1 
17. ͷࡾͭͷࡾ৐ࠜ͸ෳૉฏ໘্ͷ୯Ґԁ্ʹ͋Γ
    ͦΕͧΕͷภ֯͸›ʹͳΔͷͰʜ ࢒Γͷೋͭͷղ͸ʜ t1 = cos 2 3 π + sin 2 3 π t2 = cos 4 3 π + sin 4 3 π /FXUPO๏
    ࡾ৐ࠜ 
18. L৐ࠜʹ֦ுͰ͖Δʜ /FXUPO๏
    L৐ࠜ 
19. f(x) = xk − a ͱ͓͚͹ٻΊΔ͜ͱ͕Ͱ͖Δʁ xn+1 ← xn − f(xn ) f′(xn ) = xn − xk n − a kxk−1 n ͔ΒL৐ࠜΛܭࢉͯ͠ΈΔʜ /FXUPO๏
    L৐ࠜ 
20. # calculate n-th root def ndrt(x, n): # xͷn৐ࠜ max_iter = 100 pi = np.pi a = x ans_list = [] # real for i in range(max_iter): x = x - (x**n - a) / (n*x**(n-1) + 1e-7) # others for angle in range(n): t = x*complex(np.cos(2*angle*pi/n), np.sin(2*angle*pi/n)) ans_list.append(t) return ans_list /FXUPO๏
    L৐ࠜ 
21. /FXUPO๏
    L৐ࠜ 1 2 = 2 2 4 = 2 3 8 = 2 4 16 = 2 5 32 = 2 6 64 = 2 7 128 = 2 8 256 = 2 9 512 = 2 10 1024 = 2 ͷ΂͖৐ͷL৐ࠜ 
22. ෳૉฏ໘্ʹඳըͯ͠Έͨʜ x10 = 1 ݸͷղ /FXUPO๏
    L৐ࠜ [(1+0j), (0.8090169943749475+0.5877852522924731j), (0.30901699437494745+0.9510565162951535j), (-0.30901699437494734+0.9510565162951536j), (-0.8090169943749473+0.5877852522924732j), (-1+1.2246467991473532e-16j),
 (-0.8090169943749475-0.587785252292473j),
 (-0.30901699437494756-0.9510565162951535j),
 (0.30901699437494723-0.9510565162951536j),
 (0.8090169943749473-0.5877852522924732j)] 
23. /FXUPO๏
    L৐ࠜ x10 = 1 ͷݸͷղ [(1+0j), (0.8090169943749475+0.5877852522924731j), (0.30901699437494745+0.9510565162951535j), (-0.30901699437494734+0.9510565162951536j), (-0.8090169943749473+0.5877852522924732j), (-1+1.2246467991473532e-16j),
 (-0.8090169943749475-0.587785252292473j),
 (-0.30901699437494756-0.9510565162951535j),
 (0.30901699437494723-0.9510565162951536j),
 (0.8090169943749473-0.5877852522924732j)] 
24. /FXUPO๏
    ػցֶश χϡʔτϯ๏ xn+1 ← xn − f(xn ) f′(xn ) xn+1 = xn − η ∂f(x) ∂x ޯ഑߱Լ๏ Z
    
    ͷ࣌ͷYͷ஋Λߋ৽͢Δ ZΛ࠷খ஋ʹ͚ۙͮΔͨΊʹ
    
    Z`
    
    ͷ࣌ͷYΛߋ৽͢Δ Z`
    
    ͷ࣌ͷYͷ஋Λߋ৽͢Δ 
25. xn+1 ← xn − f(xn ) f′(xn ) Z
    
    ͷ࣌ͷYͷ஋Λߋ৽͢Δ Z`
    
    ͷ࣌ͷYͷ஋Λߋ৽͢Δ /FXUPO๏
    ػցֶश xn+1 ← xn − f′(xn ) f′′(xn ) ݩͷؔ਺ͱҰ֊ඍ෼ͷൺͰͳ͘
    Ұ֊ඍ෼ͱೋ֊ඍ෼ͷൺΛ࢖͏ ޯ഑߱Լ๏ͱಉ͡Α͏ʹ࢖͑Δ χϡʔτϯ๏ ଟม਺ܥͰ͸ϔοηߦྻΛ࢖͑Δʜ xn+1 ← xn − H−1 ∇f 
26. /FXUPO๏
    ·ͱΊ ɾ/FXUPO๏ xn+1 ← xn − f(xn ) f′(xn ) ɾL৐ࠜ L
    
    
    
    ʜ ͷۙࣅ஋ # calculate n-th root def ndrt(x, n): # xͷn৐ࠜ max_iter = 100 pi = np.pi a = x ans_list = [] # real for i in range(max_iter): x = x - (x**n - a) / (n*x**(n-1) + 1e-7) # others for angle in range(n): t = x*complex(np.cos(2*angle*pi/n), np.sin(2*angle*pi/n)) ans_list.append(t) return ans_list ɾػցֶशʹԠ༻Ͱ͖Δ xn+1 ← xn − H−1 ∇f xn+1 ← xn − f′(xn ) f′′(xn ) Ұม਺ ଟม਺ 
27. /FXUPO๏
    ৐ࠜ IUUQTHJUIVCDPN4IPILVEPHJUTBNQMFLVEPCMPCNBTUFSNBUITRVBSF@SPPUJQZOC