Newton method

Newton method

It's a simple and easy introduction of Newton method for machine learning with python written by machine-learning beginner and in only Japanese.
This presentation includes how to calculate square root, cubic root and n-th root of real number by using Newton method, and introduces simply that how do we use it on machine learning.

859485c7b13b89623542576581512da5?s=128

Shoh-kudo

May 15, 2019
Tweet

Transcript

  1. 3.

    ۙࣅख๏ͷྫ ɾ࿈෼਺Λ༻͍Δ ɾ/FXUPO๏ xn+1 ← xn − f(xn ) f′(xn

    ) ɾςΠϥʔల։ 2 = Σn k=0 f n(a) n! = 1 + 1 2 * 1! − 1 4 * 2! + 3 8 * 3! − 3 * 5 16 * 4! + . . . ɾஞ࣍ܭࢉ๏ xn+1 ← 2 xn + xn 2  2 = 1 + 1 2 + 1 2 + 1 2 + 2 − 1
  2. 4.

    ۙࣅख๏ͷྫ ɾ࿈෼਺Λ༻͍Δ ɾ/FXUPO๏ 2 = 1 + 1 2 +

    1 2 + 1 2 + 2 − 1 xn+1 ← xn − f(xn ) f′(xn ) ɾςΠϥʔల։ 2 = Σn k=0 f n(a) n! = 1 + 1 2 * 1! − 1 4 * 2! + 3 8 * 3! − 3 * 5 16 * 4! + . . . ɾஞ࣍ܭࢉ๏ xn+1 ← 2 xn + xn 2 ॳظ஋Y ʹΑͬͯ ऩଋͷ଎౓͕มΘΔ https://www.procrasist.com/entry/sqrt2 
  3. 5.

    /FXUPO๏ಋग़ ઀ઢͷํఔ͔ࣜΒಋग़ͯ͠ΈΔʜ y − f(a) = f′(a)(x − a) y

    = 0 ͱ͓͍ͨ࣌ͷ x = a − f(a) f′(a) ͸ ઀ઢͱY࣠ͷަ఺Λࣔ͢ɽ 
  4. 6.

    /FXUPO๏ਤܗతղऍ ࠇ ੺ y = x2 − 2 y =

    f′(2)(x − 2) + f(2) = 4x − 6 x = a − f(a) f′(a) = 2 − 2 4 = 3 2 ͜͜ͷY͸ ߋ৽͞ΕͨYͷ஋ʹ͓͚Δ઀ઢͷํఔࣜΛ࡞Γɼಉ༷ͷॲཧΛߦ͏ͱʜ y = f′( 3 2 )(x − 3 2 ) + f( 3 2 ) = 3x − 17 4 → x = 17 12 = 1.416... y = f′( 17 12 )(x − 17 12 ) + f( 17 12 ) = 17 6 x − 577 144 → x = 577 408 = 1.414... → 2 
  5. 7.

    f(x) = x2 − a /FXUPO๏ೋ৐ࠜ f(x) = 0 ͷ࣌

    x = a ͱ͓͚͹ ͕ٻ·ΔͷͰ xn+1 ← xn − f(xn ) f′(xn ) = xn − x2 n − a 2xn ͔Βೋ৐ࠜΛۙࣅతʹܭࢉͰ͖Δ 
  6. 8.

    # Newton method # calculate square roots def sqrt(x): max_iter

    = 10 a = x if x >= 0: for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x /FXUPO๏ೋ৐ࠜ NBY@JUFS NBY@JUFS NBY@JUFSͷ਺஋ʹΑͬͯਫ਼౓Λ޲্Ͱ͖Δ 
  7. 9.

    /FXUPO๏ೋ৐ࠜ 2 = 1.41421356... 3 = 1.7320508... 5 = 2.2360679...

    10000 = 100 12345654321 = 111111 NBY@JUFSͷݶք ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ Ұ໷Ұ໷ʹਓݟࠒ ਓฒΈʹᇋΕ΍ ෋࢜ࢁ࿢Φ΢Ϝ໐͘ 
  8. 10.

    /FXUPO๏ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾNBY@JUFSʹ͢Δ ɾॳظ஋Λม͑Δ # Newton method # calculate square

    roots def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x Yͷ஋ΛΑΓখ͘͞Ͱ͖ͨΒ ߋ৽ͷճ਺͕গͳͯ͘ྑͦ͞͏ ͲΜͳYΛઃఆ͢΂͖͔ʁ 
  9. 11.

    /FXUPO๏ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾॳظ஋Λม͑Δ # Newton method # calculate square roots

    def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x ͲΜͳYΛઃఆ͢΂͖͔ʁ 
  10. 12.

    /FXUPO๏ೋ৐ࠜ ΋ͬͱਫ਼౓ྑ͘ܭࢉ͢Δʹ͸Ͳ͏ͨ͠Βྑ͍͔ ɾॳظ஋Λม͑Δ # Newton method # calculate square roots

    def sqrt(x): max_iter = 10 a = x for i in range(max_iter): x = x - (x**2 - a) / (2*x + 1e-7) return x ݁࿦͔Βݴ͏ͱ x0 = a 2 + a ͲΜͳYΛઃఆ͢΂͖͔ʁ ຊ຤స౗ʜ ࠓͷॴࢥ͍͔ͭͳ͍ͷͰ ܁Γฦ͠ͷճ਺Λ૿΍͢͜ͱͰਫ਼౓Λ্͛Δʜ 
  11. 14.

    f(x) = x3 − a ͱ͓͚͹ٻΊΔ͜ͱ͕Ͱ͖Δʁ xn+1 ← xn −

    f(xn ) f′(xn ) = xn − x3 n − a 3x2 n ͔Βࡾ৐ࠜΛܭࢉͯ͠ΈΔʜ /FXUPO๏ࡾ৐ࠜ 
  12. 15.

    # calculate cubic roots def cbrt(x): max_iter = 100 a

    = x # real number term for i in range(max_iter): x = x - (x**3 - a) / (3*x**2 + 1e-7 ) # imaginary number terms t1 = x*complex(-1/2, sqrt(3)/2) t2 = x*complex(-1/2, -sqrt(3)/2) return list([x, t1, t2]) Ͳ͏ͤͳͷͰɼx3 − a = 0 ͷࡾͭͷղΛग़ͤΔΑ͏ʹͯ͠Έͨɽ /FXUPO๏ࡾ৐ࠜ 
  13. 16.

    /FXUPO๏ࡾ৐ࠜ 3 2 = 1.25992104989... 3 3 = 1.44224957031... 3

    8 = 2 3 27 = 3 3 64 = 4 3 1000 = 10 3 1367631 = 111 3 1 = 1 
  14. 19.

    f(x) = xk − a ͱ͓͚͹ٻΊΔ͜ͱ͕Ͱ͖Δʁ xn+1 ← xn −

    f(xn ) f′(xn ) = xn − xk n − a kxk−1 n ͔ΒL৐ࠜΛܭࢉͯ͠ΈΔʜ /FXUPO๏L৐ࠜ 
  15. 20.

    # calculate n-th root def ndrt(x, n): # xͷn৐ࠜ max_iter

    = 100 pi = np.pi a = x ans_list = [] # real for i in range(max_iter): x = x - (x**n - a) / (n*x**(n-1) + 1e-7) # others for angle in range(n): t = x*complex(np.cos(2*angle*pi/n), np.sin(2*angle*pi/n)) ans_list.append(t) return ans_list /FXUPO๏L৐ࠜ 
  16. 21.

    /FXUPO๏L৐ࠜ 1 2 = 2 2 4 = 2 3

    8 = 2 4 16 = 2 5 32 = 2 6 64 = 2 7 128 = 2 8 256 = 2 9 512 = 2 10 1024 = 2 ͷ΂͖৐ͷL৐ࠜ 
  17. 22.

    ෳૉฏ໘্ʹඳըͯ͠Έͨʜ x10 = 1 ݸͷղ /FXUPO๏L৐ࠜ [(1+0j), (0.8090169943749475+0.5877852522924731j), (0.30901699437494745+0.9510565162951535j), (-0.30901699437494734+0.9510565162951536j),

    (-0.8090169943749473+0.5877852522924732j), (-1+1.2246467991473532e-16j),
 (-0.8090169943749475-0.587785252292473j),
 (-0.30901699437494756-0.9510565162951535j),
 (0.30901699437494723-0.9510565162951536j),
 (0.8090169943749473-0.5877852522924732j)] 
  18. 23.

    /FXUPO๏L৐ࠜ x10 = 1 ͷݸͷղ [(1+0j), (0.8090169943749475+0.5877852522924731j), (0.30901699437494745+0.9510565162951535j), (-0.30901699437494734+0.9510565162951536j), (-0.8090169943749473+0.5877852522924732j),

    (-1+1.2246467991473532e-16j),
 (-0.8090169943749475-0.587785252292473j),
 (-0.30901699437494756-0.9510565162951535j),
 (0.30901699437494723-0.9510565162951536j),
 (0.8090169943749473-0.5877852522924732j)] 
  19. 24.

    /FXUPO๏ػցֶश χϡʔτϯ๏ xn+1 ← xn − f(xn ) f′(xn )

    xn+1 = xn − η ∂f(x) ∂x ޯ഑߱Լ๏ Zͷ࣌ͷYͷ஋Λߋ৽͢Δ ZΛ࠷খ஋ʹ͚ۙͮΔͨΊʹ  Z`ͷ࣌ͷYΛߋ৽͢Δ Z`ͷ࣌ͷYͷ஋Λߋ৽͢Δ 
  20. 25.

    xn+1 ← xn − f(xn ) f′(xn ) Zͷ࣌ͷYͷ஋Λߋ৽͢Δ Z`ͷ࣌ͷYͷ஋Λߋ৽͢Δ

    /FXUPO๏ػցֶश xn+1 ← xn − f′(xn ) f′′(xn ) ݩͷؔ਺ͱҰ֊ඍ෼ͷൺͰͳ͘ Ұ֊ඍ෼ͱೋ֊ඍ෼ͷൺΛ࢖͏ ޯ഑߱Լ๏ͱಉ͡Α͏ʹ࢖͑Δ χϡʔτϯ๏ ଟม਺ܥͰ͸ϔοηߦྻΛ࢖͑Δʜ xn+1 ← xn − H−1 ∇f 
  21. 26.

    /FXUPO๏·ͱΊ ɾ/FXUPO๏ xn+1 ← xn − f(xn ) f′(xn )

    ɾL৐ࠜ L  ʜ ͷۙࣅ஋ # calculate n-th root def ndrt(x, n): # xͷn৐ࠜ max_iter = 100 pi = np.pi a = x ans_list = [] # real for i in range(max_iter): x = x - (x**n - a) / (n*x**(n-1) + 1e-7) # others for angle in range(n): t = x*complex(np.cos(2*angle*pi/n), np.sin(2*angle*pi/n)) ans_list.append(t) return ans_list ɾػցֶशʹԠ༻Ͱ͖Δ xn+1 ← xn − H−1 ∇f xn+1 ← xn − f′(xn ) f′′(xn ) Ұม਺ ଟม਺