ウェブアプリケーションフレームワークの向こう側 / The other side of Web Application Frameworks

はてなインターン2015 インフラ講義

Transcript

1. ΢ΣϒΞϓϦέʔγϣϯ
   ϑϨʔϜϫʔΫͷ޲͜͏ଆ ͸ͯͳΠϯλʔϯ
   
   Πϯϑϥߨٛ dΠϯλʔϯͷ͜Ζʹ஌Γ͔ͨͬͨΠϯϑϥͷ࿩d
2. JEZ@VVLJ γεςϜϓϥοτϑΥʔϜ෦ ͸ͯͳΠϯλʔϯ
   
3. ߨٛͷ͸͡Ίʹ wޙ൒աఔʹ޲͚ͯɺ͸ͯͳͷຊ൪γεςϜʹ͍ͭ ͯཧղΛਂΊ͍ͯͩ͘͞
   w͸ͯͳʹґଘ͠ͳ͍ͳΔ΂͘Ұൠతͳ࿩Λ৺͕ ͚͍ͯ·͢
   w਺ࣈҎ֎͸ެ։ͯ͠Α͍಺༰Ͱ͢
   w՝୊͸͋Γ·ͤΜɻؾܰʹௌ͍͍ͯͩ͘͞
5. ΢ΣϒΞϓϦέʔγϣϯ ϑϨʔϜϫʔΫ
6. 63-
   3PVUFS $POUSPMMFS .PEFM .Z42- #SPXTFS HTTP Request HTTP Response 8"'Ͱίʔυॻ͍ͯΔͱ͖ͷΠϝʔδ
7. ΢ΣϒΞϓϦέʔγϣϯ ͷϒϥοΫϘοΫε
8. ʮϞσϧͷ෦෼͸͍͢͝Πϯϑϥ͕ ͦͷઌʹ͋Δʯ
9. ͍͢͝Πϯϑϥ
10. ͸ͯͳϒϩά
11. .BDLFSFM
13. 8"' Πϯϑϥ
14. ΈΜͳ͕ݺΜͰΔ
    Πϯϑϥͬͯ
15. ΞϓϦέʔγϣϯΤϯδχΞ ͔ΒΈͨΠϯϑϥ wΞϓϦέʔγϣϯࣗମ͸ͲͷΑ͏ʹͯ͠ಈ͍͍ͯΔͷ͔
    w8FCαʔό
    1SFGPSL
    σʔλϕʔε઀ଓͷҡ࣋
    
    wΞϓϦέʔγϣϯ͸ͲͷΑ͏ʹͯ͠σʔλϕʔεʹΞΫ ηε͍ͯ͠Δͷ͔
    wεϨʔϒ γϟʔσΟϯά
    wϦΫΤετ͸ΞϓϦέʔγϣϯʹͲͷΑ͏ʹͯ͠౸ୡ͢ Δͷ͔
    wϦόʔεϓϩΩγ
16. 8FCγεςϜͷجຊ
17. 8FCγεςϜͷجຊ w)551ϦΫΤετΛड͚ͯϨεϙϯεΛฦ͢
    wෳ਺ͷϢʔβ͕ಉ࣌ʹϦΫΤετΛ౤͛͏Δ
    wϨΠςϯγͱεϧʔϓοτΛಉ࣌ʹୡ੒͢Δ ඞཁ͕͋Δ
    wͲΜͳʹෳࡶͳ8FCγεςϜͰ΋جຊ͸ಉ͡
18. ΞϓϦέʔγϣϯ͸Ͳͷ Α͏ʹಈ͍͍ͯΔͷ͔
19. ΞϓϦέʔγϣϯαʔό w8FCαʔϏεʹ͓͍ͯγεςϜͷத৺ͱͳΔαʔό
    wΞϓϦέʔγϣϯϩδοΫ͸͜͜Ͱ࣮ߦ͞ΕΔ
    wҎԼͷॲཧΛ࣮ߦ͢Δ8FCαʔό
    wϦΫΤετड৴
    w)551ϦΫΤετͷղੳ
    wσʔλϕʔεΞΫηεΛؚΉΞϓϦέʔγϣϯϩδοΫͷ ࣮ߦ
    w)551Ϩεϙϯεͷߏங
    wϨεϙϯεૹ৴
20. ΞϓϦέʔγϣϯαʔόͱݴޠ w ੲ͸ωοτϫʔΫ΍)551ͷϔομॲཧ͚ͩ
    "QBDIF
    $ݴޠ
    Ͱ΍Β͍ͤͯͨ
    w $(*
    ͱ͔
    NPE@IPHF
    ͱ͔ͷಾٕज़Ͱແཧ΍Γ$ ݴޠͷ8FCαʔόͷ্ʹ1FSMͳͲΛಈ͔͢
    w ࠓͰ͸ΞϓϦέʔγϣϯίʔυͱಉ͡ݴޠͷ8FCαʔ όΛ࢖͏
    w 1FSMͳΒ4UBSMFU1MBDL
    4DBMBͳΒ+FUUZͳͲ
21. Ͳ͏΍ͬͯωοτϫʔΫ௨৴ ͢Δͷ͔ w ωοτϫʔΫ௨৴͸جຊతʹ04Χʔωϧ͕΍Δ
    w -JOVYͷ৔߹γεςϜίʔϧΛ࢖ͬͯ04Χʔωϧͷ ػೳΛωοτϫʔΫ௨৴ػೳΛݺͼग़͢
    w TPDLFU 
    CJOE 
    MJTUFO 
    BDDFQU 
    w جຊ͸ແݶϧʔϓͭͭ͠ɺΧʔωϧ͔Βσʔλड৴ Πϕϯτ͕͖ͨΒϦΫΤετॲཧʹҠΔ
22. ΫϥΠΞϯτ αʔό socket(2) bind(2) listen(2) accept(2) connect(2) write(2) (ϒϩοΫ͢Δ) socket(2) “GET / HTTP1.1\nHost: TCP 3-way handshake “HTTP/1.1 200 OK\n…” read(2) (accept͔Β໭Δ) read(2) (ϒϩοΫ͢Δ) (ϒϩοΫ͢Δ) (ϒϩοΫ͢Δ) (read͔Β໭Δ) ϦΫΤετॲཧ write(2) (read͔Β໭Δ) read(2) (ϒϩοΫ͢Δ) close(2) লུ
23. 8FCαʔόΛෳ਺ͷ
    ΫϥΠΞϯτʹରԠ͢Δ
24. worker! process worker! process worker! process worker! process parent! process Client accept loop accept loop accept loop accept loop fork fork fork fork Client Client Client 1SFGPSLํࣜ
25. thread thread thread thread Client accept loop accept loop accept loop accept loop Client Client Client εϨουϓʔϧํࣜ Process
26. Client Client Client Client main! thread select loop data! base non-blocking I/O Πϕϯτۦಈํࣜ
27. w 1FSM͸1SFGPSL 4DBMB +BWB ͸εϨουϓʔϧ ͕ଟ͍
    w Ϛϧνϓϩηε 1SFGPSL ͔ϚϧνεϨου εϨο υϓʔϧ ͔Ͱάϩʔόϧม਺ͷѻ͍͕͔ΘΔ
    w ϚϧνεϨουͰάϩʔόϧม਺ͷϩοΫڝ߹ͯ͠ ٧·ΔͳͲ
    w ٯʹϚϧνϓϩηεͳΒෳ਺ͷίϯςΩετ͔Βڞ ௨ͷΦϒδΣΫτΛࢀর͢Δ͜ͱ͸͋·Γͳ͍ ʢϑΝΠϧͳͲ04ͷϦιʔε͸ผʣ
28. ΞϓϦέʔγϣϯ͸
    ͲͷΑ͏ʹͯ͠σʔλϕʔε ʹΞΫηε͍ͯ͠Δͷ͔
29. σʔλϕʔεͱͷ઀ଓ
30. ͍ͭ઀ଓ։࢝͢Δͷ͔ w 1FSM
    w %#*DPOOFDUΛݺΜͩͱ͖
    w େ఍ɺ࣮ࡍʹϦΫΤετ͕ͱΜͰ͖ͯσʔλϕʔ εΞΫηε͢Δஈ֊Ͱ઀ଓ͢Δ
    w 4DBMB
    w ΞϓϦέʔγϣϯͷىಈ࣌ʹ઀ଓࡁΈͷ%#Φϒ δΣΫτ͕࡞੒͞ΕΔ
31. ઀ଓͷҡ࣋ʁ w σʔλϕʔεͷ઀ଓίετ
    w 5$1ͷ
    XBZ
    ϋϯυγΣʔΫ
    w %#૚ͷϋϯυγΣʔΫ
    w %#૚ͷೝূ
    w %#ଆͷ઀ଓ༻ϓϩηεεϨουͷੜ੒ίετ
    w ઀ଓ਺͕ଟ͍ͱ%#αʔόͷෛՙ͕ߴ͍
    w ઀ଓࡁΈͷιέοτΛ࢖͍·Θ͍ͨ͠
32. ઀ଓͷӬଓԽ w ౎౓઀ଓ
    w ϦΫΤετॲཧதͷΈ઀ଓΛҡ࣋͢Δ
    w ECIΦϒδΣΫτΛίϯςΩετΦϒδΣΫτͳ ͲʹΩϟογϡ͢Ε͹Α͍
    w ৗ࣌઀ଓ
    w ECIΦϒδΣΫτΛάϩʔόϧม਺ͳͲʹΩϟο γϡ͢Δ
    w ίωΫγϣϯϓʔϧΛ࢖͏
33. ઀ଓͷӬଓԽͷϋϚΓϙΠϯτ w ઀ଓ͸͞·͟·ͳཁҼͰ్தͰ੾ΕΔ͜ͱ͕͋Δ
    w %#͕͓ͪͯϑΣΠϧΦʔόͨ͠
    w ωοτϫʔΫো֐
    w ECIΦϒδΣΫτ͸উखʹ࠶઀ଓͯ͘͠Εͳ͍
    w 4DPQF$POUBJOFS%#*
    %#*Y)BOEMFS
    Λ ࢖͏
    w ϩʔυόϥϯα͔ΒॏΈௐ੔Ͱ%#୆αʔϏεΞ΢ τͰ͖ͳ͍
    
    ແཧ΍Γ੾Δ
34. 8FCαʔϏε૚ߏ੒
35. Proxy App Database request response request response transaction w ϦόʔεϓϩΩγ
    /HJOY
    "QBDIF
    w ΞϓϦέʔγϣϯ
    1FSM
    4DBMB
    w σʔλϕʔε
    .Z42-
    1PTUHSF42-
36. ͳͥෳ਺ͷαʔόΛ࢖͏ ͷ͔
37. ෳ਺ͷαʔόΛ࢖͏ཧ༝ wγεςϜͷՄ༻ੑͷ୲อ
    wෛՙ෼ࢄ
    wૄ݁߹Խ
38. Մ༻ੑͷ୲อ wαʔό͸෺ཧతʹյΕͨΓաෛՙ΍ΦϖϨʔγϣϯϛε Ͱఀࢭ
    wελϯόΠػʹॲཧΛҾ͖ܧ͙ʢϑΣΠϧΦʔόεΠο νΦʔόʣ͕୅දత
    wΞΫςΟϒɾελϯόΠؒͰૄ௨؂ࢹ
    w૬ख͕མͪͯͨΒࣗ෼͕ΞΫςΟϒʹͳΔ
    w7*1 Ծ૝*1 Ͱ઀ଓઌΛมߋ͢Δ
    wϛυϧ΢ΣΞଆͰϞʔυ੾Γସ͑ͳͲ͕ඞཁͳ͜ͱ΋
    wLFFQBMJWFE
    
    )FBSUCFBU
    ͳͲͷπʔϧ
39. ෛՙ෼ࢄ wϩʔυόϥϯαΛલஈʹஔ͍ͯৼΓ෼͚
    wಉ͡໾ׂͷαʔόΛฒ΂Δ
    wϩʔυόϥϯαʹ͸7*1ͳͲͷΤϯυϙΠϯτ͕ ͍͍ͭͯΔ
    wΞϓϦέʔγϣϯ͔Β͸ͦ͜Λࢀর͢Δ͚ͩ
    wઐ༻ͷϩʔυόϥϯα
    -74
    )"1SPYZ
    wϦόʔεϓϩΩγͰ୅༻͢Δ͜ͱ΋͋Δ
    w৑௕Խ΋݉Ͷඋ͍͑ͯΔ
40. ૄ݁߹Խ wಉ͡ϗετʹෳ਺ͷϛυϧ΢ΣΞΛಉډͤ͞Δ ͜ͱ΋Ͱ͖Δ
    w࣮ࡍʹ͸
    w໰୊ͷ੾Γ෼͚ͷ͠΍͢͞
    wෛՙ෼ࢄ
    w໾ׂ͕ҟͳΔ͔Βͱ͍ͬͯɺશ෦෼͚ͨ΄͏͕ Α͍Θ͚Ͱ͸ͳ͍
    wαʔό਺͕૿͑͗͢Δ৔߹͸ಉډ΋ݕ౼
41. Proxy App Database request response request response transaction w ϦόʔεϓϩΩγ
    /HJOY
    "QBDIF
    w ΞϓϦέʔγϣϯ
    1FSM
    4DBMB
    w σʔλϕʔε
    .Z42-
    1PTUHSF42-
42. ϦόʔεϓϩΩγ
43. w άϩʔόϧ͔ΒϦΫΤετΛ௚઀ड͚෇͚Δαʔό
    w ϦΫΤετͷ಺༰ʹ͋Θͤͯ
    w ޙଓͷαʔόʹϦΫΤετͨ͠Γ
    w ΞΫηε੍ޚͨ͠Γ
    w ੩తϑΝΠϧΛฦͨ͠Γ͢Δ
    w ੍ޚՄೳϙΠϯτΛ૿΍ͯ͠ɺύϑΥʔϚϯε޲্ ͤͨ͞ΓɺγεςϜશମͷӡ༻ޮ཰Λ্͛Δ Proxy App Database request response request response transaction
44. Proxy bot༻app request response user༻app UserAgent: ~Googlebot w CPU༻ͱVTFS༻Λ6"Ͱ൑ఆͯ͠ৼΓ෼͚Δ
    w ো֐ͷӨڹൣғΛݶఆ͍ͨ͠
    w CPUͱVTFSͰνϡʔχϯά಺༰Λม͑ͨΓ͢Δ
    w QSFGPSLܕͰ͸ಛʹޮ͖΍͍͢
45. ΞϓϦέʔγϣϯ
46. w ͍͍ͩͨطग़
    w άϩʔόϧͳϦΫΤετ͸ΞϓϦαʔόͰड͚ͯ͸ ͍͚ͳ͍ͱ͍͏Θ͚Ͱ͸ͳ͍
    w 4UBSMFU΍+FUUZΑΓɺ/HJOY"QBDIFͷ΄͏͕ ֎ʹެ։͢Δ8FCαʔόͱͯ͠͸҆৺
    w มͳϔομ৯Θ͞ΕΔͱ͔
    w খ͞ͳαʔϏεͩͱΞϓϦαʔόͱϦόʔεϓϩΩ γΛಉډͤ͞Δ͜ͱ΋͋Δ Proxy App Database request response request response transaction
47. σʔλϕʔε
48. w ͸ͯͳϒϩάن໛ͳΒϚελ̍୆Ͱࡹ͚Δ
    w ςʔϒϧσʔλͷΑ͘ࢀর͞ΕΔ෦෼͕ϝϞϦʹ৐ͬ ͍ͯͯɺద੾ʹΠϯσοΫεΛ͸Δ
    w ελϯόΠϚελͷ༻ҙ͠.)"ͷΑ͏ͳ࢓૊ΈͰϑΣ ΠϧΦʔόͤ͞Δ
    w ϚελͷεέʔϧΞοϓʹݶք͕དྷͨ
    w SFBE͕ݶք
    
    ࢀর༻εϨʔϒΛ༻ҙ
    w XSJUF͕ݶք
    
    ਫฏ෼ׂ Proxy App Database request response request response transaction
49. slave Load! Balancer master slave slave App ࢀরΫΤϦͷΈ ߋ৽ΫΤϦத৺ SELECT INSERT/UPDATE ࢀর༻εϨʔϒ
50. master A App ਫฏ෼ׂ γϟʔσΟϯά master B master C ϢʔβIDΛ 3Ͱׂͬͨ৒༨͕0 ৒༨͕1 ৒༨͕2
51. ϫϯϥϯΫ্ͷ
    γεςϜΛߏங͢Δ
52. Ωϟογϡ
53. w $16ͷΩϟογϡϝϞϦ - -ΩϟογϡͳͲʣ
    w 04ͷϖʔδΩϟογϡ
    w .Z42- *OOP%# ͷόοϑΝϓʔϧ
    w ෼ࢄΩϟογϡαʔό
    NFNDBDIFE
    w Ωϟογϡػೳ͖ͭϦόʔεϓϩΩγ
    7BSOJTI 4RVJE
54. Proxy App Database Squid/ Varnisn memcached
55. ඈͼಓ۩ͱͯ͠ͷ ,74/P42-
56. w 3FEJT
    NFNDBDIFE
    .POHP%#ͳͲ
    w 3%#.4ΑΓ͸΍͍σʔλϕʔε
    w ετϨʔδͱͯ͠ͷ҆৺ײ͸ྼΔ͔΋͠Εͳ͍
    w ো֐࣌ͷσʔλ੔߹ੑ͕ෆ҆
    w ϦϨʔγϣφϧϞσϧͷΑ͏ͳཧ࿦తʹཱ֬͞Εͨσʔ λϞσϧͰ͸ͳ͍
    w ύϑΥʔϚϯε޲্ͷͨΊʹิॿతʹ࢖༻͢Δ͜ͱ͕ଟ͍
    w .BDLFSFMͰ͸࣌ܥྻσʔλϕʔεͱͯ͠
    (SBQIJUF
    Λ ࢖͍ͬͯΔ
57. ඇಉظॲཧ
58. w ϝʔϧ΍5XJUUFS΁ͷ౤ߘͳͲ͸ɺ֎෦γεςϜ͕མ ͪͯҾ͖ͣΒΕͯ͠·͏ͷΛ๷͙
    w QSFGPSLͷϓϩηεͷେ෦෼͕ো֐தͷ֎෦γε ςϜΛୟ͍͍ͯΔͱɺࣗ෼ͷαʔϏε΋མͪΔ
    w ֎෦γεςϜʹґଘͨ͠Γɺ࣮ߦ͕஗͍ॲཧΛ͋ͱͰ ࣮ߦ͍ͨ͠
    
    ඇಉظॲཧ
    w ͸ͯͳͰ͸δϣϒΩϡʔΛ࢖͏
    w ઐ༻ͷ.Z42-ʹδϣϒΛ౤ೖ
    w ผαʔόͰಈ͘ϫʔΧʔ͕.Z42-ʹϙʔϦϯά
59. Proxy App db-master workerdb worker db-slave
60. όονॲཧ
61. w ຖேϢʔβʹϝʔϧΛ഑৴͍ͨ͠ɺఆظతʹϗοτΤ ϯτϦΛߋ৽͍ͨ͠ͳͲ
    w ઐ༻ͷόοναʔόͰDSPOʹεΫϦϓτΛஔ͘
    w όονॲཧ͸σʔλϕʔεͷ಺༰Λूܭ͢ΔΑ͏ͳॏ ͍ΫΤϦ͕౤͛ΒΕΔ͜ͱ͕ଟ͍
    w όονॲཧ༻ͷ%#εϨʔϒΛ༻ҙͯ͠ɺͦͬͪʹ ޲͚Δ
62. ͸ͯͳϒϩά
63. .BDLFSFM
64. ʮࡸ૿΍ͤ͹ʁʯ
65. ؆୯ʹ૿΍ͯ͠ղܾͰ͖ΔΑ͏ ͳΞʔΩςΫνϟΛߟ͑Δͷ͕ ͨͷ͍͠
66. ࢀߟจݙ
68. wʮ੒௕͢ΔαʔϏεʯ
    w IUUQCMPHIBSVLBTBOKQFOUSZ
    wʮΫϥ΢υͱΦϯϓϨʯ
    w IUUQCMPHIBSVLBTBOKQFOUSZ
69. ·ͱΊ wΞϓϦέʔγϣϯαʔόΛத৺ʹݱ৔ͷαʔό ߏ੒ͱͦͷצͲ͜ΖΛ঺հ
    wͳͥෳ਺ͷαʔόΛ࢖͏ͷ͔
    w8FCαʔϏε૚ߏ੒
    1SPYZ"QQ%#
    wΩϟογϡɺ/P42-ɺඇಉظॲཧɺόον
70. ͍͞͝ʹ wΞϓϦέʔγϣϯΤϯδχΞ͔ΒΦϖ ϨʔγϣϯΤϯδχΞʹస޲͢Δਓ͸ ҙ֎ͱଟ͍
    w͢͜͠Ͱ΋8FCͷΠϯϑϥʹڵຯΛ ΋ͬͯ΋Β͑Ε͹ͱࢥ͍·͢