はてなにおけるLinuxネットワークスタックパフォーマンス改善 / Linux network performance improvement at hatena

by Yuuki Tsubouchi (yuuk1)

Slide 1

Slide 1 text

͸ͯͳʹ͓͚Δ -JOVYωοτϫʔΫελοΫ ύϑΥʔϚϯεվળ JEZ@VVLJ ͸ͯͳɾϖύϘٕज़େձ ʙΠϯϑϥٕज़ج൫ʙ!ژ౎

Slide 2

Slide 2 text

!Z@VVL JEZ@VVLJ ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ !͸ͯͳ ೖࣾ೥໨

Slide 3

Slide 3 text

৽ଔ೥໨ͷ࢓ࣄ Λ঺հ

Slide 4

Slide 4 text

1 ΢ΣϒγεςϜͷ ύϑΥʔϚϯε

Slide 5

Slide 5 text

΢ΣϒγεςϜͷయܕߏ੒ LB Proxy Application Database

Slide 6

Slide 6 text

ϘτϧωοΫ͸ Ͳ͔͜ʹ͋Δ

Slide 7

Slide 7 text

LB Proxy Web App Database ϘτϧωοΫʢϩʔϧʣ

Slide 8

Slide 8 text

LB Proxy Web App Database ϘτϧωοΫʢϦιʔεʣ $16 .FN %JTL /8 $16 .FN %JTL $16 .FN %JTL /8 /8 $16 .FN %JTL /8

Slide 9

Slide 9 text

ϘτϧωοΫʢϦιʔεৄࡉʣ $16 .FN %JTL /8 )PTU VTFS TZT JPXBJU ʜ VTFE DBDIFE CVGGFS ʜ XSJUF *014 SFBE *014 SFBE MBUFODZ ʜ UY CQT SY CQT UY QQT ʜ

Slide 10

Slide 10 text

$16VTSOJDFTZTJPXBJUJSRTPGUTUFBMHVFTUJEMF BMM ιϑτׂΓࠐΈʢύέοτड৴ʣ ෛՙ͕ߴ͍

Slide 11

Slide 11 text

CPU %irq %softirq

Slide 12

Slide 12 text

େྔύέοτ ͱͷઓ͍

Slide 13

Slide 13 text

2 ωοτϫʔΫ ελοΫ

Slide 14

Slide 14 text

ωοτϫʔΫελοΫ w ωοτϫʔΫ*0Λ࣮ݱ͢ΔͨΊʹඞཁ ͳཁૉ܈ w /*$ɺΧʔωϧͷ/*$υϥΠόɺΧʔ ωϧͷ5$1*1ελοΫɺιέοτͳͲ w ύέοτૹड৴ɺύέοτόοϑΝ؅ཧɺ ϓϩτίϧॲཧͳͲΛؚΉ w ड৴ଆͷෛՙ͕ߴ͍͜ͱ͕ଟ͍

Slide 15

Slide 15 text

ωοτϫʔΫ*0ߴ଎Խ w (C&ͳͲʹΑΔωοτϫʔΫଳҬ෯޲্ʹ ΑΓɺҰఆ࣌ؒʹॲཧ͢Δύέοτ਺͕૿େ w (CQT.QQT #ZUFTQBDLFU w ωοτϫʔΫελοΫॲཧͷ$16ෛՙ͕ߴ͘ ͳΔ w $16ΫϩοΫ͸্͕ΒͣϚϧνίΞ࣌୅ʹ ಥೖ w ͞·͟·ͳߴ଎Խख๏͕࣮૷͞Ε͍ͯΔ

Slide 16

Slide 16 text

NIC Kernel Process ύέοτड৴ϑϩʔ ؆қ

Slide 17

Slide 17 text

NIC Kernel Process ύέοτड৴ ύέοτసૹ ϓϩτίϧॲཧ σʔλίϐʔ SFBE SFDW ύέοτड৴ϑϩʔ ؆қ 5$1*1ͷύέοτϔομղੳɺνΣοΫαϜܭࢉͳͲ ΞϓϦέʔγϣϯϓϩηε͸γεςϜίʔϧͰ଴ͪड͚

Slide 18

Slide 18 text

NIC Process ύέοτड৴ϑϩʔ ৄղ ύέοτ ड৴ ϓϩτίϧ ॲཧ σʔλड৴ ॲཧ ,FSOFM 3JOH #VGGFS 4PDLFU #VGGFS ΧʔωϧͷॲཧΛ෼ղ

Slide 19

Slide 19 text

NIC Process ύέοτड৴ϑϩʔ ৄղ ύέοτ ड৴ ϓϩτίϧ ॲཧ σʔλड৴ ॲཧ ,FSOFM 3JOH #VGGFS 4PDLFU #VGGFS ϋʔυ ׂΓࠐΈ ιϑτ ׂΓࠐΈ ࢒Γͷॲཧ͸ιϑτׂΓࠐΈͰඇಉظॲཧ ஗Ԇॲཧ ϋʔυׂΓࠐΈͰ͸ϦϯάόοϑΝʹύέοτΛੵΉ·Ͱ

Slide 20

Slide 20 text

ͳͥछྨͷׂΓࠐΈʁ w ड৴ύέοτ૿ׂΓࠐΈ૿ίϯςΩ ετεΠον૿ߴ$16ෛՙ w ϋʔυׂΓࠐΈ͚ͩͩͱύέοτड৴ͷͨͼ ʹϓϩτίϧॲཧ·Ͱଈ࣮ߦ͞ΕΔ w ϋʔυׂΓࠐΈͰ͸ύέοτΛόοϑΝʹੵ Ή͚ͩʹͯ͠ɺճͷׂΓࠐΈॲཧͷ$16ෛ ՙΛԼ͛Δ w ιϑτׂΓࠐΈʹΑΓɺϝΠϯͷϓϩτίϧ ॲཧΛඇಉظॲཧ

Slide 21

Slide 21 text

/"1* /FX"1* w ޙճ͠ʹͯ͠΋ड৴ύέοτ͕૿͑Ε͹ιϑτׂΓࠐΈ ͷ਺͸૿͑Δ w ෳ਺ͷύέοτΛ·ͱΊͯͭͷιϑτׂΓࠐΈͰॲཧ ͍ͨ͠ w ϋʔυׂΓࠐΈΛҰ୴ͱΊͯɺϙʔϦϯάʹΑΓϦϯά όοϑΝ͔Βಉ࣌ʹύέοτΛऔΓग़͢ w ͨͩ͠ύέοτ਺͕গͳ͍৔߹ʹಛʹϨΠςϯγ͕େ͖ ͘ͳͬͯ͠·͏ͷͰɺड৴ύέοτ͕ଟ͍ͱ͖ͷΈ༗ޮ

Slide 22

Slide 22 text

NIC Process /"1* ύέοτ ड৴ ϓϩτίϧ ॲཧ σʔλड৴ ॲཧ ,FSOFM 3JOH #VGGFS 4PDLFU #VGGFS ϋʔυׂΓࠐΈΛͱΊͯɺϙʔϦϯά QPMMJOH

Slide 23

Slide 23 text

͜͜·Ͱ͸ -JOVYωοτϫʔΫ ελοΫͷجຊతͳ࿩

Slide 24

Slide 24 text

3 ͸ͯͳͰͷࣄྫ

Slide 25

Slide 25 text

)"1SPYZͷ୆਺૿໰୊ w )"1SPYZΛ&$Ͱͷϩʔυόϥϯαͱͯ͠࢖༻ w ιϑτׂΓࠐΈෛՙʹΑΓίΞͰ཯଎ w $16ίΞ͸༨͍ͬͯΔͷʹɺ)"1SPYZͷεέʔϧ Ξ΢τ͕ඞཁ w %/4ϥ΢ϯυϩϏϯʹΑΓɺෳ਺ͷ)"1SPYZ΁ ࢀরΛৼΓ෼͚ w ϩʔυόϥϯαΛϩʔυόϥϯε͠ͳ͍ͱ͍͚ͳ ͍ࣄଶ

Slide 26

Slide 26 text

)"1SPYZ ʜʜ EC ʜʜ BQQ ʜʜ TVHPJIBQSPYZQSJWBUF

Slide 27

Slide 27 text

ʜʜ EC ʜʜ BQQ ʜʜ )"1SPYZ

Slide 28

Slide 28 text

ׂΓࠐΈ͕ଟ͍ͳΒ ׂΓࠐΈࣗମΛݮΒ͢

Slide 29

Slide 29 text

NIC Process *OUFSSVQU$PBMFTDJOH ύέοτ ड৴ ϓϩτίϧ ॲཧ σʔλड৴ ॲཧ ,FSOFM 3JOH #VGGFS 4PDLFU #VGGFS /*$͕ͭͷϋʔυׂΓࠐΈͰෳ਺ͷύέοτΛ·ͱΊͯసૹ ϋʔυ ׂΓࠐΈ ෳ਺ύέοτἧ͏·Ͱ౸ணΛ଴ͭͨΊϨΠςϯγ͸্͕Δ

Slide 30

Slide 30 text

/*$υϥΠόͷύϥϝʔλݕূ w ࢖༻/*$͸*OUFMͷJYHCF*OUFSSVQU5ISPUUMF3BUFΦ ϓγϣϯ w DYMBSHF୆͔ΒDYMBSHF୆ʹ5$1઀ଓ w σϑΥϧτͰ͸΢ΣϒαʔϏεͷΑ͏ͳγϣʔτύ έοτ͕ࢧ഑తͳঢ়گͰ͸ɺׂΓࠐΈ਺͕ߴ͘ͳΔ w ΢ΣϒαʔϏεͰ͸ͦ͜·ͰϨΠςϯγʹγϏΞͰ ͸ͳ͍ w ݕূͷ݁ՌɺιϑτׂΓࠐΈͷ$16ར༻཰ TPGU ͕࠷େ·Ͱվળ l&$Ͱ43*07Λ࢖͏ͱ͖ͷ/*$υϥΠόύϥϝʔλݕূz IUUQCMPHZVVLJPFOUSZ

Slide 31

Slide 31 text

ׂΓࠐΈॲཧΛෳ਺ͷ ίΞʹ෼ࢄͤ͞Δ

Slide 32

Slide 32 text

NIC Process ύέοτ ड৴ ϓϩτίϧ ॲཧ σʔλड৴ ॲཧ ͳͥෳ਺ͷίΞΛ࢖͑ͳ͍ͷ͔ $16 ϋʔυׂΓࠐΈ͕͔͔ͬͨ$16ίΞͱಉ͡$16ίΞ͕ ιϑτׂΓࠐΈΛ࣮ߦ͢ΔͨΊ ϋʔυ ׂΓࠐΈ ιϑτ ׂΓࠐΈ

Slide 33

Slide 33 text

NIC Process ύέοτ ड৴ ϓϩτίϧ ॲཧ σʔλड৴ ॲཧ $16 314 3FDFJWF1BDLFU4UFFSJOH NIC Process ύέοτ ड৴ ϓϩτίϧ ॲཧ σʔλड৴ ॲཧ $16 NIC Process ύέοτ ड৴ ϓϩτίϧ ॲཧ σʔλड৴ ॲཧ $16 ίΞׂؒΓࠐΈ

Slide 34

Slide 34 text

314 3FDFJWF1BDLFU4UFFSJOH w ௨ৗ͸ϋʔυׂΓࠐΈ͕͔ͬͨ$16ͱಉ͡$16Ͱ ιϑτׂΓࠐΈΛ࣮ߦ w 314͸ɺଞͷ$16ʹίΞׂؒΓࠐΈͯ͠ɺϓϩτ ίϧॲཧΛଞͷ$16ʹ೚ͤΔ w 5$1ϔομͷ TBEES TQPSU EBEES EQPSU ͷ૊ʹ ର͠$POTJTUFOU)BTIJOHʹΑΔ෼ࢄ w ݪଇಉ͡5$1ϑϩʔͳΒ͹ಉ͡$16ʹ෼ࢄ͞Ε ΔΑ͏ʹ w $16Ωϟογϡώοτ཰ΛߴΊͨ3'4 3FDFJWF 'MPX4UFFSJOH ͱ͍͏ͷ΋͋Δ l-JOVYͰϩʔυόϥϯα΍ΩϟογϡαʔόΛϚϧνίΞεέʔϧͤ͞ΔͨΊͷΧʔωϧνϡʔχϯάz IUUQ CMPHZVVLJPFOUSZMJOVYOFUXPSLTUBDLUVOJOHSGT

Slide 35

Slide 35 text

ࣾ಺ͷ࠷ۙͷ ωοτϫʔΫαʔόʹ ޿͘ద༻ࡁ

Slide 36

Slide 36 text

314Ͱ3FEJTνϡʔχϯά w 3FEJT͸εϨουಈ࡞͢ΔͨΊɺϚϧν ίΞεέʔϧ͠ͳ͍ w ΧʔωϧͷωοτϫʔΫ*0ॲཧ͢ΔεϨο υͱɺ3FEJTεϨουͷ෼ࢄ͸Ͱ͖Δ w 314͸෼ࢄઌ$16൪߸ΛࢦఆͰ͖Δ w $16 ʹ෼ࢄͤ͞Δ w UBTLTFUͰ$16ΞϑΟχςΟΛઃఆ͠ɺ 3FEJTεϨουΛ$16 Ҏ֎ʹݻఆ IUUQSFEJTJPUPQJDTCFODINBSLT

Slide 37

Slide 37 text

$16ˋݮ

Slide 38

Slide 38 text

ଞʹ΋਺ଟ͘ͷ ߴ଎Խख๏͕͋Δ νΣοΫαϜΦϑϩʔυ 540 5$14FHNFOUBUJPO0⒐PBE -30 -BSHF3FDFJWF0⒐PBE 50& 5$10⒐PBE&OHJOF 344 3FDFJWF4JEF4DBMJOH "DDFMFSBUFE3'4 914 1$*QBTTUISPVHI 43*07 *OUFM%BUB%JSFDU*0

Slide 39

Slide 39 text

4 ωοτϫʔΫελο Ϋͷઌ୺ٕज़

Slide 40

Slide 40 text

ͦ΋ͦ΋Χʔωϧʹ ॲཧͤ͞ͳ͍ protocol process NIC process hardirq packet receive consistent hashing forward

Slide 41

Slide 41 text

Ϣʔβϥϯυ΁θϩίϐʔ w ύέοτΛ/*$͔ΒϢʔβϥϯυ΁όΠύε w ࣮૷ w OFUNBQ w *OUFM%1%, w ϢʔβϥϯυͰ5$1*1࣮૷ w ஌ΔݶΓͰ͸࣮༻ஈ֊ͷωοτϫʔΫαʔό͸ ·ͩͳ͍

Slide 42

Slide 42 text

ιέοτ"1*ͷݶք w େ఍ͷωοτϫʔΫαʔόͷ࣮૷͸104*9ͷι έοτ"1*ϕʔε w ιέοτ"1*Ͱ͸θϩίϐʔʹͰ͖ͳ͍ w ΧʔωϧͷιέοτόοϑΝʹੵΉඞཁ͕͋Δ w طଘͷωοτϫʔΫαʔόͷιέοτ"1*Λஔ͖ ׵͑Δͷ͸େม w Χʔωϧ͕ϋʔυͷਐԽʹ௥͍͔ͭͳ͘ͳΕ͹ɺ ීٴ͢Δ͔΋͠Εͳ͍

Slide 43

Slide 43 text

5 ·ͱΊ

Slide 44

Slide 44 text

·ͱΊ w ΢ΣϒγεςϜʹ͓͚ΔωοτϫʔΫ*0ෛՙͷҐஔ͚ͮ w -JOVYͷωοτϫʔΫελοΫͷ࢓૊ΈΛ঺հ w ͸ͯͳͰ࣮ફͨ͠ωοτϫʔΫελοΫͷߴ଎Խख๏ w ׂΓࠐΈΛ·ͱΊΔ*OUFSSVQU$PBMFTDJOH w ϓϩτίϧॲཧΛϚϧνίΞεέʔϧͤ͞ΔͨΊͷ 3143'4 w )"1SPYZɺ3FEJTͷྫ w ઌ୺ٕज़ͱͯ͠ɺ/*$͔ΒϢʔβϥϯυ΁όΠύε͢Δ ٕज़͕͋Δ

Slide 45

Slide 45 text

04͸ ΞΠσΞͷๅݿ

Slide 46

Slide 46 text

04಺෦ͷٕज़ͷ֎ग़͠ w ιϑτׂΓࠐΈʹΑΔ஗Ԇॲཧ w δϣϒΩϡʔͷΑ͏ͳඇಉظॲཧ w ׂΓࠐΈͱϙʔϦϯά w ϑϩϯτΤϯυʹ͓͚ΔϩϯάϙʔϦϯά΍ 4FSWFS4FOU&WFOUT w ϓϩηεεέδϡʔϥ w ྲྀߦΓͷ7.΍ίϯςφͷΦʔέετϨʔγϣ ϯ

Slide 47

Slide 47 text

04ͷٕज़Λ ࣗ෼ͷιϑτ΢ΣΞʹ ͱΓ͍Ε͍ͨ

Slide 48

Slide 48 text

04΍γεςϜιϑτ΢ΣΞ ͕޷͖ͳਓ΁ IUUQIBUFOBDPSQKQSFDSVJUGSFTIPQFSBUJPOFOHJOFFS