SREグループができてこの半年間やってきたこと

SRE Tech Talks #2 XFLAG スタジオにおけるSREの紹介、MySQL, InnoDB, THPのチューニングなど

Transcript

1. SREάϧʔϓ͕Ͱ͖ͯ͜ͷ൒೥ؒ΍͖ͬͯͨ͜ͱ
   ήʔϜ։ൃ෦ SREάϧʔϓ ਗ਼ਫ ܄ @isaoshimizu
   2017.1.30 SRE Tech Talks #2
   XFLAG STUDIO
   

   View Slide

2. About me
   - ਗ਼ਫ ܄ @isaoshimizu
   - 2011.8-2014.3 SNS mixiͷӡ༻
   - XFLAG ελδΦ
   - 2014.4-2016.6 αʔόʔΤϯδχΞ
   - 2016.7- SRE
   - ओʹࠃ಺޲͚ϞϯελʔετϥΠΫΛࢧ͑Δ͓࢓ࣄ
   - ଞʹ΋ϞϯετελδΞϜɺϒϥφΠDASHͳͲ
   - ޷͖ͳ΋ͷ
   - LinuxɺMySQLɺnginxɺMemcachedͳͲͷϛυϧ΢ΣΞશൠɺGolangɺΫϥϑτϏʔϧ
   - 2016.3.1 dots. Conference Spring 2016ʮϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔ΒʯΛൃද
   - https://speakerdeck.com/isaoshimizu/monsutowozhi-eruinhurafalsejin-tokorekara
   2
   

   View Slide

3. XFLAG ελδΦʹ͓͚ΔSRE
   

   View Slide

4. XFLAG ελδΦʹ͓͚ΔSRE
   - 2016೥7݄ʹSREάϧʔϓ͕૑ઃ
   - SREʹͳ͔ͬͨΒͱ͍ͬͯಥવେ͖͘มΘͬͨ͜ͱ͸ͳ͘ɺैདྷ͔Βܧଓͯ͠΍͍ͬͯΔ͜ͱ΋ଟ͍
   - SREͷνʔϜʹ͓͚Δϛογϣϯ͕Θ͔Γ΍͘͢ͳͬͨ
   - ࣾ಺ˍࣾ֎͔Β΋Θ͔Γ΍͍͢૊৫ମ੍
   - ήʔϜʹؔΘΔػೳ։ൃ͔Βͷ෼཭
   - ෛՙରࡦɺޮ཰ԽɺࣗಈԽͳͲʹ஫ྗ
   - ϝϯόʔͷಘҙෆಘҙΛ૬ޓʹิ׬͠ͳ͕Β࡞ۀΛ͜ͳ͢
   - ౰൪੍Ͱෆଌͷࣄଶʹඋ͑Δ
   - جຊۀ຿
   - Πϯϑϥɾαʔόʔӡ༻ɺՄ༻ੑɾύϑΥʔϚϯε޲্ɺϩάऩूɺ։ൃ؀ڥ੔උɺηΩϡϦςΟରࡦɺπʔϧ։ൃ
   - ຊ൪/εςʔδϯά؀ڥ΁ͷσϓϩΠɺήʔϜσʔλͷΠϯϙʔτ
   - εςʔδϯάҎલͷ։ൃ؀ڥ͸։ൃऀࣗ਎͕SlackͷϘοτܦ༝ͰσϓϩΠɾΠϯϙʔτՄೳ
   - άϥϑΛݟΔɺϩάΛಡΉɺίʔυΛཧղ͢ΔɺGitHub IssueͰ໰୊Λڞ༗͢ΔɺPull RequestͰղܾ͢Δ
   4
   

   View Slide

5. ౰൪੍
   - ౰൪͸ܭ11໊ʢSREͷϝϯόʔ͸ݱࡏ7໊ʣ
   - 2໊ମ੍Ͱຖिϩʔςʔγϣϯ
   - GW΍೥຤೥࢝ͳͲͷ࿈ٳͷ࣌͸1೔୯ҐͰަ୅
   - جຊతʹఆ࣌֎ͱٳ೔͸౰൪͕Ұ࣍ରԠ͢Δ
   - NagiosͰো֐Λݕ஌ɺPagerDutyͱ࿈ܞͯ͠౰൪΁௨஌
   - ো֐ʹݶΒͣɺΤϯδχΞҎ֎͔Βۓٸͷ࿈བྷ͕ड͚ΒΕΔΑ͏ʹɺ

   Slack͔ΒϘοτܦ༝Ͱ౰൪΁௨஌Ͱ͖Δ࢓૊Έ΋͋Δ
   - ৔߹ʹΑͬͯ͸ΤεΧϨʔγϣϯͯ͠૯ಈһͰରԠ͢Δ
   - ࠓͷ౰൪੍͕Ͱ͖ͨͷ͸5೥Ҏ্લΒ͍͠
   5
   

   View Slide

6. Πϯϑϥߏ੒
   

   View Slide

7. ΦϯϓϨͱΫϥ΢υͷϋΠϒϦουߏ੒
   - ϞϯελʔετϥΠΫʢ೔ຊ൛ʣͷ࿩Ͱ͢
   - Πϯϑϥ͸ෳ਺DCͰΦϯϓϨϛεʢ෺ཧαʔόʣ͕ओମ
   - άϩʔόϧ൛΍ଞҊ݅Ͱ͸AWS͕ଟ͍
   - Ϋϥ΢υ΋ੵۃతʹར༻
   - ͓΋ʹຊ൪Appαʔό༻్ͱͯ͠ܭ਺ඦ୆ن໛ͷΠϯελϯεʢ࣌ظʹΑͬͯมಈʣ
   - AWS c4.4xlarge (16core, 30GB)
   - GMOΞϓϦΫϥ΢υ N-4096 (40core, 96GB)
   - ͦΕҎ֎Ͱ͸Route 53, CloudFront, Turnαʔό, ։ൃ؀ڥ, WebαΠτܥͳͲ
   - ݱࡏ͸ຊ൪Ҏ֎΋ؚΊͯશମͰ1,300୆લޙͷαʔόΛӡ༻
   7
   

   View Slide

8. ࠷ۙऔΓ૊Μͩ͜ͱͷྫ
   

   View Slide

9. ೥຤೥࢝ରࡦͷҰྫ
   

   View Slide

10. ೥຤೥࢝ରࡦͷҰྫ
    - Appαʔόͷ૿ڧ
    - ૯ίΞ਺׵ࢉʢ࿦ཧίΞʣͰ10,000ίΞΛ15,000ίΞʹҰ࣌తʹ૿ڧ
    - ϫʔΧʔ਺ͷௐ੔
    - σʔλϕʔε
    - ෛՙͷߴ͍DBΛγϟʔσΟϯά
    - ΫΤϦൃߦ਺ΛݮΒ͢
    - εέʔϧΞοϓ
    - ioDrive, ioMemory΁ͷೖΕସ͑
    - ݹ͍ϚγϯɺΠϯελϯεͷϦϓϨʔε
    - νϡʔχϯάʢޙड़ʣ
    - MySQL(InnoDB)ͷઃఆͷ࠷దԽ
    - OSͷΞοϓσʔτ
    - ͏Δ͏ඵରࡦ
    - FMి೾ࣜͷNTPαʔόʢΞϓϥΠΞϯεʣͷར༻ʢ֎෦αʔϏεʹґଘ͠ͳ͍ʣ
    - 2017೥1݄1೔9:00ͷ6࣌ؒલ͔Βঃʑʹ஗Βͤͯ1ඵؒΛௐ੔͢ΔઃఆʢLI͸ૠೖͤ͞ͳ͍ʣ
    10
    

    View Slide

11. MySQLνϡʔχϯά
    

    View Slide

12. MySQL
    - MariaDB 5.5ܥͰ౷Ұ
    - ChefͰϓϩϏδϣχϯά
    - σΟεΫ͸SSD or ioDrive/ioMemoryͷ2ύλʔϯʢSAS͸ແ͍ʣ
    - ౰વͳ͕ΒInnoDBͷΑ͋͘ΔΑ͏ͳઃఆ͸Ұ௨Γ΍ͬͯ͋ΔʢSNS࣌୅͔Β
    վળΛଓ͚͖ͯͨmy.cnfʣ
    - ࠓճɺओʹνϡʔχϯάͨ͠ͷઃఆ͸2ͭʢ࣍ϖʔδ΁ʣ
    12
    

    View Slide

13. MySQL(InnoDB) νϡʔχϯάᶃ
    - innodb_io_capacity
    - SSD, ioDrive/ioMemoryͰͦΕͧΕݻఆ஋Λઃఆ͖͕ͯͨ͠ɺ

    ༻్ʹΑͬͯม͑Δ༨஍͕͋ͬͨʢάϥϑ؍࡯ʹΑΓʣ
    - Dirty pages rate, Checkpoint Age, Disk IO, CPU Usage͋ͨΓͷ

    όϥϯεΛݟͳ͕Β࠷ద஋ΛܾΊΔ
    - ະ࢖༻ͷSlaveͰࢼ͔ͯ͠ΒMasterʹద༻
    - ioDriveͰ10,000͔Β20,000ʹม͑ͨࣄྫ΋
    - ݁Ռͱͯ͠ϨϓϦέʔγϣϯ஗ԆΛ؇࿨͢ΔޮՌ΍ɺspin lockͷճ਺͕
    ݮΔޮՌ͕ಘΒΕͨ
    13
    

    View Slide

14. MySQL(InnoDB) νϡʔχϯάᶄ
    - innodb_spin_wait_delay
    - ͍··ͰσϑΥϧτ஋ 6 Λ࢖͖ͬͯͨʢཁ͸ະઃఆʣ
    - ͔͜͜ΒMySQLʹ͓͚Δಉظॲཧͷ࿩
    - εϨουͷಉظॲཧʹspin lockΛ༏ઌͯ͠࢖͍ͬͯΔʢϏδʔ΢ΣΠτͱ΋ݴ͏ʣ
    - ௨ৗͷmutex͸ϩοΫΛऔಘͰ͖ͳ͍৔߹γάφϧΛ଴ͪଓ͚Δ͕ɺspin lock͸ϧʔϓʹΑͬͯ
    ϩοΫΛऔಘ͚ͭͮ͠Δ
    - εϨου͸ϩοΫ͕࢖༻Մೳ͔Ͳ͏͔νΣοΫΛ͢ΔͨΊʹϧʔϓ͚ͭͮ͠Δ
    - ͨͩ͠ɺϩοΫͷղ์Λ଴͍ͬͯΔؒɺϧʔϓ͍ͯ͠Δ෼CPUΛ࢖͍ଓ͚ΔͷͰɺϚϧνίΞͷ؀
    ڥ͔ͭɺ୹࣌ؒͰϩοΫ͕ղ์͞ΕΔέʔεʹ༗ޮͳख๏ͱ͍ΘΕ͍ͯΔ
    - ϩοΫ଴ͪͷϧʔϓͷճ਺͕innodb_sync_spin_loopsΛ௒͑ͨ৔߹͸ɺOSͷ
    mutex(pthread_cond_wait)ʹҾ͖ܧ͕ΕΔ
    - ҰൠతʹOS͕ఏڙ͢ΔmutexΛ࢖͏ํ͕ίετ͕ߴ͍ͱ͞Ε͍ͯΔ
    14
    

    View Slide

15. MySQL(InnoDB) νϡʔχϯάᶅ
    - MySQLͷspin lockͰ͸ɺinnodb_sync_spin_loops ෼ϧʔϓ͕ճΔ·ͰϩοΫͷऔಘΛࢼߦ͢Δ
    ͕ 0 ͔Β innodb_spin_wait_delay (default 6) ͷൣғͰϥϯμϜʹ଴ͪ࣌ؒΛઃ͚͍ͯΔ
    - MariaDBͷίʔυྫ

    https://github.com/MariaDB/server/blob/mariadb-5.5.54/storage/innobase/sync/
    sync0sync.c#L527
    - innodb_spin_wait_delay=50Λઃఆͨ͠ͱ͖ͷ࣮ࡍͷάϥϑͷྫ
    15
    ઃఆޙ͸os_waits͕ݮ͍ͬͯΔ
    

    View Slide

16. MySQL(InnoDB) νϡʔχϯά sysbench݁Ռᶃ
    - innodb_spin_wait_delayͷνϡʔχϯά͸sysbenchͷ݁Ռʹ΋޷Өڹ͕ग़ͨ
    16
    ࠨʢ݄༵೔ͷάϥϑʣ͕ઃఆલʢσϑΥϧτ஋ʣɺӈʢ౔༵೔ͷάϥϑʣ͕ઃఆޙ
    ※CPU UserͷUsage͕૿͑ͨͷ͸spin loop࣌ͷ଴͕ͪ࣌ؒ૿͑ͨͨΊʢ࣮ӡ༻Ͱ͜Ε΄ͲCPUΛ৯͏͜ͱ͸΄΅ແ͍ʣ
    ※࣮ࢪϚγϯͷεϖοΫ: Xeon X5650 2.67GHz x 2(24core), Memory 64GB, SSD 480GBx2(LVM)
    

    View Slide

17. MySQL(InnoDB) νϡʔχϯά sysbench݁Ռᶄ
    17
    ࠨʢ݄༵೔ͷάϥϑʣ͕ઃఆલʢσϑΥϧτ஋ʣɺ
    ӈʢ౔༵೔ͷάϥϑʣ͕ઃఆޙ
    ϕϯν݁Ռͷ਺஋(໿22%վળ)
    total time: -> 20401.1135s -> 15875.2834s
    read/write requests: 7847.98 -> 10082.02/s
    transactions: 490.21/s -> 629.96/s
    

    View Slide

18. OSΞοϓσʔτ
    

    View Slide

19. OSͷΞοϓσʔτ
    - 2013೥ʹαʔϏε։࢝ͨ͠౰ॳ͸Ubuntu Server 12.04 LTS
    - ݱࡏ͸Ubuntu Server 14.04 LTSΛར༻
    - 2017೥4݄ʹEOLΛܴ͑Δ12.04͔Β14.04΁ঃʑʹೖΕସ͍͑ͯΔ
    - 16.04͸systemd΁ͷରԠ΍ɺKernelͷࠩ෼͕େ͖͍͜ͱ͔Β࡞ۀίετ͕

    ΍΍ߴͦ͏ͩͬͨͷͰҰ୴ݟૹΓʢૣ͍ஈ֊ͰΞοϓσʔτ͍ͨ͠ʣ
    - 14.04Λ࢖͍࢝ΊΔͱ͖ʹى͖ͨ͜ͱ
    - Transparent Huge Page໰୊
    - ύϑΥʔϚϯεͷมԽ
    19
    

    View Slide

20. Transparent Huge Page໰୊ᶃ
    - KernelόʔδϣϯͷࠩͰى͖ΔϝϞϦ໰୊
    - ҎԼͷάϥϑ͸Kernel 3.8ͱ3.13ͷ؀ڥͰsysbenchΛ࣮ߦͨ͠ͱ͖ͷϝϞϦ࢖༻ྔ
    20
    Kernel 3.8 Kernel 3.13
    OOM Killerൃੜ
    

    View Slide

21. Transparent Huge Page໰୊ᶄ
    - Kernel 3.13Ҏ߱ͰMySQLͳͲͷϝϞϦΛେྔʹ࢖͏έʔεʹ͓͍ͯɺϝϞϦ࢖༻ྔ͕๲ΒΉͷ͸ Transparent Huge Pages(THP)͕Өڹ
    ͍ͯ͠Δ͜ͱ͕൑໌
    - THP͸Linux 2.6.38ͰϚʔδ͞Εͨ΋ͷʢ2011೥1݄ʣ
    - http://kernelnewbies.org/Linux_2_6_38#head-f28790278bf537b4c4869456ad7b84425298708b
    - /sys/kernel/mm/transparent_hugepage/enabled ͷ஋ͰTHPͷ༗ޮɾແޮ੍͕ޚͰ͖Δ
    - Ubuntuͷ৔߹ɺKernel 3.8Ͱ͸ madviseɺKernel 3.13Ҏ߱Ͱ͸ always ͕σϑΥϧτ஋ͱͯ͠ઃఆ͞ΕΔͨΊɺ3.13Ҏ߱Ͱ͸THP͕ৗ
    ʹ༗ޮͱͳΔ
    - madvise Ͱ͸໌ࣔతʹTHPΛ࢖͏Α͏ʹ͠ͳ͍ݶΓ࢖͏͜ͱ͸ͳ͍
    - never Λࢦఆ͢ΔͱTHP͸࢖Θͳ͍
    - RHELͷPerformance Tuning Guide https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-transhuge.html
    - “THP is not recommended for database workloads.”
    - THP͕༗ޮͷ৔߹ɺ/proc/meminfoͷAnonHugePagesͷ஋͕૿͑Δ
    - ࣮͸MongoDBͰ͸͓ͳ͡Έͷઃఆ

    https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
    - Brendan GreggઌੜͷεϥΠυʹ͸͍ͩͿॿ͚ΒΕͨ

    http://www.slideshare.net/brendangregg/linux-systems-performance-2016 21
    

    View Slide

22. Transparent Huge Page໰୊ᶅ
    - Kernel 3.13Ͱɺ/sys/kernel/mm/transparent_hugepage/enabled ͷ஋Λ neverʹͯ͠
    sysbenchΛ࣮ࢪ
    22
    Kernel 3.8 (madvise) Kernel 3.13 (never)
    

    View Slide

23. Transparent Huge Page໰୊ᶆ
    - ݕূͰ͓͜ͳͬͨsysbenchʹ͍ͭͯ
    - sysbench --test=oltp.lua --oltp-table-size=300000000 --mysql-table-engine=innodb --num-threads=10000 --max-requests=10000000
    --max-time=0 --oltp-test-mode=complex --mysql-host=ࣗϚγϯͷIP —mysql-user=benchuser --mysql-password=benchpass run
    - oltp.lua͸SUMͱDISTINCTͷΫΤϦΛ࡟আʢ࣮ӡ༻Ͱ͸࢖Θͳ͍ؔ਺ͷͨΊʣ
    - ͔͔ͬͨ࣌ؒ
    - Kernel 3.8: 362෼
    - Kernel 3.13: 361෼
    - Kernel 4.2: 313෼
    - 4.2͚ͩ΍ͨΒͱ଎͔ͬͨ
    - εϨου୯ମͷϕϯνϚʔΫʢ sysbench --test=threads --num-threads=$NUM_THREADS --max-time=0 run Λ5ճ࣮ࢪͨ͠ฏۉ஋ʣ
    - Kernel 3.8: 5.2s
    - Kernel 3.13: 6.0s
    - Kernel 3.16: 3.5s
    - Kernel 4.2: 4.0s
    - Kernel 3.16Ҏ߱εϨου·ΘΓ͕վળ͞Εͨʢʁʣ
    23
    

    View Slide

24. ิ଍: NUMAͱTHPʹؔΘΔϝϞϦ໰୊ᶃ
    - ࠷ۙͷϚγϯ͸CPU͝ͱʹϝϞϦ͕۠෼͚͞ΕͯɺCPUʹ͍ۙϝϞϦʹରͯ͠ߴ଎ʹΞΫηεͰ͖ΔΑ͏ʹ
    ͳͬͨʢNUMA=Non-Uniform Memory Accessʣɻ
    - NUMAʹ͓͚ΔσϑΥϧτͷڍಈͱͯ͠͸ɺϓϩηε͕ར༻͢ΔCPU͔Β͍ۙϝϞϦϊʔυ͕༏ઌతʹ࢖ΘΕ
    ͍ͯ͘ɻେྔʹϝϞϦΛ࢖͏έʔεͳͲͰɺ͍ۙϊʔυΛ࢖͍ਚͯ͘͠͠·͏ͱଞํͷԕ͍ϝϞϦϊʔυΛ࢖͍
    ͸͡ΊΔ͕ɺswap͕ൃੜͯ͠͠·͏໰୊͕͋Δʢswap insanityͱݺ͹ΕΔ)ɻswapͷൃੜʹΑͬͯύϑΥʔ
    Ϛϯε͸ྼԽͯ͠͠·͏ɻ
    - ໌ࣔతʹinterleaveϙϦγʔΛ࢖͏͜ͱͰCPUʹ͍ۙϝϞϦϊʔυΛ࢖͏ͷͰ͸ͳ͘ɺϥ΢ϯυϩϏϯʹϝϞϦ
    ϊʔυΛબ୒͢Δ͜ͱ͕Մೳɻ͜Ε͸ɺnumactlίϚϯυͷ--interleveΦϓγϣϯΛࢦఆ࣮ͯ͠ߦ͢Δ͜ͱͰಉ
    ҰϓϩηεͰinterleaveϙϦγʔ͕࣮ݱͰ͖ΔɻϝϞϦΛେྔʹ࢖͏έʔεͰ͸swapΛ๷͙͜ͱ͕Ͱ͖Δͨ
    Ίɺ͜Ε͕ਪ঑͞ΕΔɻ
    24
    

    View Slide

25. ิ଍: NUMAͱTHPʹؔΘΔϝϞϦ໰୊ᶄ
    - THP͸ϖʔδαΠζΛ4KB͔Β2MBʹେ͖͘͢Δ͜ͱͰɺPTE(Page Table Entry)ͷΩϟογϡͰ͋ΔTLB(Translation
    Lookaside Buffer)ར༻ͷޮ཰Խ͕ਤΒΕΔ͕ɺαΠζ͕େ͖͘ͳΔ෼ϝϞϦ֬อʹ͔͔͔࣌ؒΔΑ͏ʹͳΓɺ2MBҎ
    ԼͷϝϞϦ֬อ࣌ʹແବ͕ੜ·ΕΔͱ͍͏σϝϦοτ͕͋Δɻ
    - NUMAʹ͓͍ͯɺϖʔδαΠζ͕4KBͷͱ͖ʹ2MB֬อͨ͠৔߹͸ɺCPUʹ͍ۙϊʔυʹ෼ࢄͯ͠ϝϞϦ͕֬อ͞ΕΔ
    ͕ɺϖʔδαΠζ͕2MBͰ2MB෼֬อ͢Δͱɺಛఆͷϊʔυʹ͔֬͠อ͞Εͳ͘ͳΓɺԕํͷCPU͔ΒΞΫηε͞ΕΔ
    ࡍʹΦʔόʔϔου͕ੜ·ΕΔ͜ͱ͕͋Δ( http://www.fabiengaud.net/resources/gaud14large-slides.pdf )ɻ
    - εϨου΍ϓϩηεΛࢀর͍ͯ͠ΔϝϞϦͷۙ͘ʹҠߦͨ͠Γɺࢀর͍ͯ͠ΔσʔλΛۙ͘ͷϝϞϦʹҠߦ͢ΔͷΛࣗ
    ಈతʹ΍ͬͯ͘ΕΔNUMA Auto Balacingͱ͍͏ػೳ͕͋Δ͕ɺҠߦ͢Δ͜ͱʹΑͬͯPage Fault͕૿Ճ͢Δέʔε͕
    ͋Δɻ
    - zone_reclaim_mode͕༗ޮʹͳ͍ͬͯΔͱɺCPUʹ͍ۙϝϞϦΛੵۃతʹ࢖͏Α͏ʹͳΔ͕ɺͦͷͨΊʹΩϟογϡΛ
    ࣺͯΔ͜ͱ΋ࣙ͞ͳ͍ɻmemoryͷnodeͷڑ཭Λݟͯɺԕ͍ͱࣗಈతʹ1ͱ͢Δ΍͔͍ͬͳ΋ͷͳͷͰɺmemoryΛ୔
    ࢁ࢖͏Α͏ͳworkloadͰ͸ݻఆతʹ0ʹ͢Δͷ͕ৗ౟खஈ( zone_reclaim_mode = 0 )ɻ
    - Ҏ্ΑΓɺNUMAɺಛʹϝϞϦΛ࢖͏σʔλϕʔεʹ͓͍ͯ͸ɺTHPɺNUMA Auto Balancingɺzone_reclaim_mode
    ΛແޮԽ͠ͳ͍ͱෆརӹͳ͜ͱ͕ଟ͍ɻ
    25
    

    View Slide

26. ิ଍: NUMAͱTHPʹؔΘΔϝϞϦ໰୊ᶅ
    - ࣮ࡍͷઃఆ஋
    - echo never > /sys/kernel/mm/transparent_hugepage/enabled
    - echo 0 > /proc/sys/kernel/numa_balancing
    - zone_reclaim_mode = 0
    THPͱNUMA Auto Balancing͸GRUBͷઃఆͰແޮԽͰ͖Δ
    - /etc/default/grub ʹ transparent_hugepage=never numa_balancing=disable Λ
    ௥هͯ͠update-grub͢Δ
    - OSͷॳظઃఆ࣌ʹ͜ΕΒΛઃఆ͍ͯ͠Δ
    26
    

    View Slide

27. ิ଍: NUMAͱTHPʹؔΘΔϝϞϦ໰୊ᶆ
    Amazon Linux AMI 2016.09.1.20170119 x86_64 HVM (Kernel 4.4.41-36.55)
    $ cat /sys/kernel/mm/transparent_hugepage/enabled
    always [madvise] never
    Red Hat Enterprise Linux 7.3 (HVM) (Kernel 3.10.0-514)
    $ cat /sys/kernel/mm/transparent_hugepage/enabled
    [always] madvise never
    CentOS Linux 7 (Kernel 3.10.0-327.10.1)
    $ cat /sys/kernel/mm/transparent_hugepage/enabled
    [always] madvise never
    Ubuntu Server 16.04 LTS (HVM) (Kernel 4.4.0.53)
    $ cat /sys/kernel/mm/transparent_hugepage/enabled
    [always] madvise never
    27
    

    View Slide

28. ·ͱΊ
    

    View Slide

29. ·ͱΊ
    - ͜ͷ൒೥͘Β͍ʹى͖ͨ͜ͱɺରࡦͨ͜͠ͱͷҰ෦Λ঺հ͠·ͨ͠
    - MySQLͷڍಈ͸Ͱ͖ΔݶΓՄࢹԽͯ͠νϡʔχϯά
    - NUMA΍THP·ΘΓͷ໰୊͸ཁ஫ҙʢແޮԽ͕ແ೉ʣ
    - SREͷ͓࢓ࣄͷ෯͸ͱͯ΋޿͘ɺ΍ΕΔ͜ͱ͸·ͩ·ͩͨ͘͞Μ͋Γ·͢
    - େن໛ͳΒͰ͸ͷ΍Γ͕͍΋ͨ͘͞Μ͋Γ·͢
    - ҰॹʹγεςϜΛվળ͍͖ͯ͠·ͤΜ͔ʁ
    - https://xflag.com/recruit/
    29
    

    View Slide

30. ͪΐͬͱ͚ͩએ఻
    - ʲDMM GAMESओ࠵ʂʳʮෳࡶɾେن໛webαʔϏεΛࢧ͑Δٕज़ษڧձʯ
    - 2017೥2݄7೔(Ր) 19:30 ʙ 22:00ɺ৔ॴ͸͜͜υοπͰʂ
    - https://eventdots.jp/event/610781 ʢநબ੍ʣ
    - ʰେ͖͘ͳͬͨγεςϜΛվળ͍ͯ͘͠ͱ͍͏͜ͱʱͱ͍͏λΠτϧͰɺ

    ฐࣾ SREάϧʔϓ Ϧʔμʔͷҏ౻͕ొஃ͠·͢ɻͥͻ͝ొ࿥͍ͩ͘͞ɻ
    30
    

    View Slide

31. ࢀߟจݙ
    - Linux Systems Performance 2016 http://www.slideshare.net/brendangregg/linux-systems-performance-2016
    - Performance Analysis and Tuning - Part 1 http://www.slideshare.net/tommylee98229/shak-
    larryjederperfandtuningsummit14part1final
    - Optimizing Linux Memory Management for Low-latency / High-throughput Databases https://
    engineering.linkedin.com/performance/optimizing-linux-memory-management-low-latency-high-throughput-databases
    - zone_reclaim_modeͷσϑΥϧτ஋͕มΘΓ·͢ http://mkosaki.blog46.fc2.com/blog-entry-936.html
    - MySQL Performance: InnoDB IO Capacity & Flushing http://dimitrik.free.fr/blog/archives/2010/07/mysql-
    performance-innodb-io-capacity-flushing.html
    - MySQL΍SSDͱ͔ͷ࿩ ޙฤ http://www.slideshare.net/takanorisejima/mysqlssd-56045479
    - InnoDB ͷ mutex ͷ࿩ʢೖ໳ฤʣ http://labs.gree.jp/blog/2015/12/15043/
    - show innodb status http://www.slideshare.net/myfinder/show-innodb-status-6345058
    - Dead Lock Analysis of spin_lock() in Linux Kernel (english) http://www.slideshare.net/SneekerYeh/dead-lock-
    analysis-of-spin-lock-in-linux-kernel-english
    - Ծ૝ϝϞϦʔΛࢧ͑Δ΋͏ͻͱͭͷΩϟογϡ TLB http://ascii.jp/elem/000/000/567/567889/
    31
    

    View Slide

32. Thank you!
    

    View Slide