Upgrade to Pro — share decks privately, control downloads, hide ads and more …

障害にすてるところなし

 障害にすてるところなし

tomoya yokota

March 11, 2017
Tweet

More Decks by tomoya yokota

Other Decks in Technology

Transcript

  1. ো֐ʹͯ͢Δͱ͜Ζͳ͠
    αΠϘ΢ζגࣜձࣾ @yokotaso

    View Slide

  2. We’re sorry for …

    View Slide

  3. ӡ༻ো֐ͷ࿩
    • ͝໎࿭Λ͓͔͚ͨ͠Έͳ͞· ͝ΊΜͳ͍͞
    • ؂ࢹɾରࡦɾௐࠪɾվળΛ೔ʑੋվળ
    • վળ͸·ͩ·ͩಓ൒͹
    • αΠϘ΢ζΛ຤Ӭ͘ΑΖ͓͘͠ئ͍͠·͢

    View Slide

  4. Ϋϥ΢υͷن໛ͱՔಇ཰
    • ͝ܖ໿͍͍͍ͨͩͯΔձ༷ࣾ ໿1ສ8ઍࣾ

    View Slide

  5. ӡ༻ো֐

    View Slide


  6. Boo!

    View Slide

  7. ͦΜͳ͜ͱͳ͍

    View Slide


  8. ŧŔŕŪGOOD

    View Slide

  9. ࣦഊ͔ΒపఈతʹֶͿ
    ো֐ֶश
    4

    View Slide

  10. جૅ஌ࣝΛ͚ͭΔ

    View Slide

  11. جૅతͳૉཆΛຏ͘
    ίϯϐϡʔλͷجૅΛ஌Ζ͏

    View Slide

  12. ੜ͖ͨσʔλ͕ো֐ʹ͸͋Δ
    • ϘτϧωοΫ͸Ͳ͜ʁ
    • େن໛αʔϏεٕज़ೖ໳Λ

    ࢀর͠ͳ͕Βௐࠪ͢ΔͱษڧʹͳΔ
    • ϝτϦΫε؂ࢹπʔϧͰ؍࡯͠Α͏
    • جૅతͳ஌ࣝ΍ಎ࡯͕

    ো֐ௐࠪͷͨͼʹຏ͔ΕΔ

    View Slide

  13. ஌ࣝͱܦݧ͸ཌྷΛत͚ͯ͘ΕΔ
    • ະ஌ͷ໰୊ʹग़ձͬͨΒཱͪฦΔ৔ॴ
    • ೉͍͠໰୊΄Ͳղܾͷࢳޱʹ
    • ҰਓͰௐࠪΛਐΊΒΕΔΑ͏ʹͳͬͨ
    • ϝτϦΫε͔ΒݪҼΛ༧૝Ͱ͖ΔΑ͏ʹ

    View Slide

  14. ίϯϐϡʔλʔͷେࣄͳࣄ͸
    ো֐͕ڭ͑ͯ͘Εͨ

    View Slide

  15. ஌ࣝΛਂ۷Γ͢Δ

    View Slide

  16. ಈ࡞ݪཧΛਂ͘ཧղ͢Δ
    ো֐͸࠷ߴͷྑ໰Ͱ͋Δ

    View Slide

  17. MySQLͷΠϯσοΫε͕࢖͑ͳ͍࿩

    View Slide

  18. খ͞ͳෆ۩߹վम͕ݪҼ
    • ॏෳআڈͷͨΊʹGROUP BY Λ҆қʹ͚ͭͨͷ͕ݪҼ
    • JOIN࣌ʹར༻ͨ͠ΠϯσοΫε͕GROUP BY ʹ࠶ར༻Ͱ͖ͳ͍
    • େྔͷϨίʔυʹରͯ͠filesort͕૸ͬͯྦ໨

    View Slide

  19. ཧղͨͭ͠΋Γʹͳͬͯ·ͤΜ͔
    • ো֐ͷͨͼʹجຊతͳಈ࡞ݪཧΛ֬ೝ
    • ো֐ௐࠪΛ௨ͯ͠ಈ࡞ݪཧͷཧղ͕ਂ·Δ
    • ಈ࡞ݪཧ͕ਂ͘ཧղͰ͖͍ͯͳ͍ؾ͖ͮ

    View Slide

  20. ো֐͸࠷ߴͷྑ໰Ͱ͋Δ

    View Slide

  21. νʔϜͰڧ͘ͳΔ

    View Slide

  22. ࣦഊΛΏΔ͢จԽ
    ࣦഊͷڞ༗͸ஏ͕ͩ໾ʹཱͭʂ

    View Slide

  23. Seasar2 S2DAO Ҡߦࣦഊ

    View Slide

  24. ͝ΊΜͳ͍͞!͝ΊΜͳ͍͞!
    • SeasarͷEOLʹ൐͏Spring FrameworkҠߦ
    • Seasar S2DAO͔ΒHibernateͷ΁Ҡߦࣦഊ
    • ύϑΥʔϚϯεྼԽɺOutOfMemoryError…
    • ׬શഊ๺Ͱ͢ɻ͝໎࿭Λ͓͔͚͠·ͨ͠

    View Slide

  25. ࣦഊΛड͚ೖΕΔจԽ

    View Slide

  26. PostmortemΛॻ͜͏
    • ໰୊Λ٬؍తʹ੔ཧ͢ΔͨΊʹ༗༻
    • ࣦഊΛ෼ੳɾڞ༗ɾ๷ࢭ
    • AWS S3 େن໛ো֐
    • git lab ຊ൪σʔλ࡟আ

    View Slide

  27. ࣦഊΛڞ༗Ͱ͖ΔจԽΛ࡞Δ
    • ࣦഊΛڞ༗ͯ͠܁Γฦ͞ͳ͍จԽΛ࡞Ζ͏
    • ࣦഊʹ͍ͭͯޠΔਓͷଘࡏҙٛ͸େ͖͍
    • ࣦഊΛڞ༗ͯ͘͠ΕΔਓʹѹ౗తײँ
    • ݠڏɾଚܟɾ৴པ(HRT) ΛԿ౓΋֬ೝ͠Α͏
    • ݸਓͷࣦഊͰ͸ͳ͍ɺ૊৫ͷࣦഊͩ

    View Slide

  28. View Slide

  29. ࣦഊ͸੒ޭͷதؒ఺Ͱ͔͠ͳ͍

    View Slide

  30. సΜͰ΋ͨͩͰ͸ى͖͕͋Βͳ͍ʂ
    • ࣦഊͷչ͕͠͞վળʹͭͳ͕Δ
    • S2Dao͔ΒHibernate΁ͷҠߦࣦഊΛ׆͔ͯ͠
    • JDBCTemplateΛར༻ͯ͠࠶ઃܭ
    • OSSͱͯ͠ެ։༧ఆ

    View Slide

  31. ࣦഊͷڞ༗͸ஏ͕ͩ໾ʹཱͭʂ

    View Slide

  32. Α͜͠·ͳੜ͖ํ

    View Slide

  33. ͣͿͱ͘ੜ͖Δ
    ੜ͖͍ͯΔ͚ͩͰ·Δ΋͏͚

    View Slide

  34. Java ṖͷύϑΥʔϚϯεྼԽ

    View Slide

  35. ղܾ͠·ͨ͠ (๑•̀ŷ•́)و✧
    • JIT Compiler͕ఀࢭ͢Δέʔε͕͋Δ
    • JVM͕JIT Compileͨ͠ίʔυΛࣺͯΔ
    • JIT Compiler͕෮ؼͤͣɺ࠷దԽͷ͔͔Βͳ͍

    Byte CodeΛJava͕࣮ߦ͠ଓ͚Δ
    • ύϑΥʔϚϯεྼԽΛى͜͢

    View Slide

  36. ղܾͷࢳޱ
    • GCʹׂΓ౰ͯΔThread਺͸

    CPU਺͔Βܾ·Δ
    • GCʹׂΓ౰ͯΒΕͨThread͸

    CPUΛ100%ઐ༗͢Δ

    View Slide

  37. ղܾͷࢳޱ
    • GCʹΑΔCPUར༻཰Λߟྀͯ͠΋༨༟͕͋Δ
    • Full GC͸ൃੜ͍ͯ͠ͳ͍
    • GCΛݪҼ͔Βഉআ
    • CPUෛՙ্͕͕ΔέʔεΛMLͰಡΈ͋͞Δ

    View Slide

  38. ϒϩάʹެ։

    View Slide

  39. View Slide

  40. ͝Ѫಡ͋Γ͕ͱ͏͍͟͝·͢ʂ
    • ͜ͷهࣄ͚ͩͰ 1ສ5ઍPV
    • ࠷ऴతʹ 700͸ͯϒ͘Β͍
    • OSS͸͓ੈ࿩ʹͳΓͬͺͳ͠
    • ৘ใެ։Ͱߩݙ͍ͨ͠ؾ࣋ͪ

    View Slide

  41. ͣͿͱ͘ੜ͖ΔΑ͏ʹ
    • ໘ന͍ٕज़هࣄʹͳͬͨΒṶ͚΋ͷ
    • ໘ന͍ࣄ͋ͬͨΒ͍͍ͳ͕ϞνϕʔγϣϯʹมԽ
    • ࠶ىಈͰճආͭͭ͠௕ظؒɺো֐͕ൃੜ͍ͯͨ͠
    • Θ͔Βͳͯ͘΋͍͍ɾಀ͍͍͛ͯ
    • ੜ͖͍ͯΔ͚ͩͰ·Δ΋͏͚ͷਫ਼ਆ

    View Slide

  42. ௐ΂Δ͚ͩͰ·Δ΋͏͚ʂ

    View Slide

  43. ো֐ֶशɺ͸͔ͲΓ·͔ͬ

    View Slide

  44. ো֐ʹͯ͢Δͱ͜Ζͳ͠

    View Slide