障害にすてるところなし

Transcript

1. ো֐ʹͯ͢Δͱ͜Ζͳ͠
   αΠϘ΢ζגࣜձࣾ @yokotaso
   

   View Slide

2. We’re sorry for …
   

   View Slide

3. ӡ༻ো֐ͷ࿩
   • ͝໎࿭Λ͓͔͚ͨ͠Έͳ͞· ͝ΊΜͳ͍͞
   • ؂ࢹɾରࡦɾௐࠪɾվળΛ೔ʑੋվળ
   • վળ͸·ͩ·ͩಓ൒͹
   • αΠϘ΢ζΛ຤Ӭ͘ΑΖ͓͘͠ئ͍͠·͢
   

   View Slide

4. Ϋϥ΢υͷن໛ͱՔಇ཰
   • ͝ܖ໿͍͍͍ͨͩͯΔձ༷ࣾ ໿1ສ8ઍࣾ
   

   View Slide

5. ӡ༻ো֐
   

   View Slide

6. 
   Boo!
   

   View Slide

7. ͦΜͳ͜ͱͳ͍
   

   View Slide

8. 
   ŧŔŕŪGOOD
   

   View Slide

9. ࣦഊ͔ΒపఈతʹֶͿ
   ো֐ֶश
   4
   

   View Slide

10. جૅ஌ࣝΛ͚ͭΔ
    

    View Slide

11. جૅతͳૉཆΛຏ͘
    ίϯϐϡʔλͷجૅΛ஌Ζ͏
    

    View Slide

12. ੜ͖ͨσʔλ͕ো֐ʹ͸͋Δ
    • ϘτϧωοΫ͸Ͳ͜ʁ
    • େن໛αʔϏεٕज़ೖ໳Λ

    ࢀর͠ͳ͕Βௐࠪ͢ΔͱษڧʹͳΔ
    • ϝτϦΫε؂ࢹπʔϧͰ؍࡯͠Α͏
    • جૅతͳ஌ࣝ΍ಎ࡯͕

    ো֐ௐࠪͷͨͼʹຏ͔ΕΔ
    

    View Slide

13. ஌ࣝͱܦݧ͸ཌྷΛत͚ͯ͘ΕΔ
    • ະ஌ͷ໰୊ʹग़ձͬͨΒཱͪฦΔ৔ॴ
    • ೉͍͠໰୊΄Ͳղܾͷࢳޱʹ
    • ҰਓͰௐࠪΛਐΊΒΕΔΑ͏ʹͳͬͨ
    • ϝτϦΫε͔ΒݪҼΛ༧૝Ͱ͖ΔΑ͏ʹ
    

    View Slide

14. ίϯϐϡʔλʔͷେࣄͳࣄ͸
    ো֐͕ڭ͑ͯ͘Εͨ
    

    View Slide

15. ஌ࣝΛਂ۷Γ͢Δ
    

    View Slide

16. ಈ࡞ݪཧΛਂ͘ཧղ͢Δ
    ো֐͸࠷ߴͷྑ໰Ͱ͋Δ
    

    View Slide

17. MySQLͷΠϯσοΫε͕࢖͑ͳ͍࿩
    

    View Slide

18. খ͞ͳෆ۩߹վम͕ݪҼ
    • ॏෳআڈͷͨΊʹGROUP BY Λ҆қʹ͚ͭͨͷ͕ݪҼ
    • JOIN࣌ʹར༻ͨ͠ΠϯσοΫε͕GROUP BY ʹ࠶ར༻Ͱ͖ͳ͍
    • େྔͷϨίʔυʹରͯ͠filesort͕૸ͬͯྦ໨
    

    View Slide

19. ཧղͨͭ͠΋Γʹͳͬͯ·ͤΜ͔
    • ো֐ͷͨͼʹجຊతͳಈ࡞ݪཧΛ֬ೝ
    • ো֐ௐࠪΛ௨ͯ͠ಈ࡞ݪཧͷཧղ͕ਂ·Δ
    • ಈ࡞ݪཧ͕ਂ͘ཧղͰ͖͍ͯͳ͍ؾ͖ͮ
    

    View Slide

20. ো֐͸࠷ߴͷྑ໰Ͱ͋Δ
    

    View Slide

21. νʔϜͰڧ͘ͳΔ
    

    View Slide

22. ࣦഊΛΏΔ͢จԽ
    ࣦഊͷڞ༗͸ஏ͕ͩ໾ʹཱͭʂ
    

    View Slide

23. Seasar2 S2DAO Ҡߦࣦഊ
    

    View Slide

24. ͝ΊΜͳ͍͞!͝ΊΜͳ͍͞!
    • SeasarͷEOLʹ൐͏Spring FrameworkҠߦ
    • Seasar S2DAO͔ΒHibernateͷ΁Ҡߦࣦഊ
    • ύϑΥʔϚϯεྼԽɺOutOfMemoryError…
    • ׬શഊ๺Ͱ͢ɻ͝໎࿭Λ͓͔͚͠·ͨ͠
    

    View Slide

25. ࣦഊΛड͚ೖΕΔจԽ
    

    View Slide

26. PostmortemΛॻ͜͏
    • ໰୊Λ٬؍తʹ੔ཧ͢ΔͨΊʹ༗༻
    • ࣦഊΛ෼ੳɾڞ༗ɾ๷ࢭ
    • AWS S3 େن໛ো֐
    • git lab ຊ൪σʔλ࡟আ
    

    View Slide

27. ࣦഊΛڞ༗Ͱ͖ΔจԽΛ࡞Δ
    • ࣦഊΛڞ༗ͯ͠܁Γฦ͞ͳ͍จԽΛ࡞Ζ͏
    • ࣦഊʹ͍ͭͯޠΔਓͷଘࡏҙٛ͸େ͖͍
    • ࣦഊΛڞ༗ͯ͘͠ΕΔਓʹѹ౗తײँ
    • ݠڏɾଚܟɾ৴པ(HRT) ΛԿ౓΋֬ೝ͠Α͏
    • ݸਓͷࣦഊͰ͸ͳ͍ɺ૊৫ͷࣦഊͩ
    

    View Slide

28. View Slide

29. ࣦഊ͸੒ޭͷதؒ఺Ͱ͔͠ͳ͍
    

    View Slide

30. సΜͰ΋ͨͩͰ͸ى͖͕͋Βͳ͍ʂ
    • ࣦഊͷչ͕͠͞վળʹͭͳ͕Δ
    • S2Dao͔ΒHibernate΁ͷҠߦࣦഊΛ׆͔ͯ͠
    • JDBCTemplateΛར༻ͯ͠࠶ઃܭ
    • OSSͱͯ͠ެ։༧ఆ
    

    View Slide

31. ࣦഊͷڞ༗͸ஏ͕ͩ໾ʹཱͭʂ
    

    View Slide

32. Α͜͠·ͳੜ͖ํ
    

    View Slide

33. ͣͿͱ͘ੜ͖Δ
    ੜ͖͍ͯΔ͚ͩͰ·Δ΋͏͚
    

    View Slide

34. Java ṖͷύϑΥʔϚϯεྼԽ
    

    View Slide

35. ղܾ͠·ͨ͠ (๑•̀ŷ•́)و✧
    • JIT Compiler͕ఀࢭ͢Δέʔε͕͋Δ
    • JVM͕JIT Compileͨ͠ίʔυΛࣺͯΔ
    • JIT Compiler͕෮ؼͤͣɺ࠷దԽͷ͔͔Βͳ͍

    Byte CodeΛJava͕࣮ߦ͠ଓ͚Δ
    • ύϑΥʔϚϯεྼԽΛى͜͢
    

    View Slide

36. ղܾͷࢳޱ
    • GCʹׂΓ౰ͯΔThread਺͸

    CPU਺͔Βܾ·Δ
    • GCʹׂΓ౰ͯΒΕͨThread͸

    CPUΛ100%ઐ༗͢Δ
    

    View Slide

37. ղܾͷࢳޱ
    • GCʹΑΔCPUར༻཰Λߟྀͯ͠΋༨༟͕͋Δ
    • Full GC͸ൃੜ͍ͯ͠ͳ͍
    • GCΛݪҼ͔Βഉআ
    • CPUෛՙ্͕͕ΔέʔεΛMLͰಡΈ͋͞Δ
    

    View Slide

38. ϒϩάʹެ։
    

    View Slide

39. View Slide

40. ͝Ѫಡ͋Γ͕ͱ͏͍͟͝·͢ʂ
    • ͜ͷهࣄ͚ͩͰ 1ສ5ઍPV
    • ࠷ऴతʹ 700͸ͯϒ͘Β͍
    • OSS͸͓ੈ࿩ʹͳΓͬͺͳ͠
    • ৘ใެ։Ͱߩݙ͍ͨ͠ؾ࣋ͪ
    

    View Slide

41. ͣͿͱ͘ੜ͖ΔΑ͏ʹ
    • ໘ന͍ٕज़هࣄʹͳͬͨΒṶ͚΋ͷ
    • ໘ന͍ࣄ͋ͬͨΒ͍͍ͳ͕ϞνϕʔγϣϯʹมԽ
    • ࠶ىಈͰճආͭͭ͠௕ظؒɺো֐͕ൃੜ͍ͯͨ͠
    • Θ͔Βͳͯ͘΋͍͍ɾಀ͍͍͛ͯ
    • ੜ͖͍ͯΔ͚ͩͰ·Δ΋͏͚ͷਫ਼ਆ
    

    View Slide

42. ௐ΂Δ͚ͩͰ·Δ΋͏͚ʂ
    

    View Slide

43. ো֐ֶशɺ͸͔ͲΓ·͔ͬ
    

    View Slide

44. ো֐ʹͯ͢Δͱ͜Ζͳ͠
    

    View Slide