How we monitor microservices at Mercari microservices platform team

Talk at SRE session #1
https://sre-lounge.connpass.com/event/123701/

Explained the overview of our microservices monitoring.

Transcript

1. )PXXFNPOJUPS
   NJDSPTFSWJDFT
   BU.FSDBSJ.JDSPTFSWJDFT1MBUGPSNUFBN
   

   View Slide

2. !TQFTOPWB
   

   View Slide

3. $VSSFOU4UBUVT
   
   FOHJOFFST
   
   NJDSPTFSWJDFT
   
   DIBOHFTXFFL
   

   View Slide

4. ,VCFSOFUFT$MVTUFS (,&
   
   $VSSFOU4UBDL
   %# 4QBOOFSFUD
   
   .JDSPTFSWJDFT (PBQQFUD
   
   

   View Slide

5. "SDIJUFDUVSF0WFSWJFX
   ($-#
   DMJFOU
   "1*
   HBUFXBZ
   #''
   NJDSPTFSWJDFT
   NJDSPTFSWJDFT
   %#T
   

   View Slide

6. )PXXFNPOJUPS
   NJDSPTFSWJDFT
   

   View Slide

7. 4UBSUJOHGSPN
   l8IZXFNPOJUPS z
   

   View Slide

8. 8FXBOUUPLFFQTZTUFNTXPSLQSPQFSMZ
   8IZ
   

   View Slide

9. 8IFOBTZTUFNJTOPUXPSLJOHQSPQFSMZ XFOFFEUP
   EFUFDUJU JOWFTUJHBUFJUBOEpYJU
   

   View Slide

10. 4P IPXDBOXFEFUFDUBTZTUFNJTOUXPSLJOH
    

    View Slide

11. $PMMFDUNFUSJDT
    UPLOPXlXIBUz
    

    View Slide

12. .FUSJDT
    

    View Slide

13. ,VCFSOFUFT/PEF
    .JDSPTFSWJDFQPET
    .POJUPSJOHBHFOUQPE
    $PMMFDUJOHBSDIJUFDUVSFPWFSWJFX
    .FUSJDTTUPSBHF
    

    View Slide

14. .FUSJDTUFMMZPVXIFUIFSBTZTUFNJTXPSLJOHQSPQFSMZPSOPU
    

    View Slide

15. 5IJTNFUSJDUFMMZPVBTZTUFNXBTTMPXUIFO
    

    View Slide

16. 5IJTNFUSJDUFMMZPVBTZTUFNSFUVSOFENBOZFSSPST
    

    View Slide

17. :PVDBOLOPXBTZTUFNJTXPSLJOHCZDIFDLJOHNFUSJDT
    

    View Slide

18. .FUSJDTEPO`UUFMMZPVIPXTMPXBOEXIZTMPX
    )PXFWFS
    

    View Slide

19. $PMMFDUUSBDFT
    UPLOPXlIPXz
    

    View Slide

20. 5SBDFT
    

    View Slide

21. ,VCFSOFUFT/PEF
    .JDSPTFSWJDFQPET
    .POJUPSJOHBHFOUQPE
    $PMMFDUJOHBSDIJUFDUVSFPWFSWJFX
    5SBDFTTUPSBHF
    

    View Slide

22. 5SBDFTUFMMZPVIPXTMPX IPXCSPLFO
    

    View Slide

23. 5IJTUSBDFUFMMZPVUIFSFEQBSUUPPLNPTUPGUJNF
    

    View Slide

24. 5IJTUSBDFUFMMZPVXIFSFBOFSSPSIBQQFOFE
    

    View Slide

25. 1SPQBHBUJOHASFRVFTU@JEAGPSEJTUSJCVUFEUSBDJOH
    

    View Slide

26. 5PQPMPHZ TFSWJDF
    NBQ
    

    View Slide

27. 5SBDFTEPO`UUFMMZPVXIZ
    )PXFWFS
    

    View Slide

28. $PMMFDUMPHT
    UPLOPXlXIZz
    

    View Slide

29. -PHT
    

    View Slide

30. ,VCFSOFUFT/PEF
    .JDSPTFSWJDFQPET
    .POJUPSJOHBHFOUQPE
    $PMMFDUJOHBSDIJUFDUVSFPWFSWJFX
    -PHTTUPSBHF
    

    View Slide

31. -PHTUFMMZPVXIZJUIBQQFOFE
    

    View Slide

32. 1SPQBHBUJOHASFRVFTU@JEAGPSEJTUSJCVUFEMPHHJOH
    

    View Slide

33. 4FBSDIJOHMPHTXJUIlSFRVFTU@JEYYYYzBDSPTTNJDSPTFSWJDFT
    

    View Slide

34. %FpOJOHUIFMPHGPSNBUTUBOEBSEUPUPBOBMZ[FBDSPTTTFSWJDFT
    

    View Slide

35. View Slide

36. &BDINJDSPTFSWJDFTIPVMECFBCMFUPEFpOFJUTPXOMPHGPSNBU
    TPUIBUUIFTUBOEBSEEFpOFTPOMZDPNNPOBUUSJCVUFOBNFT
    

    View Slide

37. &OGPSDJOHUIFTUBOEBSEUPIVOESFETEFWFMPQFSTUBLFTUJNF
    #FQBUJFOUBOEIBWFBMPOHUFSNWJTJPO
    

    View Slide

38. 5IFUISFFQJMMBS
    PGPCTFSWBCJMJUZ
    8IBUEBUBEPXFDPMMFDU
    

    View Slide

39. 5IFUISFFQJMMBS
    PGPCTFSWBCJMJUZ
    .FUSJDT
    5SBDFT
    -PHT
    

    View Slide

40. 5IF3&%64&.FUIPE
    PS8PSLBOE3FTPVSDF.FUIPE
    
    )PXEPXFVTFUIFEBUB
    

    View Slide

41. 3FRVFTUT
    &SSPST
    %VSBUJPO
    XPSLNFUSJDT
    
    

    View Slide

42. 6UJMJ[BUJPO
    4BUVSBUJPO
    &SSPST
    SFTPVSDFNFUSJDT
    
    

    View Slide

43. IUUQTXXXEBUBEPHIRDPNCMPHNPOJUPSJOHJOWFTUJHBUJPO
    *OWFTUJHBUJPO'MPX
    $IFDL3&% XPSL
    NFUSJDTUPLOPX8)"5
    $IFDLUSBDFTUPLOPX)08
    $IFDL64& SFTPVSDF
    NFUSJDTUPLOPX8):
    $IFDLMPHTUPLOPX8):
    

    View Slide

44. IUUQTXXXEBUBEPHIRDPNCMPHNPOJUPSJOHJOWFTUJHBUJPO
    *OWFTUJHBUJPO'MPX
    

    View Slide

45. "MFSUJOH
    

    View Slide

46. 6TF3&%NFUSJDTGPSBMFSUJOH
    

    View Slide

47. 8FEPO`UVTF64&NFUSJDTGPSBMFSUJOH
    UPBWPJEGBMTFBMFSUJONPTUDBTFT
    

    View Slide

48. .POJUPSJOHTZTUFN
    .JDSPTFSWJDF
    $IBUOPUJpDBUJPO
    4.4QVTIOPUJpDBUJPO
    1IPOF$BMM
    $SJUJDBMBMFSU
    8BSOJOHBMFSU
    

    View Slide

49. IUUQTNPO[PDPNCMPHPODBMM
    8BSOJOH
    "IVNBOSFTQPOTFJTSFRVJSFE CVUOPUVSHFOUMZ
    5IFTFTIPVMEO`UXBLFZPVBUOJHIU
    CVUTIPVMECFMPPLFEBUXJUIJOFJHIUIPVST
    

    View Slide

50. IUUQTNPO[PDPNCMPHPODBMM
    $SJUJDBM
    "OVSHFOUSFTQPOTFJTSFRVJSFE
    BOETIPVMEHFUUIFBUUFOUJPOPGBOPODBMMFSJNNFEJBUFMZ
    

    View Slide

51. 4-*TBOE4-0T
    

    View Slide

52. 6TF3&%NFUSJDTGPS4-*T
    

    View Slide

53. &TQFDJBMMZ XFVTF&SSPSTBOE%VSBUJPOGPS4-*T
    

    View Slide

54. View Slide

55. View Slide

56. %FDJEJOH4-0JTOPUFBTZ
    

    View Slide

57. 8FQSPWJEFBSPVHIFYQFSJNFOUBMHVJEFMJOF
    

    View Slide

58. View Slide

59. View Slide

60. 0O$BMM
    

    View Slide

61. 8F`SFBEPQUJOHlZPVCVJMEJU ZPVSVOJUzDVMUVSF
    

    View Slide

62. 1BHFS%VUZVTFST
    

    View Slide

63. .BOZFOHJOFFSTOPXCFDPNF0O$BMM
    1MBUGPSNUFBNOFFETUPQSFQBSFUIF0O$BMMUSBJOJOH
    

    View Slide

64. 5ISFFMBZFST FTDBMBUFTBGUFSNJOT
    

    View Slide

65. IUUQTSFTQPOTFQBHFSEVUZDPNPODBMMCFJOH@PODBMM
    

    View Slide

66. View Slide

67. %PO`UXBLFVQFOHJOFFSTJONJEOJHIUGPSXBSOJOHBMFSUT
    

    View Slide

68. .POJUPSJOHTZTUFN
    .JDSPTFSWJDF
    $IBUOPUJpDBUJPO
    4.4QVTIOPUJpDBUJPO
    1IPOF$BMM
    $SJUJDBMBMFSU
    8BSOJOHBMFSU
    

    View Slide

69. /FYUUSZJTJOUSPEVDJOH*ODJEFOU$PNNBOEFSSPMF
    

    View Slide

70. 3FTQPOTJCJMJUZ%FTJHO
    

    View Slide

71. ,VCFSOFUFT/PEF
    .JDSPTFSWJDFQPET
    .POJUPSJOHBHFOUQPE
    5SBDFTTUPSBHF
    %FWFMPQFSTNPOJUPSUIFJSNJDSPTFSWJDFT
    

    View Slide

72. ,VCFSOFUFT/PEF
    .JDSPTFSWJDFQPET
    .POJUPSJOHBHFOUQPE
    5SBDFTTUPSBHF
    1MBUGPSNUFBNNPOJUPSTDPNNPOQBSUT
    

    View Slide

73. ,VCFSOFUFT/PEF
    .JDSPTFSWJDFQPET
    .POJUPSJOHBHFOUQPE
    5SBDFTTUPSBHF
    1MBUGPSNUFBNLFFQTNPOJUPSJOHFOWJSPONFOUXPSLT
    

    View Slide

74. 1MBZCPPL
    

    View Slide

75. 8F`WFTUBSUFEXSJUJOHQMBZCPPLTBTFWFOJGJUJTWFSZ
    SPVHI JU`TCFUUFSUIBOOPUIJOH
    

    View Slide

76. 8FDBOOPUQSFQBSFXFMMEFTDSJCFEQMBZCPPLTJOPWFSOJHIU
    

    View Slide

77. $POUJOVPVTQMBZCPPLTJNQSPWFNFOUXJUIHJU
    

    View Slide

78. View Slide

79. View Slide

80. 7JTVBMJ[BUJPO
    

    View Slide

81. )BWFBTJOHMFQVSQPTFGPSBHSBQI
    

    View Slide

82. .VMUJQVSQPTFHSBQIJTIBSEUPFWBMVBUF
    

    View Slide

83. View Slide

84. *GZPVIBWFBOPUIFSQVSQPTF
    BEEPOFNPSFHSBQIUPTIPXJU
    

    View Slide

85. View Slide

86. 4FFBTBNFNFUSJDGSPNEJ⒎FSFOUBOHMFT
    

    View Slide

87. 4FFBTBNFNFUSJDGSPNEJ⒎FSFOUBOHMFT
    

    View Slide

88. 4FFBTBNFNFUSJDGSPNEJ⒎FSFOUBOHMFT
    

    View Slide

89. 3FDBQ
    "MXBZTTUBSUGSPNl8IZXFNPOJUPS z
    $PMMFDUNFUSJDT USBDFTBOEMPHT
    *OWFTUJHBUFXJUI3&%64&NFUIPE
    6TF3&%GPSBMFSUJOH
    %FpOF4-0XJUI3&%
    &WFSZEFWFMPQFSEPFTNPOJUPSJOH
    )BWFBTJOHMFQVSQPTFGPSBHSBQI
    

    View Slide

90. 5IBOLZPVGPSTIBSJOHCFBVUJGVMQJDUVSFT
    0MFH.BHOJ 4IVZV,JN ,BU+BZOF +BWPO4XBCZ "EPOZJ(ÂCPS
    

    View Slide