How we monitor microservices at Mercari microservices platform team

How we monitor microservices at Mercari microservices platform team

Talk at SRE session #1
https://sre-lounge.connpass.com/event/123701/

Explained the overview of our microservices monitoring.

32f2e5ddb187baa2abac66d7e8b283fe?s=128

Seigo Uchida

April 10, 2019
Tweet

Transcript

  1. )PXXFNPOJUPS NJDSPTFSWJDFT BU.FSDBSJ.JDSPTFSWJDFT1MBUGPSNUFBN

  2. !TQFTOPWB

  3. $VSSFOU4UBUVT   FOHJOFFST   NJDSPTFSWJDFT   DIBOHFTXFFL

  4. ,VCFSOFUFT$MVTUFS (,& $VSSFOU4UBDL %# 4QBOOFSFUD .JDSPTFSWJDFT (PBQQFUD

  5. "SDIJUFDUVSF0WFSWJFX ($-# DMJFOU "1* HBUFXBZ #'' NJDSPTFSWJDFT NJDSPTFSWJDFT %#T

  6. )PXXFNPOJUPS NJDSPTFSWJDFT

  7. 4UBSUJOHGSPN l8IZXFNPOJUPS z

  8. 8FXBOUUPLFFQTZTUFNTXPSLQSPQFSMZ 8IZ

  9. 8IFOBTZTUFNJTOPUXPSLJOHQSPQFSMZ XFOFFEUP EFUFDUJU JOWFTUJHBUFJUBOEpYJU

  10. 4P IPXDBOXFEFUFDUBTZTUFNJTOUXPSLJOH

  11. $PMMFDUNFUSJDT UPLOPXlXIBUz

  12. .FUSJDT

  13. ,VCFSOFUFT/PEF .JDSPTFSWJDFQPET .POJUPSJOHBHFOUQPE $PMMFDUJOHBSDIJUFDUVSFPWFSWJFX .FUSJDTTUPSBHF

  14. .FUSJDTUFMMZPVXIFUIFSBTZTUFNJTXPSLJOHQSPQFSMZPSOPU

  15. 5IJTNFUSJDUFMMZPVBTZTUFNXBTTMPXUIFO

  16. 5IJTNFUSJDUFMMZPVBTZTUFNSFUVSOFENBOZFSSPST

  17. :PVDBOLOPXBTZTUFNJTXPSLJOHCZDIFDLJOHNFUSJDT

  18. .FUSJDTEPO`UUFMMZPVIPXTMPXBOEXIZTMPX )PXFWFS

  19. $PMMFDUUSBDFT UPLOPXlIPXz

  20. 5SBDFT

  21. ,VCFSOFUFT/PEF .JDSPTFSWJDFQPET .POJUPSJOHBHFOUQPE $PMMFDUJOHBSDIJUFDUVSFPWFSWJFX 5SBDFTTUPSBHF

  22. 5SBDFTUFMMZPVIPXTMPX IPXCSPLFO

  23. 5IJTUSBDFUFMMZPVUIFSFEQBSUUPPLNPTUPGUJNF

  24. 5IJTUSBDFUFMMZPVXIFSFBOFSSPSIBQQFOFE

  25. 1SPQBHBUJOHASFRVFTU@JEAGPSEJTUSJCVUFEUSBDJOH

  26. 5PQPMPHZ TFSWJDF NBQ

  27. 5SBDFTEPO`UUFMMZPVXIZ )PXFWFS

  28. $PMMFDUMPHT UPLOPXlXIZz

  29. -PHT

  30. ,VCFSOFUFT/PEF .JDSPTFSWJDFQPET .POJUPSJOHBHFOUQPE $PMMFDUJOHBSDIJUFDUVSFPWFSWJFX -PHTTUPSBHF

  31. -PHTUFMMZPVXIZJUIBQQFOFE

  32. 1SPQBHBUJOHASFRVFTU@JEAGPSEJTUSJCVUFEMPHHJOH

  33. 4FBSDIJOHMPHTXJUIlSFRVFTU@JEYYYYzBDSPTTNJDSPTFSWJDFT

  34. %FpOJOHUIFMPHGPSNBUTUBOEBSEUPUPBOBMZ[FBDSPTTTFSWJDFT

  35. None
  36. &BDINJDSPTFSWJDFTIPVMECFBCMFUPEFpOFJUTPXOMPHGPSNBU  TPUIBUUIFTUBOEBSEEFpOFTPOMZDPNNPOBUUSJCVUFOBNFT

  37. &OGPSDJOHUIFTUBOEBSEUPIVOESFETEFWFMPQFSTUBLFTUJNF #FQBUJFOUBOEIBWFBMPOHUFSNWJTJPO

  38. 5IFUISFFQJMMBS PGPCTFSWBCJMJUZ 8IBUEBUBEPXFDPMMFDU

  39. 5IFUISFFQJMMBS PGPCTFSWBCJMJUZ .FUSJDT 5SBDFT -PHT

  40. 5IF3&%64&.FUIPE PS8PSLBOE3FTPVSDF.FUIPE )PXEPXFVTFUIFEBUB

  41. 3FRVFTUT &SSPST %VSBUJPO XPSLNFUSJDT

  42. 6UJMJ[BUJPO 4BUVSBUJPO &SSPST SFTPVSDFNFUSJDT

  43. IUUQTXXXEBUBEPHIRDPNCMPHNPOJUPSJOHJOWFTUJHBUJPO *OWFTUJHBUJPO'MPX  $IFDL3&% XPSL NFUSJDTUPLOPX8)"5  $IFDLUSBDFTUPLOPX)08  $IFDL64&

    SFTPVSDF NFUSJDTUPLOPX8):  $IFDLMPHTUPLOPX8):
  44. IUUQTXXXEBUBEPHIRDPNCMPHNPOJUPSJOHJOWFTUJHBUJPO *OWFTUJHBUJPO'MPX

  45. "MFSUJOH

  46. 6TF3&%NFUSJDTGPSBMFSUJOH

  47. 8FEPO`UVTF64&NFUSJDTGPSBMFSUJOH UPBWPJEGBMTFBMFSUJONPTUDBTFT

  48. .POJUPSJOHTZTUFN .JDSPTFSWJDF $IBUOPUJpDBUJPO 4.4QVTIOPUJpDBUJPO 1IPOF$BMM $SJUJDBMBMFSU 8BSOJOHBMFSU

  49. IUUQTNPO[PDPNCMPHPODBMM 8BSOJOH "IVNBOSFTQPOTFJTSFRVJSFE CVUOPUVSHFOUMZ 5IFTFTIPVMEO`UXBLFZPVBUOJHIU  CVUTIPVMECFMPPLFEBUXJUIJOFJHIUIPVST

  50. IUUQTNPO[PDPNCMPHPODBMM $SJUJDBM "OVSHFOUSFTQPOTFJTSFRVJSFE  BOETIPVMEHFUUIFBUUFOUJPOPGBOPODBMMFSJNNFEJBUFMZ

  51. 4-*TBOE4-0T

  52. 6TF3&%NFUSJDTGPS4-*T

  53. &TQFDJBMMZ XFVTF&SSPSTBOE%VSBUJPOGPS4-*T

  54. None
  55. None
  56. %FDJEJOH4-0JTOPUFBTZ

  57. 8FQSPWJEFBSPVHIFYQFSJNFOUBMHVJEFMJOF

  58. None
  59. None
  60. 0O$BMM

  61. 8F`SFBEPQUJOHlZPVCVJMEJU ZPVSVOJUzDVMUVSF

  62.  1BHFS%VUZVTFST

  63. .BOZFOHJOFFSTOPXCFDPNF0O$BMM  1MBUGPSNUFBNOFFETUPQSFQBSFUIF0O$BMMUSBJOJOH

  64. 5ISFFMBZFST FTDBMBUFTBGUFSNJOT

  65. IUUQTSFTQPOTFQBHFSEVUZDPNPODBMMCFJOH@PODBMM

  66. None
  67. %PO`UXBLFVQFOHJOFFSTJONJEOJHIUGPSXBSOJOHBMFSUT

  68. .POJUPSJOHTZTUFN .JDSPTFSWJDF $IBUOPUJpDBUJPO 4.4QVTIOPUJpDBUJPO 1IPOF$BMM $SJUJDBMBMFSU 8BSOJOHBMFSU

  69. /FYUUSZJTJOUSPEVDJOH*ODJEFOU$PNNBOEFSSPMF

  70. 3FTQPOTJCJMJUZ%FTJHO

  71. ,VCFSOFUFT/PEF .JDSPTFSWJDFQPET .POJUPSJOHBHFOUQPE 5SBDFTTUPSBHF %FWFMPQFSTNPOJUPSUIFJSNJDSPTFSWJDFT

  72. ,VCFSOFUFT/PEF .JDSPTFSWJDFQPET .POJUPSJOHBHFOUQPE 5SBDFTTUPSBHF 1MBUGPSNUFBNNPOJUPSTDPNNPOQBSUT

  73. ,VCFSOFUFT/PEF .JDSPTFSWJDFQPET .POJUPSJOHBHFOUQPE 5SBDFTTUPSBHF 1MBUGPSNUFBNLFFQTNPOJUPSJOHFOWJSPONFOUXPSLT

  74. 1MBZCPPL

  75. 8F`WFTUBSUFEXSJUJOHQMBZCPPLTBTFWFOJGJUJTWFSZ SPVHI JU`TCFUUFSUIBOOPUIJOH

  76. 8FDBOOPUQSFQBSFXFMMEFTDSJCFEQMBZCPPLTJOPWFSOJHIU

  77. $POUJOVPVTQMBZCPPLTJNQSPWFNFOUXJUIHJU

  78. None
  79. None
  80. 7JTVBMJ[BUJPO

  81. )BWFBTJOHMFQVSQPTFGPSBHSBQI

  82. .VMUJQVSQPTFHSBQIJTIBSEUPFWBMVBUF

  83. None
  84. *GZPVIBWFBOPUIFSQVSQPTF  BEEPOFNPSFHSBQIUPTIPXJU

  85. None
  86. 4FFBTBNFNFUSJDGSPNEJ⒎FSFOUBOHMFT

  87. 4FFBTBNFNFUSJDGSPNEJ⒎FSFOUBOHMFT

  88. 4FFBTBNFNFUSJDGSPNEJ⒎FSFOUBOHMFT

  89. 3FDBQ "MXBZTTUBSUGSPNl8IZXFNPOJUPS z $PMMFDUNFUSJDT USBDFTBOEMPHT *OWFTUJHBUFXJUI3&%64&NFUIPE 6TF3&%GPSBMFSUJOH %FpOF4-0XJUI3&% &WFSZEFWFMPQFSEPFTNPOJUPSJOH )BWFBTJOHMFQVSQPTFGPSBHSBQI

  90. 5IBOLZPVGPSTIBSJOHCFBVUJGVMQJDUVSFT  0MFH.BHOJ 4IVZV,JN ,BU+BZOF +BWPO4XBCZ "EPOZJ(ÂCPS