みんなのIMAPを可視化する / Visualize IMAP Everybody

Fukuoka.go#16 で発表した資料です。Junly 14, 2020

Transcript

1. Fukuoka.go#16 Tomohisa Oda / July 14, 2020 Visualize
   IMAP Everybody
2. Fukuoka.go#16 @linyows Blog: tomohisaoda.com Affiliation: GMO pepabo, inc.
3. Fukuoka.go#16 ๏ ϗεςΟϯάࣄۀ෦
   Senior Engineering Lead ࣄۀ෦ͷCTO΍VPoEͷΑ͏ͳ΋ͷ
   ๏ ϖύϘݚڀॴ
   ۝भେֶڞಉݚڀ
   2020
   Պݚඅ࠾୒
   IUUQTLBLFOOJJBDKQHSBOU,",&/)*130+&$5,
   ܰྔίϯςφʹΑΔେن໛ߴूੵϝʔϧϗεςΟϯάج൫ʹ͓͚Δૹ৴ػೳͷߴػೳԽ In Pepabo…
4. Fukuoka.go#16 Private Projects ΞϓϦέʔγϣϯࣗ਎͕ϦϙδτϦ͔Β࠷৽ͷίʔυΛݕ ஌ͯࣗ͠ಈͰσϓϩΠͱσϦόϦʔΛߦ͏πʔϧ MJOVYͷVTFS໊લղܾΛ(JU)VCͷ5FBNϝϯόʔ΍Ϧϙδτ ϦͷΞΫηεݖݶऀ͔Βߦ͏πʔϧ
5. Fukuoka.go#16 Pepabo͸ϗετ͍ͯ͠·͢
   
6. Fukuoka.go#16 ๏ Domain — DNS ๏ Web — HTTP, SSH, FTP and WebDav… ๏ Mail — SMTP, POP, IMAP… ๏ Database — (MySQL, SQLite…) ϩϦϙοϓ΍ϔςϜϧ
   ͕ϗετ͍ͯ͠Δ΋ͷ ͍ΖΜͳٕज़Λѻ͍ͬͯ·͢
7. Fukuoka.go#16 ๏ Domain — DNS ๏ Web — HTTP, SSH, FTP and WebDav… ๏ Mail — SMTP, POP, IMAP… ๏ Database — (MySQL, SQLite…) ϩϦϙοϓ΍ϔςϜϧ
   ͕ϗετ͍ͯ͠Δ΋ͷ ຊ೔͸ίίͷ࿩
8. Fukuoka.go#16 Mail ࢖͍ͬͯ·͔͢ʁ
   
9. Fukuoka.go#16 ͍·Ͳ͖ϝʔϧʁ͔ͭͬͯͳ͍ʁ͍΍͍΍͍΍ʜ — Chat Type — Messaging Type
10. Fukuoka.go#16 ΄Βɺࢥ͍ग़͠·͔ͨ͠ʁͪΌΜͱ࢖͍ͬͯ·͢Ͷʁ
11. Fukuoka.go#16
    େมਃ͠༁͋Γ·ͤΜ
    
12. Fukuoka.go#16 େମ(NBJM
    PS
    (
    4VJUFͰ͢ΑͶʁ
13. Fukuoka.go#16 ͍ʹ͑͠ͷٕज़ͱ͸͍͑ɺMailʹมΘΔ
    ΦʔϓϯͳMessagingγεςϜ͸ͳ͍
    
14. Fukuoka.go#16 Mailͷ࢓૊Έ
    
15. Fukuoka.go#16 Architecture RFC 5598 ʹ͋Γ·͕͢ʜ
16. Fukuoka.go#16 Architecture ΋ͬͱγϯϓϧͳ΍ͭ
    ͜ΕͰ࠷খߏ੒ͳͷͰࠜຊతʹ
    ΢ΣϒΑΓෳࡶ Mail Submission Agent :587 Mail Transferring Agent :25 Mail Delivery Agent :993 Postfix Postfix Dovecot ✉ ✉ SMTP SMTP LMTP IMAP
17. Fukuoka.go#16 ϩϦϙοϓ΍ϔςϜϧ
    ʹ͍ͭͯ
18. Fukuoka.go#16 Our Status 120 100k 2.2k Over /min ͜ͷΑ͏ͳن໛ײ
19. Fukuoka.go#16 Our Status 120 100k 2.2k Over /min ͜ͷΑ͏ͳن໛ײ ϝʔϧαʔό୆ʹ৐Δ
    ϝʔϧΞΧ΢ϯτ Max ϝʔϧαʔόͷ਺ ෼ؒʹૹΔϝʔϧ ௨਺ʢDeliveredʣ
20. Fukuoka.go#16 Our Challenges ๏ ૹ৴
    ๏ IP BanɺBlack ListೖΓ໰୊
    ΞΧ΢ϯτ৐ͬऔΓ΍ෆਖ਼ར༻໰୊
    ѱҙͷͳ͍େྔϝʔϧ഑৴
    ๏ ड৴
    ๏ ϑΟογϯάɺ΢ΟϧεɺεύϜϝʔϧഉআ໰୊
    ๏ Disk I/Oෛՙ໰୊
21. Fukuoka.go#16 ๏ ૹ৴
    ๏ IP BanɺBlack ListೖΓ໰୊
    ΞΧ΢ϯτ৐ͬऔΓ΍ෆਖ਼ར༻໰୊
    ѱҙͷͳ͍େྔϝʔϧ഑৴
    ๏ ड৴
    ๏ ϑΟογϯάɺ΢ΟϧεɺεύϜϝʔϧഉআ໰୊
    ๏ Disk I/Oෛՙ໰୊ Our Challenges
22. Fukuoka.go#16 ๏ ͍͍ετϨʔδΛങ͑͹ղܾ͢Δ
    ๏ ௿Ձ֨ଳͷαʔϏεͳͷͰݫ͍͠
    ๏ ݱঢ়ɺෛՙͷߴ͍ΞΧ΢ϯτΛҠઃͯ͠Ϧιʔε ফඅΛۉ͢ઓུ
    ๏ ࣌ʹ͸SFTUBSUʜ IMAP Disk I/O ໰୊
23. Fukuoka.go#16 ๏ ओʹPrometheusΛ࢖͍ͬͯΔ
    ๏ ֤Proxy΍ϝʔϧαʔόʹ͍Εͨnode_exporterʹΑΔϦιʔε؂ࢹ
    ๏ mtailͰϝʔϧΩϡʔ΍େखϓϩόΠμʔ΁ͷό΢ϯεϝʔϧͷ؂ࢹ
    ๏ blackbox_exporterͰ
    smtp, pop, imapͷPort؂ࢹ
    ݱঢ়ͷ؂ࢹ͸े෼Ͱ͸ͳ͍ Monitoring
24. Fukuoka.go#16
25. Fukuoka.go#16
26. Fukuoka.go#16
27. Fukuoka.go#16 imap_exporterΛ࡞ͬͨ
28. Fukuoka.go#16 ๏ ϛυϧ΢ΣΞʹґଘ͠ͳ͍ΑΓந৅తͳϓϩτίϧϨϕϧͳϝτϦΫεΛऔಘ͢Δ
    ๏ Mail User Agent͕ඞͣߦ͏Ұ࿈ͷίϚϯυΛൃߦ͔͔ͯͬͨ࣌ؒ͠ΛͱΔ
    ๏ Connection / Login / List / Select / Fetch / Logout ๏ IMAP Client͸ɺطଘϞδϡʔϧΛ࢖͏ͷͰͳࣗ͘લͰ࣮૷͢ΔʢIMAPͷؾ࣋ͪʹͳ Δʣ imap_exporter
29. Fukuoka.go#16 ๏ RFC2060ΛΑʔ͠ಡΉͧͱ ࢥͬͨΒ్தͰ૬౰ͳϘϦϡʔ Ϝ͕͋Δ͜ͱʹؾͮ͘
    ๏ SMTP͕ϖʔδ΄Ͳʹରͯ͠ ϖʔδ IMAPͷ࢓༷͸RFC2060
30. Fukuoka.go#16 ๏ ͜Μͳ͜ͱ΋͋Ζ͏͔ͱɺຊ୨͔ΒऔΓग़͢ ిࢠϝʔϧϓϩτίϧʕجຊɾ࣮૷ɾӡ༻ 2000/11
31. Fukuoka.go#16 ๏ IMAPʹ͸ͭͷεςʔτ͕͋Γ
    ๏ ະೝূ
    ๏ ೝূࡁ
    ๏ બ୒ࡁ
    ๏ ϩάΞ΢τ
    ๏ ͦΕͧΕͷεςʔτʹ͓͍ͯΫϥΠΞϯτ͕ൃߦ͢ΔλάΛ࢖ͬͯαʔόʔͱΫϥΠΞϯ τͷίϛϡχέʔγϣϯΛߦ͏ IMAP ίϚϯυ λά
32. Fukuoka.go#16 IMAP ΫϥΠΞϯτͷ࣮૷
33. Fukuoka.go#16 Ϩεϙϯεʹλά͕͋Δ͔Ͳ͏͔Λ
    ൑ఆ͍ͯ͠Δ͚ͩͷγϯϓϧͳ࣮૷
34. Fukuoka.go#16 telnet΍opensslΛ࢖͏Α͏ͳΠϝʔδͰίϚϯυΛ࣮ߦ͢Δ
35. Fukuoka.go#16 ๏ ࠨͷάϥϑ͸खݩ͔ΒαϯϓϧΞΧ΢ϯτʹ઀ ଓͨ͠ྫ
    ๏ ͜ΕΛશͯͷϝʔϧαʔόʹಋೖ͢Δ༧ఆ
    ๏ Disk I/OෛՙʹରԠ͢Δࢳޱ͕Έ͔ͭΕ͹ͱ͍ ͏ఔ౓
    ๏ IMAPͷঢ়ଶΛՄࢹԽ͢Δ͜ͱʹ͸ؒҧ͍ͳ͍
    ๏ Ұ࿈ͷಈ࡞ΛFirst Loading Latencyͱ͍͏໊ લΛ͚ͭͨ First Loading Latency
36. Fukuoka.go#16 ๏ imap_exporterΛ࡞ͬͯMail Delivery AgentΛՄࢹԽ͠Α͏ͱ͍ͯ͠Δ
    ๏ IMAPϓϩτίϧ͸SMTPͱҧͬͯ૬౰ෳࡶ
    ๏ ϓϩτίϧ࢓༷Λཧղ͢Δ͜ͱͰGoඪ४ϥΠϒϥϦͰͬ͘͞ͱΫϥΠΞϯτ࣮૷Ͱ͖ͨ
    ͜ͷΑ͏ʹɺEngineeringͰϗεςΟϯάͷϝʔϧΛࢧ͍͑ͯ·͢ɻ࠷ۙɺMREͱ͍͏৬छ ΛͿ্͍ͪ͛ͯ·͢ʂʂʂʂ Conclusion
37. Fukuoka.go#16 Messaging Reliability EngineeringʢMREʣ͸ɺਓͱਓͱ͕ιϑτ΢ΣΞΛ௨ͯ͡ ίϛϡχέʔγϣϯΛ͢ΔγεςϜʹ͓͍ͯɺεέʔϥϒϧͰ৴པੑͷߴ͍ιϑτ ΢ΣΞγεςϜΛ࡞੒͠·͢ɻMRE͸ɺΠϯϑϥετϥΫνϟͱӡ༻ΛదԠͤ͞Δ Site Reliability EngineeringͷจԽΛMessaging෼໺Ͱड͚ܧ͍ͩ΋ͷͰ͢ɻجຊ తͳ͜ͱ͸SREͱಉ͡Ͱ͕͢ɺѻ͏ٕज़͕ओʹϝʔϧͰɺϝʔϧͷӡ༻Λιϑτ΢Σ ΞͰΤϯδχΞϦϯά͢Δ͜ͱʹͳΓ·͢ɻʮMRE
    ϖύϘʯͰݕࡧʂ Messaging Reliability Engineeringͱ͸
38. Fukuoka.go#16 Thank you
    aʕ⊙౪⚆ʔ