Data aggregation design with Goroutine, Channel, "sync" package, and Simple diagram

GoConference ’19 Summer in Fukuoka

DEMOはこちら https://gist.github.com/k1LoW/732360772475203fd6b7d8eb62309c83

Transcript

1. খࢁ݈Ұ࿠(.01FQBCP *OD
   (P$POGFSFODF`4VNNFSJO'VLVPLB
   %BUBBHHSFHBUJPOEFTJHO
   XJUI(PSPVUJOF $IBOOFM
   TZODQBDLBHF
   BOE4JNQMFEJBHSBN
   

   View Slide

2. ϓϦϯγύϧΤϯδχΞ
   খࢁ݈Ұ࿠!L-P8
   ϗεςΟϯάࣄۀ෦ϗεςΟϯάάϧʔϓϚωʔδυΫϥ΢υνʔϜ
   IUUQTHJUIVCDPNL-P8
   
   
   

   View Slide

3. w UCMTJTB$*'SJFOEMZUPPMGPS
   EPDVNFOUBEBUBCBTF XSJUUFOJO(P
   w σʔλϕʔεʹ઀ଓͯ͠υΩϡϝϯτΛ
   ੜ੒ͯ͘͠ΕΔίϚϯυϥΠϯπʔϧ
   w 1PTUHSF42-.Z42-42-JUF42-4FSWFS
   #JH2VFSZ"NB[PO3FETIJGU
   w IUUQTHJUIVCDPNL-P8UCMT
   

   View Slide

4. View Slide

5. View Slide

6. %BUBBHHSFHBUJPO
   

   View Slide

7. %BUBBHHSFHBUJPO
   wʮσʔλΛऩूͯ͠ॲཧ͢Δʯ
   wେن໛ͳσʔλू໿ج൫ͷ͓࿩Ͱ͸͋Γ·ͤΜɻ͝ΊΜͳ͍͞
   wຊηογϣϯͰ͸ɺ୯७ʹʮೖྗ͕͋ͬͯɺͦΕΛ͍͍ײ͡ʹॲཧ͢
   ΔɻٻΊΔग़ྗΛಘΔʯͱ͍͏΋ͷΛࢦ͠·͢
   

   View Slide

8. Ϟνϕʔγϣϯ
   wαʔόʔΦϖϨʔγϣϯΛΑΓޮ཰Խ͍ͨ͠
   wʮٕܳͷίʔυʹΑΔϘτϜΞοϓʯ
   wʮϩάʯಛʹੜϩάΛޮՌతʹऔΓѻ͑ΔΑ͏ʹͳΕ͹ɺΦϖϨʔ
   γϣϯ΋ޮ཰తʹͳΔͷͰ͸ʁ
   wϩάͷऔΓѻ͍ʮೖྗ͕͋ͬͯɺͦΕΛ͍͍ײ͡ʹॲཧͯ͠ɺٻΊ
   Δग़ྗΛಘΔʯ
   

   View Slide

9. ྫ'MVFOUE
   w ೖྗΛड͚ͯʢJOQVUʣ
   w ॲཧΛͯ͠ʢpMUFSCV⒎FSSPVUJOH
   
   w ग़ྗ͢ΔʢPVUQVUʣ
   l8IBUJT'MVFOUE zΑΓIUUQTXXXqVFOUEPSHBSDIJUFDUVSF
   

   View Slide

10. ೖྗॲཧग़ྗ
    

    View Slide

11. ೖྗॲཧग़ྗ
    wσʔλΛऩू͢Δͱ͍͏͜ͱ͸ɺͲ͔͜ʹσʔλ͕͋Δ
    wͦͷσʔλΛೖྗͱͯ͠ॲཧ͍ͯ͘͜͠ͱʹͳΔ
    wٻΊΔग़ྗ͸σʔλͷ֨ೲ΍ɺ୯७ʹඪ४ग़ྗ΁ͷදࣔͳͲɻ
    w֨ೲ΋ɺೖྗσʔλΛͦͷ··֨ೲ͢Δ͜ͱ΋͋Ε͹ɺϑΟϧλ΍Ճ޻ͷॲ
    ཧ͕͸͍Δ͜ͱ΋͋Δ
    wೖྗΛޮ཰తʹॲཧ͠ग़ྗ͢Δ͜ͱΛߟ͑Δͱɺඞવతʹೖྗ͔Βग़ྗ·Ͱͷ
    ؒ͸ετϦʔϜσʔλͷॲཧʹͳ͍ͬͯ͘
    σʔλऩूॲཧ͸ύΠϓϥΠϯॲཧ
    

    View Slide

12. ύΠϓϥΠϯ
    wೖྗͱॲཧͱग़ྗΛεςʔδͱͯ͠෼͚ͯߟ͑Δ
    w ,BUIFSJOF$PY#VEBZஶɺࢁޱೳ᫫༁ʮ(PݴޠʹΑΔฒߦॲཧʯύΠϓϥΠϯ
    ೖྗ ॲཧ ग़ྗ
    ೖྗ ॲཧ ग़ྗ
    ॲཧ
    εςʔδ
    εςʔδ
    ྫ
    

    View Slide

13. ύΠϓϥΠϯͱฒߦॲཧ
    wεςʔδͦΕͧΕͰฒߦʹॲཧΛ૸ΒͤΔ
    w(PSPVUJOF
    wσʔλ͸ͦΕͧΕͷεςʔδͰฒߦʹॲཧ͞Εͭͭɺ࣍ͷεςʔδ΁ྲྀΕ͍ͯ
    ͘
    

    View Slide

14. εςʔδ4UBHF
    wೖྗΛॲཧͯ͠ग़ྗ͢Δͷ͕·ͱ·Γʹͳ͍ͬͯΔ
    ग़ྗ
    JP3FBEFSPSDIBO
    ೖྗ
    JP3FBEFSPSDIBO
    ೖྗ͔ΒͷσʔλΛ଴ͪड͚ͯ
    ೖྗ͕དྷͨΒ౎౓ॲཧΛͯ͠
    ग़ྗʹॲཧͨ͠σʔλΛॻ͖ࠐΉ
    

    View Slide

15. εςʔδͱεςʔδΛͭͳ͛Δ
    wલஈͷग़ྗͷܕͱޙஈͷೖྗͷܕ͕߹͍ͬͯΕ͹ܨ͛ΒΕΔ
    ೖྗ
    JP3FBEFS
    ग़ྗ
    JP3FBEFS
    

    View Slide

16. 4UBHFJP3FBEFSWFS
    w ೖྗJP3FBEFS
    w ग़ྗJP3FBEFS
    w )BOEMF಺ͰJP1JQF
    Ͱ࿈݁
    w DBODFM
    ͕ൃՐͨ͠Βऴྃ
    w ೖྗ͔ΒJP&0'͕དྷͯ΋ऴྃ
    ˞஫ҙޙड़͠·͢
    

    View Slide

17. 4UBHFDIBOWFS
    w ೖྗDIBO*O
    w ग़ྗDIBO0VU
    w DUY%POF
    ͕ൃՐͨ͠Βऴྃ
    w DIBO*O͕DMPTFͯ͠΋ऴྃ
    ˞஫ҙޙड़͠·͢
    

    View Slide

18. ͪΐͬͱ଴ͬͨʂ
    

    View Slide

19. ⚠
    %POPUTUPSF$POUFYUTJOTJEFBTUSVDUUZQF
    JOTUFBE QBTTB$POUFYUFYQMJDJUMZUPFBDI
    GVODUJPOUIBUOFFETJU
    IUUQTHPMBOHPSHQLHDPOUFYU
    ߏ଄ମʹ$POUFYU͸࣋ͨͤͳ͍Α͏ʹ͍ͯͩ͘͠͞
    ୅ΘΓʹ$POUFYUΛඞཁͱ͍ͯ͠Δؔ਺ʹ౉͍ͯͩ͘͠͞
    

    View Slide

20. ͝ΊΜͳ͍͞
    

    View Slide

21. 4UBHFJP3FBEFSWFS
    w ೖྗJP3FBEFS
    w ग़ྗJP3FBEFS
    w )BOEMF಺ͰJP1JQF
    Ͱ࿈݁
    w DBODFM
    ͕ൃՐͨ͠Βऴྃ
    w ೖྗ͔ΒJP&0'͕དྷͯ΋ऴྃ
    

    View Slide

22. 4UBHFDIBOWFS
    w ೖྗDIBO*O
    w ग़ྗDIBO0VU
    w DUY%POF
    ͕ൃՐͨ͠Βऴྃ
    w DIBO*O͕DMPTFͯ͠΋ऴྃ
    

    View Slide

23. ύΠϓϥΠϯͷਖ਼ৗऴྃ
    wग़ྗΛࢭΊΒΕΔͷ͸ɺͦͷग़ྗΛੜ੒͍ͯ͠Δεςʔδ
    wύΠϓϥΠϯͷ࠷ॳͷεςʔδͷग़ྗ͕ऴྃ͢Ε͹ɺ࣍ʑʹ࿈࠯ͯ͠
    εςʔδ͕ऴྃ͢Δ
    wJP&0'ɺDMPTF DIBO
    
    

    View Slide

24. ύΠϓϥΠϯͷΩϟϯηϧ
    wύΠϓϥΠϯͷ్தͷεςʔδΛ͍͖ͳΓΩϟϯηϧͰ͖ͯ͠·͏ͷ
    ͸ྑ͘ͳ͍
    લஈͷεςʔδ͸ग़ྗΛ΍Ίͯ͘Εͳ͍
    ޙஈͷεςʔδͷ(PSPVUJOF΋ࢭ·ͬͯ͘Εͳ͍
    wύΠϓϥΠϯͷશͯΛΩϟϯηϧͰ͖ΔΑ͏ʹ͢Δ
    DPOUFYU$POUFYUɺDPOUFYU$BODFM'VOD
    શͯͷεςʔδʹDPOUFYU$POUFYUΛ౉͓ͯ͘͠
    

    View Slide

25. $POUFYUUSFF
    w $POUFYUͱ͔ͦ͜Β೿ੜͨ͠$POUFYUͰ਌ࢠؔ܎ͷπϦʔ͕ܗ੒͞ΕΔΠϝʔδ
    w $POUFYU8JUI$BODFM
    w ֤$POUFYU೿ੜ$POUFYUͷ$BODFM'VODͷ఻ൖൣғΛ͏·͘ར༻ͯ͠ύΠϓϥ
    Πϯʹ૊ΈࠐΜͰ͍͘
    

    View Slide

26. έʔεελσΟ
    

    View Slide

27. ඪ४ग़ྗʹ৭Λ͚͍ͭͨ
    

    View Slide

28. DPMS
    w೚ҙͷ਺ͷจࣈྻΛࢦఆ࣮ͯ͠ߦ͢Δͱɺඪ४ೖྗͰड͚औͬͨετ
    ϦʔϜʹͦͷจࣈྻ͕͋ͬͨΒ೚ҙͷ৭Λ͚ͭͯ͘ΕΔ
    wIUUQTHJUIVCDPNL-P8DPMS
    wUBJM'͍ͯ͠Δϩάͷจࣈྻʹ৭Λ͚ͭͯϩάͷ܏޲ͳͲΛࢹ֮తʹ
    Θ͔Γ΍͔ͨͬͨ͘͢͠
    wطଘπʔϧ͸ଘࡏ͔͕ͨͬͨ͠ɺ୯७Ͱࣗ෼ͷ࢖͍উख͕ྑ͍΋ͷΛ
    ࡞ͬͨ
    

    View Slide

29. %&.0
    

    View Slide

30. DPMS
    wೖྗʜඪ४ೖྗ
    wॲཧʜࢦఆͨ͠จࣈྻʹ৭Λ͚ͭΔ
    wग़ྗʜඪ४ग़ྗ
    

    View Slide

31. DPMS
    

    View Slide

32. UBJM'ͷετϦʔϜΛࢭΊͣʹ
    Կ౓΋HSFQΛࢼ͍ͨ͠
    

    View Slide

33. pMU
    wετϦʔϜͷύΠϓઌʹpMUΛࢦఆ͓ͯ͘͜͠ͱͰɺετϦʔϜΛࢭΊ
    ͣʹύΠϓઌͷίϚϯυΛԿ౓Ͱ΋มߋͰ͖Δ
    wIUUQTHJUIVCDPNL-P8pMU
    wHSFQ΍BXL΍TFE͕Լखͦ͘ͳͷͰɺUSJBMFSSPSͰࢼ͔ͨͬͨ͠
    wTUFSOͳͲɺ1PEʹϩά͕ཷ·͍ͬͯΔͱɺ࠷ॳͷग़ྗ·Ͱʹ࣌ؒ
    ͕͔͔ΔͷͰɺ$USM$ͰࢭΊΔͷΛ᪳᪯͢Δ͕࣌͋ͬͨ
    

    View Slide

34. %&.0
    

    View Slide

35. pMU
    w ೖྗʜඪ४ೖྗ
    w ॲཧʜ
    w ͦͷ··ೖྗΛग़ྗʹͭͳ͛Δ
    w αϒϓϩηεʹೖྗΛ౉͠ɺίϚϯυΛ࣮ߦͨ݁͠ՌΛग़ྗʹͭͳ͛Δ
    w ग़ྗʜ
    w ඪ४ग़ྗ
    w EFWOVMM
    

    View Slide

36. pMU
    

    View Slide

37. pMU
    wೖྗͷྲྀྔΛ͏·੍͘ޚͰ͖Ε͹ɺεςʔδΛ੾Γସ͑ΔΑ͏ͳύΠ
    ϓϥΠϯΛߏஙՄೳ
    wpMUͷ৔߹͸ϓϩϯϓτ͕ग़͍ͯΔؒ͸ग़ྗΛEFWOVMMʹࣺ͍ͯͯ
    Δ
    w1JQF͢Δ4VCQSPDFTT΋෇͚֎͠Ͱ͖ΔΑ͏ʹ͍ͯ͠Δ
    

    View Slide

38. ෺ཧαʔόͷϩά΋
    LT1PEͷϩά΋
    ೚ҙͷλʔήοτΛࢦఆͯ͠
    ԣஅతʹ࣌ܥྻͰऔಘͨ͠Γ
    UBJM'ͨ͠Γ͍ͨ͠
    

    View Slide

39. )BSWFTU
    w͋Β͔͡ΊઃఆΛ༻ҙ͓ͯ͘͜͠ͱͰɺෳ਺αʔό΍ίϯςφʹ෼ࢄ
    ͨ͠ϩά͔Β೚ҙͷϩάΛࢦఆͯ͠ͷऔಘ΍UBJM͕Ͱ͖Δϙʔλϒϧͳ
    ϩάऩूπʔϧ
    wIUUQTHJUIVCDPNL-P8IBSWFTU
    wΤʔδΣϯτϨεɻ
    w44)઀ଓͰ͖Δαʔό͚ͩͰͳ͘,VCFSOFUFTͷ1PEͷϩά΋αϙʔ
    τɻͦΕͧΕΛ۠ผ͢Δ͜ͱͳ͘ಁաతʹऔಘՄೳɻ
    

    View Slide

40. %&.0
    

    View Slide

41. )BSWFTU
    

    View Slide

42. )BSWFTU
    

    View Slide

43. )BSWFTU
    w ೖྗʜ
    w αʔό಺ͷϩάϑΝΠϧʢ44)ͰϩάΠϯʣ
    w ,VCFSOFUFT"1*ܦ༝Ͱͷ1PEͷϩά
    w ॲཧʜ
    w ϩά͔Β࣌ࠁΛύʔεͯ͠औಘ͢Δ
    w ग़ྗʜ
    w ࡞੒ͨ͠42-JUFʹϩά৘ใΛ*/4&35͢Δ
    w ඪ४ग़ྗ
    

    View Slide

44. )BSWFTU
    

    View Slide

45. TZOD8BJU(SPVQ
    wTZOD8BJU(SPVQͰϑΝϯΞ΢
    τɾϑΝϯΠϯΛ࣮૷
    w$IBOOFMͰฒྻ౓Λௐ੔
    ࢀߟIUUQTNBUUOLBPSJZBOFUTPGUXBSFMBOHHPIUN
    

    View Slide

46. UBJM'ͷ݁ՌΛ
    /ඵ͝ͱʹΧ΢ϯτ͍ͨ͠
    

    View Slide

47. FWSZ
    wඪ४ೖྗͷ಺༰ΛɺҰఆ࣌ؒ͝ͱʢ΋͘͠͸Ұఆߦ͝ͱʣʹ·ͱΊͯ
    ࢦఆͷίϚϯυʹ౉͠ɺͦͷ࣮ߦ݁ՌΛγʔέϯγϟϧʹग़ྗ͢Δί
    Ϛϯυ
    wIUUQTHJUIVCDPNL-P8FWSZ
    

    View Slide

48. %&.0
    

    View Slide

49. FWSZ
    

    View Slide

50. FWSZ
    

    View Slide

51. ฒྻ࣮ߦͭͭ͠
    ݩͷॱͰγʔέϯγϟϧʹग़ྗ͢Δ
    

    View Slide

52. TZOD.VUFY
    

    View Slide

53. ϩοΫ෇͖࣮ߦ݁ՌσʔλΧϓηϧ
    wίϚϯυͷ࣮ߦ݁ՌΛอ࣋͢Δ
    $BQTVMFΛ༻ҙ
    w/FX$BQTVMF
    ͰD-PDL
    ͯ͠౉͢
    wD0VUQVU
    Λ࣮ߦͯ͠΋ɺD6OMPDL
    
    ͕͞ΕΔ·Ͱग़ྗ͞Εͳ͍
    wग़ྗΛ଴ͭ
    

    View Slide

54. 4FD4QMJUUFS
    wFYFDVUFS&YFDVUFS D ETU
    ͷதͰɺ
    ίϚϯυͷ࣮ߦ݁Ռ͕ͰͨΒ
    D6OMPDL
    ͕ൃՐ
    UJDLFSͷॱͰγʔέϯγϟϧʹग़ྗ
    

    View Slide

55. ·ͱΊ
    

    View Slide

56. ·ͱΊ
    wσʔλऩूॲཧ͸ύΠϓϥΠϯ
    wࣗ෼ͷதͰύλʔϯΛ࡞ΕΕ͹ॻ͚Δɻಈ͘
    w(Pͷ͓͔͛Ͱɺϓϩάϥϛϯάͷ෯͕޿͕ͬͨɻͰ͖Δ͜ͱ͕޿
    ͕ͬͨɻ΍Γ͍ͨ͜ͱ͕޿͕ͬͨ
    

    View Slide

57. (PݴޠʹΑΔฒߦॲཧ
    w ຊηογϣϯͰ͓࿩͢Δ͜ͱͷ΄ͱ
    ΜͲ͕ຊॻ͔Βಘͨ஌ݟ͕ݩʹͳͬ
    ͍ͯΔ
    w ಘͨ஌ݟΛݩʹ࣮૷ͨ͠Β͏·͘
    ͍ͬͨ
    w ຊ౰ʹΦεεϝʂʂ
    

    View Slide

58. 5IBOLZPV
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU
    

    View Slide