Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data aggregation design with Goroutine, Channel, "sync" package, and Simple diagram

Data aggregation design with Goroutine, Channel, "sync" package, and Simple diagram

GoConference ’19 Summer in Fukuoka

DEMOはこちら https://gist.github.com/k1LoW/732360772475203fd6b7d8eb62309c83

Ken’ichiro Oyama

July 13, 2019
Tweet

More Decks by Ken’ichiro Oyama

Other Decks in Technology

Transcript

  1. খࢁ݈Ұ࿠(.01FQBCP *OD
    (P$POGFSFODF`4VNNFSJO'VLVPLB
    %BUBBHHSFHBUJPOEFTJHO
    XJUI(PSPVUJOF $IBOOFM
    TZODQBDLBHF
    BOE4JNQMFEJBHSBN

    View Slide

  2. ϓϦϯγύϧΤϯδχΞ
    খࢁ݈Ұ࿠!L-P8
    ϗεςΟϯάࣄۀ෦ϗεςΟϯάάϧʔϓϚωʔδυΫϥ΢υνʔϜ
    IUUQTHJUIVCDPNL-P8


    View Slide

  3. w UCMTJTB$*'SJFOEMZUPPMGPS
    EPDVNFOUBEBUBCBTF XSJUUFOJO(P
    w σʔλϕʔεʹ઀ଓͯ͠υΩϡϝϯτΛ
    ੜ੒ͯ͘͠ΕΔίϚϯυϥΠϯπʔϧ
    w 1PTUHSF42-.Z42-42-JUF42-4FSWFS
    #JH2VFSZ"NB[PO3FETIJGU
    w IUUQTHJUIVCDPNL-P8UCMT

    View Slide

  4. View Slide

  5. View Slide

  6. %BUBBHHSFHBUJPO

    View Slide

  7. %BUBBHHSFHBUJPO
    wʮσʔλΛऩूͯ͠ॲཧ͢Δʯ
    wେن໛ͳσʔλू໿ج൫ͷ͓࿩Ͱ͸͋Γ·ͤΜɻ͝ΊΜͳ͍͞
    wຊηογϣϯͰ͸ɺ୯७ʹʮೖྗ͕͋ͬͯɺͦΕΛ͍͍ײ͡ʹॲཧ͢
    ΔɻٻΊΔग़ྗΛಘΔʯͱ͍͏΋ͷΛࢦ͠·͢

    View Slide

  8. Ϟνϕʔγϣϯ
    wαʔόʔΦϖϨʔγϣϯΛΑΓޮ཰Խ͍ͨ͠
    wʮٕܳͷίʔυʹΑΔϘτϜΞοϓʯ
    wʮϩάʯಛʹੜϩάΛޮՌతʹऔΓѻ͑ΔΑ͏ʹͳΕ͹ɺΦϖϨʔ
    γϣϯ΋ޮ཰తʹͳΔͷͰ͸ʁ
    wϩάͷऔΓѻ͍ʮೖྗ͕͋ͬͯɺͦΕΛ͍͍ײ͡ʹॲཧͯ͠ɺٻΊ
    Δग़ྗΛಘΔʯ

    View Slide

  9. ྫ'MVFOUE
    w ೖྗΛड͚ͯʢJOQVUʣ
    w ॲཧΛͯ͠ʢpMUFSCV⒎FSSPVUJOH

    w ग़ྗ͢ΔʢPVUQVUʣ
    l8IBUJT'MVFOUE zΑΓIUUQTXXXqVFOUEPSHBSDIJUFDUVSF

    View Slide

  10. ೖྗॲཧग़ྗ

    View Slide

  11. ೖྗॲཧग़ྗ
    wσʔλΛऩू͢Δͱ͍͏͜ͱ͸ɺͲ͔͜ʹσʔλ͕͋Δ
    wͦͷσʔλΛೖྗͱͯ͠ॲཧ͍ͯ͘͜͠ͱʹͳΔ
    wٻΊΔग़ྗ͸σʔλͷ֨ೲ΍ɺ୯७ʹඪ४ग़ྗ΁ͷදࣔͳͲɻ
    w֨ೲ΋ɺೖྗσʔλΛͦͷ··֨ೲ͢Δ͜ͱ΋͋Ε͹ɺϑΟϧλ΍Ճ޻ͷॲ
    ཧ͕͸͍Δ͜ͱ΋͋Δ
    wೖྗΛޮ཰తʹॲཧ͠ग़ྗ͢Δ͜ͱΛߟ͑Δͱɺඞવతʹೖྗ͔Βग़ྗ·Ͱͷ
    ؒ͸ετϦʔϜσʔλͷॲཧʹͳ͍ͬͯ͘
    σʔλऩूॲཧ͸ύΠϓϥΠϯॲཧ

    View Slide

  12. ύΠϓϥΠϯ
    wೖྗͱॲཧͱग़ྗΛεςʔδͱͯ͠෼͚ͯߟ͑Δ
    w ,BUIFSJOF$PY#VEBZஶɺࢁޱೳ᫫༁ʮ(PݴޠʹΑΔฒߦॲཧʯύΠϓϥΠϯ
    ೖྗ ॲཧ ग़ྗ
    ೖྗ ॲཧ ग़ྗ
    ॲཧ
    εςʔδ
    εςʔδ

    View Slide

  13. ύΠϓϥΠϯͱฒߦॲཧ
    wεςʔδͦΕͧΕͰฒߦʹॲཧΛ૸ΒͤΔ
    w(PSPVUJOF
    wσʔλ͸ͦΕͧΕͷεςʔδͰฒߦʹॲཧ͞Εͭͭɺ࣍ͷεςʔδ΁ྲྀΕ͍ͯ
    ͘

    View Slide

  14. εςʔδ4UBHF
    wೖྗΛॲཧͯ͠ग़ྗ͢Δͷ͕·ͱ·Γʹͳ͍ͬͯΔ
    ग़ྗ
    JP3FBEFSPSDIBO
    ೖྗ
    JP3FBEFSPSDIBO
    ೖྗ͔ΒͷσʔλΛ଴ͪड͚ͯ
    ೖྗ͕དྷͨΒ౎౓ॲཧΛͯ͠
    ग़ྗʹॲཧͨ͠σʔλΛॻ͖ࠐΉ

    View Slide

  15. εςʔδͱεςʔδΛͭͳ͛Δ
    wલஈͷग़ྗͷܕͱޙஈͷೖྗͷܕ͕߹͍ͬͯΕ͹ܨ͛ΒΕΔ
    ೖྗ
    JP3FBEFS
    ग़ྗ
    JP3FBEFS

    View Slide

  16. 4UBHFJP3FBEFSWFS
    w ೖྗJP3FBEFS
    w ग़ྗJP3FBEFS
    w )BOEMF಺ͰJP1JQF
    Ͱ࿈݁
    w DBODFM
    ͕ൃՐͨ͠Βऴྃ
    w ೖྗ͔ΒJP&0'͕དྷͯ΋ऴྃ
    ˞஫ҙޙड़͠·͢

    View Slide

  17. 4UBHFDIBOWFS
    w ೖྗDIBO*O
    w ग़ྗDIBO0VU
    w DUY%POF
    ͕ൃՐͨ͠Βऴྃ
    w DIBO*O͕DMPTFͯ͠΋ऴྃ
    ˞஫ҙޙड़͠·͢

    View Slide

  18. ͪΐͬͱ଴ͬͨʂ

    View Slide


  19. %POPUTUPSF$POUFYUTJOTJEFBTUSVDUUZQF
    JOTUFBE QBTTB$POUFYUFYQMJDJUMZUPFBDI
    GVODUJPOUIBUOFFETJU
    IUUQTHPMBOHPSHQLHDPOUFYU
    ߏ଄ମʹ$POUFYU͸࣋ͨͤͳ͍Α͏ʹ͍ͯͩ͘͠͞
    ୅ΘΓʹ$POUFYUΛඞཁͱ͍ͯ͠Δؔ਺ʹ౉͍ͯͩ͘͠͞

    View Slide

  20. ͝ΊΜͳ͍͞

    View Slide

  21. 4UBHFJP3FBEFSWFS
    w ೖྗJP3FBEFS
    w ग़ྗJP3FBEFS
    w )BOEMF಺ͰJP1JQF
    Ͱ࿈݁
    w DBODFM
    ͕ൃՐͨ͠Βऴྃ
    w ೖྗ͔ΒJP&0'͕དྷͯ΋ऴྃ

    View Slide

  22. 4UBHFDIBOWFS
    w ೖྗDIBO*O
    w ग़ྗDIBO0VU
    w DUY%POF
    ͕ൃՐͨ͠Βऴྃ
    w DIBO*O͕DMPTFͯ͠΋ऴྃ

    View Slide

  23. ύΠϓϥΠϯͷਖ਼ৗऴྃ
    wग़ྗΛࢭΊΒΕΔͷ͸ɺͦͷग़ྗΛੜ੒͍ͯ͠Δεςʔδ
    wύΠϓϥΠϯͷ࠷ॳͷεςʔδͷग़ྗ͕ऴྃ͢Ε͹ɺ࣍ʑʹ࿈࠯ͯ͠
    εςʔδ͕ऴྃ͢Δ
    wJP&0'ɺDMPTF DIBO

    View Slide

  24. ύΠϓϥΠϯͷΩϟϯηϧ
    wύΠϓϥΠϯͷ్தͷεςʔδΛ͍͖ͳΓΩϟϯηϧͰ͖ͯ͠·͏ͷ
    ͸ྑ͘ͳ͍
    લஈͷεςʔδ͸ग़ྗΛ΍Ίͯ͘Εͳ͍
    ޙஈͷεςʔδͷ(PSPVUJOF΋ࢭ·ͬͯ͘Εͳ͍
    wύΠϓϥΠϯͷશͯΛΩϟϯηϧͰ͖ΔΑ͏ʹ͢Δ
    DPOUFYU$POUFYUɺDPOUFYU$BODFM'VOD
    શͯͷεςʔδʹDPOUFYU$POUFYUΛ౉͓ͯ͘͠

    View Slide

  25. $POUFYUUSFF
    w $POUFYUͱ͔ͦ͜Β೿ੜͨ͠$POUFYUͰ਌ࢠؔ܎ͷπϦʔ͕ܗ੒͞ΕΔΠϝʔδ
    w $POUFYU8JUI$BODFM
    w ֤$POUFYU೿ੜ$POUFYUͷ$BODFM'VODͷ఻ൖൣғΛ͏·͘ར༻ͯ͠ύΠϓϥ
    Πϯʹ૊ΈࠐΜͰ͍͘

    View Slide

  26. έʔεελσΟ

    View Slide

  27. ඪ४ग़ྗʹ৭Λ͚͍ͭͨ

    View Slide

  28. DPMS
    w೚ҙͷ਺ͷจࣈྻΛࢦఆ࣮ͯ͠ߦ͢Δͱɺඪ४ೖྗͰड͚औͬͨετ
    ϦʔϜʹͦͷจࣈྻ͕͋ͬͨΒ೚ҙͷ৭Λ͚ͭͯ͘ΕΔ
    wIUUQTHJUIVCDPNL-P8DPMS
    wUBJM'͍ͯ͠Δϩάͷจࣈྻʹ৭Λ͚ͭͯϩάͷ܏޲ͳͲΛࢹ֮తʹ
    Θ͔Γ΍͔ͨͬͨ͘͢͠
    wطଘπʔϧ͸ଘࡏ͔͕ͨͬͨ͠ɺ୯७Ͱࣗ෼ͷ࢖͍উख͕ྑ͍΋ͷΛ
    ࡞ͬͨ

    View Slide

  29. %&.0

    View Slide

  30. DPMS
    wೖྗʜඪ४ೖྗ
    wॲཧʜࢦఆͨ͠จࣈྻʹ৭Λ͚ͭΔ
    wग़ྗʜඪ४ग़ྗ

    View Slide

  31. DPMS

    View Slide

  32. UBJM'ͷετϦʔϜΛࢭΊͣʹ
    Կ౓΋HSFQΛࢼ͍ͨ͠

    View Slide

  33. pMU
    wετϦʔϜͷύΠϓઌʹpMUΛࢦఆ͓ͯ͘͜͠ͱͰɺετϦʔϜΛࢭΊ
    ͣʹύΠϓઌͷίϚϯυΛԿ౓Ͱ΋มߋͰ͖Δ
    wIUUQTHJUIVCDPNL-P8pMU
    wHSFQ΍BXL΍TFE͕Լखͦ͘ͳͷͰɺUSJBMFSSPSͰࢼ͔ͨͬͨ͠
    wTUFSOͳͲɺ1PEʹϩά͕ཷ·͍ͬͯΔͱɺ࠷ॳͷग़ྗ·Ͱʹ࣌ؒ
    ͕͔͔ΔͷͰɺ$USM$ͰࢭΊΔͷΛ᪳᪯͢Δ͕࣌͋ͬͨ

    View Slide

  34. %&.0

    View Slide

  35. pMU
    w ೖྗʜඪ४ೖྗ
    w ॲཧʜ
    w ͦͷ··ೖྗΛग़ྗʹͭͳ͛Δ
    w αϒϓϩηεʹೖྗΛ౉͠ɺίϚϯυΛ࣮ߦͨ݁͠ՌΛग़ྗʹͭͳ͛Δ
    w ग़ྗʜ
    w ඪ४ग़ྗ
    w EFWOVMM

    View Slide

  36. pMU

    View Slide

  37. pMU
    wೖྗͷྲྀྔΛ͏·੍͘ޚͰ͖Ε͹ɺεςʔδΛ੾Γସ͑ΔΑ͏ͳύΠ
    ϓϥΠϯΛߏஙՄೳ
    wpMUͷ৔߹͸ϓϩϯϓτ͕ग़͍ͯΔؒ͸ग़ྗΛEFWOVMMʹࣺ͍ͯͯ
    Δ
    w1JQF͢Δ4VCQSPDFTT΋෇͚֎͠Ͱ͖ΔΑ͏ʹ͍ͯ͠Δ

    View Slide

  38. ෺ཧαʔόͷϩά΋
    LT1PEͷϩά΋
    ೚ҙͷλʔήοτΛࢦఆͯ͠
    ԣஅతʹ࣌ܥྻͰऔಘͨ͠Γ
    UBJM'ͨ͠Γ͍ͨ͠

    View Slide

  39. )BSWFTU
    w͋Β͔͡ΊઃఆΛ༻ҙ͓ͯ͘͜͠ͱͰɺෳ਺αʔό΍ίϯςφʹ෼ࢄ
    ͨ͠ϩά͔Β೚ҙͷϩάΛࢦఆͯ͠ͷऔಘ΍UBJM͕Ͱ͖Δϙʔλϒϧͳ
    ϩάऩूπʔϧ
    wIUUQTHJUIVCDPNL-P8IBSWFTU
    wΤʔδΣϯτϨεɻ
    w44)઀ଓͰ͖Δαʔό͚ͩͰͳ͘,VCFSOFUFTͷ1PEͷϩά΋αϙʔ
    τɻͦΕͧΕΛ۠ผ͢Δ͜ͱͳ͘ಁաతʹऔಘՄೳɻ

    View Slide

  40. %&.0

    View Slide

  41. )BSWFTU

    View Slide

  42. )BSWFTU

    View Slide

  43. )BSWFTU
    w ೖྗʜ
    w αʔό಺ͷϩάϑΝΠϧʢ44)ͰϩάΠϯʣ
    w ,VCFSOFUFT"1*ܦ༝Ͱͷ1PEͷϩά
    w ॲཧʜ
    w ϩά͔Β࣌ࠁΛύʔεͯ͠औಘ͢Δ
    w ग़ྗʜ
    w ࡞੒ͨ͠42-JUFʹϩά৘ใΛ*/4&35͢Δ
    w ඪ४ग़ྗ

    View Slide

  44. )BSWFTU

    View Slide

  45. TZOD8BJU(SPVQ
    wTZOD8BJU(SPVQͰϑΝϯΞ΢
    τɾϑΝϯΠϯΛ࣮૷
    w$IBOOFMͰฒྻ౓Λௐ੔
    ࢀߟIUUQTNBUUOLBPSJZBOFUTPGUXBSFMBOHHPIUN

    View Slide

  46. UBJM'ͷ݁ՌΛ
    /ඵ͝ͱʹΧ΢ϯτ͍ͨ͠

    View Slide

  47. FWSZ
    wඪ४ೖྗͷ಺༰ΛɺҰఆ࣌ؒ͝ͱʢ΋͘͠͸Ұఆߦ͝ͱʣʹ·ͱΊͯ
    ࢦఆͷίϚϯυʹ౉͠ɺͦͷ࣮ߦ݁ՌΛγʔέϯγϟϧʹग़ྗ͢Δί
    Ϛϯυ
    wIUUQTHJUIVCDPNL-P8FWSZ

    View Slide

  48. %&.0

    View Slide

  49. FWSZ

    View Slide

  50. FWSZ

    View Slide

  51. ฒྻ࣮ߦͭͭ͠
    ݩͷॱͰγʔέϯγϟϧʹग़ྗ͢Δ

    View Slide

  52. TZOD.VUFY

    View Slide

  53. ϩοΫ෇͖࣮ߦ݁ՌσʔλΧϓηϧ
    wίϚϯυͷ࣮ߦ݁ՌΛอ࣋͢Δ
    $BQTVMFΛ༻ҙ
    w/FX$BQTVMF
    ͰD-PDL
    ͯ͠౉͢
    wD0VUQVU
    Λ࣮ߦͯ͠΋ɺD6OMPDL

    ͕͞ΕΔ·Ͱग़ྗ͞Εͳ͍
    wग़ྗΛ଴ͭ

    View Slide

  54. 4FD4QMJUUFS
    wFYFDVUFS&YFDVUFS D ETU
    ͷதͰɺ
    ίϚϯυͷ࣮ߦ݁Ռ͕ͰͨΒ
    D6OMPDL
    ͕ൃՐ
    UJDLFSͷॱͰγʔέϯγϟϧʹग़ྗ

    View Slide

  55. ·ͱΊ

    View Slide

  56. ·ͱΊ
    wσʔλऩूॲཧ͸ύΠϓϥΠϯ
    wࣗ෼ͷதͰύλʔϯΛ࡞ΕΕ͹ॻ͚Δɻಈ͘
    w(Pͷ͓͔͛Ͱɺϓϩάϥϛϯάͷ෯͕޿͕ͬͨɻͰ͖Δ͜ͱ͕޿
    ͕ͬͨɻ΍Γ͍ͨ͜ͱ͕޿͕ͬͨ

    View Slide

  57. (PݴޠʹΑΔฒߦॲཧ
    w ຊηογϣϯͰ͓࿩͢Δ͜ͱͷ΄ͱ
    ΜͲ͕ຊॻ͔Βಘͨ஌ݟ͕ݩʹͳͬ
    ͍ͯΔ
    w ಘͨ஌ݟΛݩʹ࣮૷ͨ͠Β͏·͘
    ͍ͬͨ
    w ຊ౰ʹΦεεϝʂʂ

    View Slide

  58. 5IBOLZPV
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU

    View Slide