Study Golang by developing mini crawler

Published at gocon 2018 spring

Transcript

1. Ϋϩʔϥʔ࣮૷ͰֶͿGo
   Katsuya Kawabe
   2018/4/15(೔) @ GoCon 2018 Spring
   1
   

   View Slide

2. ࣗݾ঺հ ɹ
   2
   •Name := ઒෦উ໵
   •Twitter := @KKawabe108
   •GitHub := @Bo0km4n
   ✦ ॴଐ
   - ࣳӜ޻ۀେֶେֶӃ ిؾిࢠ৘ใ޻ֶઐ߈ M1
   - גࣜձࣾ3-shake (Πϯλʔϯ)
   ✦ ීஈͷ׆ಈ
   - 3-shakeͰDMPͷόοΫΤϯυΛgoͰΨϦΨϦॻ͍ͯ·͢
   - ࠷ۙ͸झຯͰ8086ͷσΟεΞηϯϒϥΛgoͰ࣮૷ͨ͠Γ
   

   View Slide

3. LTͷྲྀΕ ɹ
   3
   1. LTͷ૝ఆର৅ऀ
   2. Ϋϩʔϥʔͷ࢓༷
   3. ֶ΂Δ͜ͱ
   4. σϞ
   5. ·ͱΊ
   

   View Slide

4. ର৅ऀ
   4
   GoΛֶΜͰΈ͍͚ͨͲ͍
   ͖ͳΓOSSΛ৮Δͷ͸࠳
   ંͦ͠͏ɻɻɻ
   ͱΓ͋͑ͣԿ͔࡞ͬͯ
   Goͷײ֮Λ௫Έ͍͚ͨͲ
   ΞΠσΟΞ͕෸͔ͳ͍ɻɻɻ
   GoΛֶͼ͍ͨܟ伲ͳΤϯδχΞ
   Ճ͑ͯ…
   ৽ਓϓϩάϥϜͷڭҭΛ୲͏ϕςϥϯΤϯδχΞ
   

   View Slide

5. Ϋϩʔϥʔͷ࣮૷ͱ͍͏͓୊
   5
   ڭत
   ֶੜ
   ɾ৽͍͠ϓϩάϥϛϯάݴޠͷษڧ
   ɾ৽ਓ΍ֶੜͷϕϯνϚʔΫʹ΋ͳΔ
   Ϋϩʔϥʔ࡞Γͳ͍͞
   Ϋϩʔϥʔͷ࢓༷
   

   View Slide

6. 6
   Ϋϩʔϥͷ࢓༷
   WebϖʔδͷURLΛࢦఆͯ͠ϖʔδΛϩʔΧϧʹอଘɻ
   ϩʔΧϧͰhtmlΛ։͍ͯݟͨ໨Λ࠶ݱ͢Δ
   

   View Slide

7. Ϋϩʔϥ࣮૷ͷྲྀΕ Ϋϩʔϥʔͷ࢓༷
   7
   ̍ஈ֊ URLΛࢦఆͯ̍͠ϖʔδͷΈ
   ΫϩʔϦϯά͢Δ
   ̎ஈ֊ ֊૚Λࢦఆͯ̍͠ϖʔδ໨ͷϦϯΫઌ͔
   Β࠶ؼతʹΫϩʔϦϯά͢Δ
   ̏ஈ֊ ͜ΕΒͷॲཧΛฒྻԽ͠ɺߴ଎Խ͢Δ
   

   View Slide

8. 8
   ̍ஈ֊ ̍ϖʔδͷΈΫϩʔϦϯά͢Δ
   

   View Slide

9. ̍ஈ֊໨
   9
   HTMLύʔαϥΠϒϥϦΛ༻͍ͯɺ
   ࢦఆͨ͠WebϖʔδΛϩʔΧϧʹอଘͯ͠
   ݟͨ໨Λ࠶ݱ͢Δɻ
   html, javascript, css, imgϑΝΠϧΛμ΢ϯϩʔυ
   ※ htmlͷύεΛϩʔΧϧͷύεʹॻ͖׵͑Δ
   Ϋϩʔϥʔͷ࢓༷
   ͭ·Γ…
   

   View Slide

10. ̍ஈ֊໨
    10
    Ϋϩʔϥʔͷ࢓༷
    
    
    
    
    
    
    
    
    Replace
    ύεΛFQDNʹඥ͚ͮͯμ΢ϯϩʔυ͢Δ
    

    View Slide

11. 11
    ̎ஈ֊
    ֊૚Λࢦఆͯ̍͠ϖʔδ໨ͷϦϯΫઌ͔Β
    ࠶ؼతʹΫϩʔϦϯά͢Δ
    

    View Slide

12. ̎ஈ֊໨
    12
    $ go run main.go -root https://golang.org/doc/ -depth 2
    Root (Depth 0)
    Depth 1
    Depth 2
    rootͷhtmlʹషΒΕ͍ͯΔ
    ϦϯΫͷϖʔδΛ
    Ϋϩʔϧ͢Δ
    ى఺ͱͳΔϖʔδ͸
    https://golang.org/doc
    Ϋϩʔϧ͢Δਂ͞͸̎
    Ϋϩʔϥʔͷ࢓༷
    

    View Slide

13. 13
    ̏ஈ֊ ͜ΕΒͷॲཧΛฒྻԽͯ͠ߴ଎ԽΛܭΔ
    

    View Slide

14. ̏ஈ֊໨ Ϋϩʔϥͷ࢓༷
    14
    cpus := runtime.NumCPU()
    runtime.GOMAXPROCS(cpus)
    c := make(chan bool, cpus)
    for i := 0; i < ֊૚਺; i++ {
    var wg sync.WaitGroup
    for _, p := range webϖʔδͷ഑ྻ {
    c <- true
    wg.Add(1)
    go func(p Page) {
    defer func() { <-c }()
    // ΫϩʔϦϯά
    wg.Done()
    }(p)
    }
    wg.Wait()
    }
    goroutineΛ༻͍Δ͜ͱͰൺֱత؆୯ʹ
    ฒྻॲཧΛ࣮૷Ͱ͖Δ
    ※ go func()಺ͷॲཧ͕͍ܰͱ
    Ή͠Ζ஗͘ͳΔՄೳੑ΋͋Δ
    ※ ੜ੒͢Δgoroutineͷ਺΋੍ޚ͠ͳ͍ͱ
    Ϋϩʔϧ͢Δϖʔδ͕ଟ͍৔߹panic͢Δ͜ͱ΋͋Δ
    ͔͠͠
    

    View Slide

15. ֶ΂Δ͜ͱ
    • GoͰͷιϑτ΢ΣΞઃܭ(ύοέʔδ΍ߏ଄ମ, ΠϯλʔϑΣʔε)
    • จࣈྻॲཧ (ύεपΓͰ݁ߏటष͍ॲཧΛ͢Δඞཁ͕͋Δ)
    • ඇಉظॲཧ (goroutineͷجຊతͳ࢖͍ํ)
    • ϑΝΠϧೖग़ྗ
    15
    

    View Slide

16. σϞ
    16
    

    View Slide

17. ·ͱΊ
    17
    Go binary
    Goroutine
    Goroutine
    Goroutine
    Web Page Web Page
    Web Page
    HTML
    Img
    Css
    JS
    HTML
    Img
    Css
    JS HTML
    Img
    Css
    JS
    Download
    Download
    Download
    

    View Slide

18. ·ͱΊ
    • ࠓճ࣮૷ͨ͠Ϋϩʔϥʔ͸ https://github.com/Bo0km4n/avarus
    ʹ͋Γ·͢
    • ߦ਺͸600ߦ͙Β͍Ͱ͓खܰͰ͢(೔਺Ͱݴ͑͹Ұि͙ؒΒ͍ʁ)
    • ଞͷݴޠΛֶͿࡍʹ΋༗ޮͩͱࢥ͍·͢
    • ੋඇ͓Ջͳ࣌ʹଉൈ͖ʹͰ΋࣮૷ͯ͠Έ͍ͯͩ͘͞ʂʂ
    18
    

    View Slide

19. 19
    Have a good Golang life!!!
    Thank you for listening.
    

    View Slide