Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Study Golang by developing mini crawler

Study Golang by developing mini crawler

Published at gocon 2018 spring

Bo0km4n

April 15, 2018
Tweet

More Decks by Bo0km4n

Other Decks in Programming

Transcript

  1. Ϋϩʔϥʔ࣮૷ͰֶͿGo
    Katsuya Kawabe
    2018/4/15(೔) @ GoCon 2018 Spring
    1

    View full-size slide

  2. ࣗݾ঺հ ɹ
    2
    •Name := ઒෦উ໵
    •Twitter := @KKawabe108
    •GitHub := @Bo0km4n
    ✦ ॴଐ
    - ࣳӜ޻ۀେֶେֶӃ ిؾిࢠ৘ใ޻ֶઐ߈ M1
    - גࣜձࣾ3-shake (Πϯλʔϯ)
    ✦ ීஈͷ׆ಈ
    - 3-shakeͰDMPͷόοΫΤϯυΛgoͰΨϦΨϦॻ͍ͯ·͢
    - ࠷ۙ͸झຯͰ8086ͷσΟεΞηϯϒϥΛgoͰ࣮૷ͨ͠Γ

    View full-size slide

  3. LTͷྲྀΕ ɹ
    3
    1. LTͷ૝ఆର৅ऀ
    2. Ϋϩʔϥʔͷ࢓༷
    3. ֶ΂Δ͜ͱ
    4. σϞ
    5. ·ͱΊ

    View full-size slide

  4. ର৅ऀ
    4
    GoΛֶΜͰΈ͍͚ͨͲ͍
    ͖ͳΓOSSΛ৮Δͷ͸࠳
    ંͦ͠͏ɻɻɻ
    ͱΓ͋͑ͣԿ͔࡞ͬͯ
    Goͷײ֮Λ௫Έ͍͚ͨͲ
    ΞΠσΟΞ͕෸͔ͳ͍ɻɻɻ
    GoΛֶͼ͍ͨܟ伲ͳΤϯδχΞ
    Ճ͑ͯ…
    ৽ਓϓϩάϥϜͷڭҭΛ୲͏ϕςϥϯΤϯδχΞ

    View full-size slide

  5. Ϋϩʔϥʔͷ࣮૷ͱ͍͏͓୊
    5
    ڭत
    ֶੜ
    ɾ৽͍͠ϓϩάϥϛϯάݴޠͷษڧ
    ɾ৽ਓ΍ֶੜͷϕϯνϚʔΫʹ΋ͳΔ
    Ϋϩʔϥʔ࡞Γͳ͍͞
    Ϋϩʔϥʔͷ࢓༷

    View full-size slide

  6. 6
    Ϋϩʔϥͷ࢓༷
    WebϖʔδͷURLΛࢦఆͯ͠ϖʔδΛϩʔΧϧʹอଘɻ
    ϩʔΧϧͰhtmlΛ։͍ͯݟͨ໨Λ࠶ݱ͢Δ

    View full-size slide

  7. Ϋϩʔϥ࣮૷ͷྲྀΕ Ϋϩʔϥʔͷ࢓༷
    7
    ̍ஈ֊ URLΛࢦఆͯ̍͠ϖʔδͷΈ
    ΫϩʔϦϯά͢Δ
    ̎ஈ֊ ֊૚Λࢦఆͯ̍͠ϖʔδ໨ͷϦϯΫઌ͔
    Β࠶ؼతʹΫϩʔϦϯά͢Δ
    ̏ஈ֊ ͜ΕΒͷॲཧΛฒྻԽ͠ɺߴ଎Խ͢Δ

    View full-size slide

  8. 8
    ̍ஈ֊ ̍ϖʔδͷΈΫϩʔϦϯά͢Δ

    View full-size slide

  9. ̍ஈ֊໨
    9
    HTMLύʔαϥΠϒϥϦΛ༻͍ͯɺ
    ࢦఆͨ͠WebϖʔδΛϩʔΧϧʹอଘͯ͠
    ݟͨ໨Λ࠶ݱ͢Δɻ
    html, javascript, css, imgϑΝΠϧΛμ΢ϯϩʔυ
    ※ htmlͷύεΛϩʔΧϧͷύεʹॻ͖׵͑Δ
    Ϋϩʔϥʔͷ࢓༷
    ͭ·Γ…

    View full-size slide

  10. ̍ஈ֊໨
    10
    Ϋϩʔϥʔͷ࢓༷








    Replace
    ύεΛFQDNʹඥ͚ͮͯμ΢ϯϩʔυ͢Δ

    View full-size slide

  11. 11
    ̎ஈ֊
    ֊૚Λࢦఆͯ̍͠ϖʔδ໨ͷϦϯΫઌ͔Β
    ࠶ؼతʹΫϩʔϦϯά͢Δ

    View full-size slide

  12. ̎ஈ֊໨
    12
    $ go run main.go -root https://golang.org/doc/ -depth 2
    Root (Depth 0)
    Depth 1
    Depth 2
    rootͷhtmlʹషΒΕ͍ͯΔ
    ϦϯΫͷϖʔδΛ
    Ϋϩʔϧ͢Δ
    ى఺ͱͳΔϖʔδ͸
    https://golang.org/doc
    Ϋϩʔϧ͢Δਂ͞͸̎
    Ϋϩʔϥʔͷ࢓༷

    View full-size slide

  13. 13
    ̏ஈ֊ ͜ΕΒͷॲཧΛฒྻԽͯ͠ߴ଎ԽΛܭΔ

    View full-size slide

  14. ̏ஈ֊໨ Ϋϩʔϥͷ࢓༷
    14
    cpus := runtime.NumCPU()
    runtime.GOMAXPROCS(cpus)
    c := make(chan bool, cpus)
    for i := 0; i < ֊૚਺; i++ {
    var wg sync.WaitGroup
    for _, p := range webϖʔδͷ഑ྻ {
    c <- true
    wg.Add(1)
    go func(p Page) {
    defer func() { <-c }()
    // ΫϩʔϦϯά
    wg.Done()
    }(p)
    }
    wg.Wait()
    }
    goroutineΛ༻͍Δ͜ͱͰൺֱత؆୯ʹ
    ฒྻॲཧΛ࣮૷Ͱ͖Δ
    ※ go func()಺ͷॲཧ͕͍ܰͱ
    Ή͠Ζ஗͘ͳΔՄೳੑ΋͋Δ
    ※ ੜ੒͢Δgoroutineͷ਺΋੍ޚ͠ͳ͍ͱ
    Ϋϩʔϧ͢Δϖʔδ͕ଟ͍৔߹panic͢Δ͜ͱ΋͋Δ
    ͔͠͠

    View full-size slide

  15. ֶ΂Δ͜ͱ
    • GoͰͷιϑτ΢ΣΞઃܭ(ύοέʔδ΍ߏ଄ମ, ΠϯλʔϑΣʔε)
    • จࣈྻॲཧ (ύεपΓͰ݁ߏటष͍ॲཧΛ͢Δඞཁ͕͋Δ)
    • ඇಉظॲཧ (goroutineͷجຊతͳ࢖͍ํ)
    • ϑΝΠϧೖग़ྗ
    15

    View full-size slide

  16. ·ͱΊ
    17
    Go binary
    Goroutine
    Goroutine
    Goroutine
    Web Page Web Page
    Web Page
    HTML
    Img
    Css
    JS
    HTML
    Img
    Css
    JS HTML
    Img
    Css
    JS
    Download
    Download
    Download

    View full-size slide

  17. ·ͱΊ
    • ࠓճ࣮૷ͨ͠Ϋϩʔϥʔ͸ https://github.com/Bo0km4n/avarus
    ʹ͋Γ·͢
    • ߦ਺͸600ߦ͙Β͍Ͱ͓खܰͰ͢(೔਺Ͱݴ͑͹Ұि͙ؒΒ͍ʁ)
    • ଞͷݴޠΛֶͿࡍʹ΋༗ޮͩͱࢥ͍·͢
    • ੋඇ͓Ջͳ࣌ʹଉൈ͖ʹͰ΋࣮૷ͯ͠Έ͍ͯͩ͘͞ʂʂ
    18

    View full-size slide

  18. 19
    Have a good Golang life!!!
    Thank you for listening.

    View full-size slide