Study Golang by developing mini crawler

Published at gocon 2018 spring

Transcript

1. Ϋϩʔϥʔ࣮૷ͰֶͿGo Katsuya Kawabe 2018/4/15(೔) @ GoCon 2018 Spring 1
2. ࣗݾ঺հ ɹ 2 •Name := ઒෦উ໵ •Twitter := @KKawabe108 •GitHub := @Bo0km4n ✦ ॴଐ - ࣳӜ޻ۀେֶେֶӃ ిؾిࢠ৘ใ޻ֶઐ߈ M1 - גࣜձࣾ3-shake (Πϯλʔϯ) ✦ ීஈͷ׆ಈ - 3-shakeͰDMPͷόοΫΤϯυΛgoͰΨϦΨϦॻ͍ͯ·͢ - ࠷ۙ͸झຯͰ8086ͷσΟεΞηϯϒϥΛgoͰ࣮૷ͨ͠Γ
3. LTͷྲྀΕ ɹ 3 1. LTͷ૝ఆର৅ऀ 2. Ϋϩʔϥʔͷ࢓༷ 3. ֶ΂Δ͜ͱ 4. σϞ 5. ·ͱΊ
4. ର৅ऀ 4 GoΛֶΜͰΈ͍͚ͨͲ͍ ͖ͳΓOSSΛ৮Δͷ͸࠳ ંͦ͠͏ɻɻɻ ͱΓ͋͑ͣԿ͔࡞ͬͯ Goͷײ֮Λ௫Έ͍͚ͨͲ ΞΠσΟΞ͕෸͔ͳ͍ɻɻɻ GoΛֶͼ͍ͨܟ伲ͳΤϯδχΞ Ճ͑ͯ… ৽ਓϓϩάϥϜͷڭҭΛ୲͏ϕςϥϯΤϯδχΞ
5. Ϋϩʔϥʔͷ࣮૷ͱ͍͏͓୊ 5 ڭत ֶੜ ɾ৽͍͠ϓϩάϥϛϯάݴޠͷษڧ ɾ৽ਓ΍ֶੜͷϕϯνϚʔΫʹ΋ͳΔ Ϋϩʔϥʔ࡞Γͳ͍͞ Ϋϩʔϥʔͷ࢓༷
6. 6 Ϋϩʔϥͷ࢓༷ WebϖʔδͷURLΛࢦఆͯ͠ϖʔδΛϩʔΧϧʹอଘɻ ϩʔΧϧͰhtmlΛ։͍ͯݟͨ໨Λ࠶ݱ͢Δ
7. Ϋϩʔϥ࣮૷ͷྲྀΕ Ϋϩʔϥʔͷ࢓༷ 7 ̍ஈ֊ URLΛࢦఆͯ̍͠ϖʔδͷΈ ΫϩʔϦϯά͢Δ ̎ஈ֊ ֊૚Λࢦఆͯ̍͠ϖʔδ໨ͷϦϯΫઌ͔ Β࠶ؼతʹΫϩʔϦϯά͢Δ ̏ஈ֊ ͜ΕΒͷॲཧΛฒྻԽ͠ɺߴ଎Խ͢Δ
8. 8 ̍ஈ֊ ̍ϖʔδͷΈΫϩʔϦϯά͢Δ
9. ̍ஈ֊໨ 9 HTMLύʔαϥΠϒϥϦΛ༻͍ͯɺ ࢦఆͨ͠WebϖʔδΛϩʔΧϧʹอଘͯ͠ ݟͨ໨Λ࠶ݱ͢Δɻ html, javascript, css, imgϑΝΠϧΛμ΢ϯϩʔυ ※ htmlͷύεΛϩʔΧϧͷύεʹॻ͖׵͑Δ Ϋϩʔϥʔͷ࢓༷ ͭ·Γ…
10. ̍ஈ֊໨ 10 Ϋϩʔϥʔͷ࢓༷ <a href="hoge/fuga.html"> <img src="img/gocon.png"> <script src="https://gocon/src.js"></script> <link rel="stylesheet" type="text/css" href="gocon.css"> <a href="/Users/gocon/hoge/fuga.html"> <img src="/Users/gocon/img/gocon.png"> <script src="/Users/gocon/src.js"></script> <link rel="stylesheet" type="text/css" href="/Users/gocon.css"> Replace ύεΛFQDNʹඥ͚ͮͯμ΢ϯϩʔυ͢Δ
11. 11 ̎ஈ֊ ֊૚Λࢦఆͯ̍͠ϖʔδ໨ͷϦϯΫઌ͔Β ࠶ؼతʹΫϩʔϦϯά͢Δ
12. ̎ஈ֊໨ 12 $ go run main.go -root https://golang.org/doc/ -depth 2 Root (Depth 0) Depth 1 Depth 2 rootͷhtmlʹషΒΕ͍ͯΔ ϦϯΫͷϖʔδΛ Ϋϩʔϧ͢Δ ى఺ͱͳΔϖʔδ͸ https://golang.org/doc Ϋϩʔϧ͢Δਂ͞͸̎ Ϋϩʔϥʔͷ࢓༷
13. 13 ̏ஈ֊ ͜ΕΒͷॲཧΛฒྻԽͯ͠ߴ଎ԽΛܭΔ
14. ̏ஈ֊໨ Ϋϩʔϥͷ࢓༷ 14 cpus := runtime.NumCPU() runtime.GOMAXPROCS(cpus) c := make(chan bool, cpus) for i := 0; i < ֊૚਺; i++ { var wg sync.WaitGroup for _, p := range webϖʔδͷ഑ྻ { c <- true wg.Add(1) go func(p Page) { defer func() { <-c }() // ΫϩʔϦϯά wg.Done() }(p) } wg.Wait() } goroutineΛ༻͍Δ͜ͱͰൺֱత؆୯ʹ ฒྻॲཧΛ࣮૷Ͱ͖Δ ※ go func()಺ͷॲཧ͕͍ܰͱ Ή͠Ζ஗͘ͳΔՄೳੑ΋͋Δ ※ ੜ੒͢Δgoroutineͷ਺΋੍ޚ͠ͳ͍ͱ Ϋϩʔϧ͢Δϖʔδ͕ଟ͍৔߹panic͢Δ͜ͱ΋͋Δ ͔͠͠
15. ֶ΂Δ͜ͱ • GoͰͷιϑτ΢ΣΞઃܭ(ύοέʔδ΍ߏ଄ମ, ΠϯλʔϑΣʔε) • จࣈྻॲཧ (ύεपΓͰ݁ߏటष͍ॲཧΛ͢Δඞཁ͕͋Δ) • ඇಉظॲཧ (goroutineͷجຊతͳ࢖͍ํ) • ϑΝΠϧೖग़ྗ 15
16. σϞ 16
17. ·ͱΊ 17 Go binary Goroutine Goroutine Goroutine Web Page Web Page Web Page HTML Img Css JS HTML Img Css JS HTML Img Css JS Download Download Download
18. ·ͱΊ • ࠓճ࣮૷ͨ͠Ϋϩʔϥʔ͸ https://github.com/Bo0km4n/avarus ʹ͋Γ·͢ • ߦ਺͸600ߦ͙Β͍Ͱ͓खܰͰ͢(೔਺Ͱݴ͑͹Ұि͙ؒΒ͍ʁ) • ଞͷݴޠΛֶͿࡍʹ΋༗ޮͩͱࢥ͍·͢ • ੋඇ͓Ջͳ࣌ʹଉൈ͖ʹͰ΋࣮૷ͯ͠Έ͍ͯͩ͘͞ʂʂ 18
19. 19 Have a good Golang life!!! Thank you for listening.