Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
pt&Goroutines
Search
monochromegane
May 31, 2014
Technology
20
15k
pt&Goroutines
pt(the_platinum_searcher) を高速化するために Goroutines まわりで試したことを発表しました。
http://connpass.com/event/6370/
monochromegane
May 31, 2014
Tweet
Share
More Decks by monochromegane
See All by monochromegane
Go言語での実装を通して学ぶLLMファインチューニングの仕組み / fukuokago22-llm-peft
monochromegane
0
100
不確実性下における目的と手段の統合的探索に向けた連続腕バンディットの応用 / iot70_gp_rff_mab
monochromegane
2
150
なめらかなシステムと運用維持の終わらぬ未来 / dicomo2025_coherently_fittable_system
monochromegane
0
2.5k
ベクトル検索システムの気持ち
monochromegane
36
11k
Go言語での実装を通して学ぶ、高速なベクトル検索を支えるクラスタリング技術/fukuokago-kmeans
monochromegane
1
210
Go言語でターミナルフレンドリーなAIコマンド、afaを作った/fukuokago20_afa
monochromegane
2
280
多様かつ継続的に変化する環境に適応する情報システム/thesis-defense-presentation
monochromegane
1
1k
Online Nonstationary and Nonlinear Bandits with Recursive Weighted Gaussian Process
monochromegane
0
650
AIを前提とした体験の実現に向けて/toward_ai_based_experiences
monochromegane
2
1k
Other Decks in Technology
See All in Technology
Goss: Faiss向けの新しい本番環境対応 Goバインディング #coefl_go_jp
bengo4com
1
1.4k
Oracle Cloud Infrastructure:2025年8月度サービス・アップデート
oracle4engineer
PRO
0
150
Grafana MCPサーバーによるAIエージェント経由でのGrafanaダッシュボード動的生成
hamadakoji
0
750
kintone開発チームの紹介
cybozuinsideout
PRO
0
73k
ヘブンバーンズレッドにおける、世界観を活かしたミニゲーム企画の作り方
gree_tech
PRO
0
320
Goss: New Production-Ready Go Binding for Faiss #coefl_go_jp
bengo4com
1
1.1k
MCPで変わる Amebaデザインシステム「Spindle」の開発
spindle
PRO
1
140
ライブサービスゲームQAのパフォーマンス検証による品質改善の取り組み
gree_tech
PRO
0
320
実践アプリケーション設計 ②トランザクションスクリプトへの対応
recruitengineers
PRO
4
1.1k
microCMS 最新リリース情報(microCMS Meetup 2025)
microcms
0
330
現場が抱える様々な問題は “組織設計上” の問題によって生じていることがある / Team-oriented Organization Design 20250827
mtx2s
7
66k
AWS環境のリソース調査を Claude Code で効率化 / aws investigate with cc devio2025
masahirokawahara
2
640
Featured
See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
51
5.6k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
How to train your dragon (web standard)
notwaldorf
96
6.2k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
RailsConf 2023
tenderlove
30
1.2k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Raft: Consensus for Rubyists
vanstee
140
7.1k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Optimising Largest Contentful Paint
csswizardry
37
3.4k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
6k
Bash Introduction
62gerente
614
210k
Transcript
pt &Goroutine - GoCon 2014 spring -
MIYAKE Yusuke (@monochromegane)
GMO Pepabo, Inc.
grep ͯ͠·͔͢ʁ
grep?
ack?
ag?
pt The Platinum Searcher
Written in Golang
Mac OSX Linux Windows
UTF-8 EUC-JP Shift-JIS
AND
fast ! ack go 6.24s user 1.06s system 99% cpu
7.304 total # ack ag go 0.88s user 1.39s system 221% cpu 1.027 total # ag pt go 1.09s user 1.01s system 235% cpu 0.892 total # pt
How?
Goroutine & Channel
͍ͬ͠ΐʹߴԽͯ͠Έ·͠ΐ͏
1. ϑΝΠϧΛݕࡧͯ͠(find) 2. จࣈྻΛݕࡧͯ͠(grep) 3. ݁ՌΛදࣔ͢Δ(print) ύλʔϯݕࡧͱ
Approach-0 ! ॱ൪ʹ
find
find grep
find grep print
// find find := find.Find{Option: self.Option} find.Do(self.Root) ! // grep
grep := grep.Grep{ Files: find.Files, // result Pattern: self.Pattern, Option: self.Option} grep.Do() ! // print print := print.Print{ Matches: grep.Matches, // result Pattern: self.Pattern, Option: self.Option} print.Do()
> the_simple_searcher go $GOROOT > /dev/null
0.79 seconds
Approach-1 ! ฒߦʹ
Goroutine
• GoݴޠͰฒߦॲཧΛ࣮ݱ͢Δ • εϨουɺίϧʔνϯͱҧ͏ • Concurrency(ฒߦ)ͱParallelism(ฒྻ) • ܰྔ • go
f()
find grep print go go go
Channel
• Goroutineؒͷϝοηʔδϯά • ͷૹड৴ • όοϑΝʹΑΔϒϩοΫ
find grep print go go go
$IBOFM find grep print $IBOFM go go go
$IBOFM find grep print $IBOFM go go go
// channel files := make(chan *string, self.Option.Cap) matches := make(chan
*grep.Match, self.Option.Cap) done := make(chan bool) ! // find find := find.Find{Files: files, Option: self.Option} go find.Do(self.Root) ! // grep grep := grep.Grep{ Files: files, Matches: matches, Pattern: self.Pattern, Option: self.Option} go grep.Do() ! // print print := print.Print{ Done: done, Matches: matches, Pattern: self.Pattern, Option: self.Option} go print.Do() ! <-done // block
walkFunc := func(path string, info os.FileInfo, err error) error {
if info.IsDir() { return nil } self.Files <- &path // send return nil } ! filepath.Walk(root, walkFunc) close(self.Files) // close
for file := range self.Files { // receive ( <-self.Files
) fh, err := os.Open(*file) if err != nil { panic(err) } ! f := bufio.NewReader(fh) ! var buf []byte var lineNum = 1 for { buf, _, err = f.ReadLine() if err != nil { break } line := string(buf) if strings.Contains(line, self.Pattern) { self.Matches <- &Match{*file, lineNum, line} // send } lineNum++ } fh.Close() } close(self.Matches) // close
for match := range self.Matches { // receive fmt.Printf("%s:%d:%s\n", match.Path,
match.Num, match.Match) } self.Done <- true // send
> the_simple_searcher go $GOROOT > /dev/null
0.79 -> 0.87 seconds
?
buffer
• Channelͷड༰ྔ • ch := make(chan ܕ, ༰ྔ) • ༰ྔ·Ͱड
• ༰ྔ͑Δͱૹ৴ଆडͪ • ड৴͢Δͱ༰ྔ͕ͻͱۭͭ͘ • ༰ྔ͕0ͷ߹ɺৗʹͭ
// channel with buffer files := make(chan *string, self.Option.Cap) matches
:= make(chan *grep.Match, self.Option.Cap) done := make(chan bool) // always wait
> the_simple_searcher go $GOROOT > /dev/null
0.79 -> 0.8 seconds
Approach-2 ! ͬͱฒߦʹ
$IBOFM find grep print $IBOFM go go go
$IBOFM find grep print $IBOFM go go go grep grep
grep
var wg sync.WaitGroup for file := range self.Files { wg.Add(1)
// goroutineͷىಈΛΠϯΫϦϝϯτ (தུ) go func(self *Grep, file *string) { defer wg.Done() // goroutine͕ྃͨ͠ΒىಈΛσΫϦϝϯτ for { ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ(தུ) } fh.Close() ! }(self, file) // ΫϩʔδϟΛgoroutineʹ͢Δͱ͖มͷڞ༗ʹҙ ! } wg.Wait() // ෆಛఆͷgoroutine͕શͯऴྃ͢ΔͷΛͭ close(self.Matches)
> the_simple_searcher go $GOROOT > /dev/null
panic ! too many open files
var wg sync.WaitGroup sem := make(chan bool, self.Option.Cap) // ىಈ͢ΔgoroutineͷΛ੍ޚ͢Δchannel
for file := range self.Files { sem <- true // goroutineͷىಈ(channelͷbuffer)͕͍ͬͺ͍ͳΒͭ wg.Add(1) (தུ) go func(self *Grep, file *string) { defer wg.Done() for { ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ(தུ) } fh.Close() <-sem // ಉ࣌ىಈchannelͷbufferʹۭ͖Λͭ͘Δ ! }(self, file) ! } wg.Wait() close(self.Matches)
> the_simple_searcher go $GOROOT > /dev/null
0.79 -> 0.8 seconds
Approach-3 ! ฒྻʹ
GOMAXPROCS
• Goroutineͷฒྻ • σϑΥϧτ1 • runtime.NumCPU()ͰίΞΛऔಘ • runtime.GOMAXPROCS()ͰฒྻΛઃఆ
> the_simple_searcher go $GOROOT > /dev/null
0.79 -> 0.55 ! seconds
benchmark ! • Mac OSX(10.9.3) • CPU: 2.5GHz Core i5(2Core)
• Memory: 8GB • Go: 1.2.2
#V⒎FS (0."9130$4 "QQSPBDI
ฒߦԽͯ͠ͳ͍ͷͰ ฒྻԽͯ͠มΘΒͣ ίΞҎ্ͷࢦఆ ޮՌͳ͠ ଌఆͯ͠ௐ͠ͳ͍ͱ ߹ʹΑͬͯ͘ͳΔ #V⒎FS (0."9130$4 "QQSPBDI
–Rob Pike • Concurrency is powerful. • Concurrency is not
parallelism. • Concurrency enables parallelism. • Concurrency makes parallelism (and scaling and everything else) easy.
એ ϖύϘͰΤϯδχΞΛืू͍ͯ͠·͢ɻ ڞʹαʔϏεΛੜΈग़͠ҭͯͯ͘ΕΔ৽͍ؒ͠ Λ͍ͬͯ·͢ɻ ! http://pepabo.com/recruit/career/engineer/
͓ΘΓ