Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Go in production - case of Gannoy -
monochromegane
October 17, 2017
Technology
1
270
Go in production - case of Gannoy -
fukuokago#9
https://connpass.com/event/68358/
monochromegane
October 17, 2017
Tweet
Share
More Decks by monochromegane
See All by monochromegane
迅速な学習機構を用いて逐次適応性を損なうことなく非線形性を扱う文脈付き多腕バンディット手法/extreme_neural_linear_bandits
monochromegane
0
220
再帰化への認知的転回/the-turn-to-recursive-system
monochromegane
0
310
仮想的な探索を用いて文脈や時間の経過による番狂わせにも迅速に追従する多腕バンディット手法/wi2_lkf_bandits
monochromegane
0
320
Synapse: 文脈と時間経過に応じて推薦手法の選択を最適化するメタ推薦システム/smash21-synapse
monochromegane
0
230
なめらかなシステムと運用維持の未来/dicomo2021-coherently-fittable-system
monochromegane
1
9.3k
go:embedでExplainable Binaryを作る/fukoukago17_go_code_embedding
monochromegane
2
180
非定常な多腕バンディット問題において効率的に変化を察知する方式の検討/wsa8_predictive_exploratory_model
monochromegane
0
2.2k
変化検出と要約データ構造を用いた利用者の嗜好の変化に迅速に追従する多腕バンディット手法/iots2020-adaptive-linear-mab
monochromegane
0
620
嗜好伝達コミュニケーションの効率化を目指した伝達方式の検討/wsa7_local_preference
monochromegane
0
2.6k
Other Decks in Technology
See All in Technology
API連携に伴う規制と対応 / Regulations and responses to API linkage
moneyforward
0
110
20230121_データ分析系コミュニティ_サテライト企画
doradora09
0
510
NGINXENG JP#2 - 3-NGINX Plus・プロダクトのアップデート
hiropo20
0
180
日経電子版だけじゃない! 日経の新規Webメディアの開発 - NIKKEI Tech Talk #3
sztm
0
140
AWS re:Invent 2022で発表された新機能を試してみた ~Cloud OperationとSecurity~ / New Cloud Operation and Security Features Announced at AWS reInvent 2022
yuj1osm
1
130
2年で10→70人へ! スタートアップの 情報セキュリティ課題と施策
miekobayashi
1
200
【Oracle Cloud ウェビナー】事例から見る規模別クラウド・データベースの選び方 (Oracle Database) (2023年1月18日)
oracle4engineer
PRO
0
100
OCIコンテナサービス関連の技術詳細 /oke-ocir-details
oracle4engineer
PRO
0
750
OpenShift.Run2023_create-aro-with-terraform
ishiitaiki20fixer
1
160
Akiba-dot-SaaS-ExtraHop
sakaitakeshi
1
110
230125 古いタブレットの活用 かーでぃさん
comucal
PRO
0
4.9k
Pentesting Password Reset Functionality
anugrahsr
0
210
Featured
See All Featured
Building Adaptive Systems
keathley
27
1.3k
How New CSS Is Changing Everything About Graphic Design on the Web
jensimmons
214
12k
Scaling GitHub
holman
453
140k
WebSockets: Embracing the real-time Web
robhawkes
58
6k
From Idea to $5000 a Month in 5 Months
shpigford
374
44k
The Brand Is Dead. Long Live the Brand.
mthomps
48
2.9k
Producing Creativity
orderedlist
PRO
335
37k
The Web Native Designer (August 2011)
paulrobertlloyd
76
2.2k
4 Signs Your Business is Dying
shpigford
171
20k
Building Flexible Design Systems
yeseniaperezcruz
314
35k
Building Your Own Lightsaber
phodgson
96
4.9k
Reflections from 52 weeks, 52 projects
jeffersonlam
338
18k
Transcript
- case of Gannoy - ࡾ༔հ / Pepabo R&D Institute,
GMO Pepabo, Inc. 2017.10.17 Fukuoka.go#9 Go in production
ϓϦϯγύϧΤϯδχΞ ࡾ ༔հ / @monochromegane 2 http://blog.monochromegane.com Yusuke Miyake ϖύϘݚڀॴ
ݚڀһ
1. Goຊ൪ӡ༻ͷཁ݅ 2. Goຊ൪ӡ༻ͷ࣮ݱ 3. ·ͱΊ 3 ࣍
1. GannoyʹΈΔ Goຊ൪ӡ༻ͷཁ݅
Gannoy Approximate nearest neighbor search server and dynamic index written
in Golang. https://github.com/monochromegane/gannoy
Similar images search system using Gannoy 6 Features Similar items
Gannoy [2048]float64 query by http find similar features mapping similar features to items response Deep CNN index Features [2048]float64 Deep CNN register by http
• HTTP based Approximate nearest neighbor search server • Using
labstack/echo • Go 1.8+ 7 Gannoy
Goຊ൪ӡ༻ͷཁ݅
• Graceful restart • Daemonize • Monitoring • Log rotation
9 Goຊ൪ӡ༻ͷཁ݅ • Configuration • Build • Deploy
2. GannoyʹΈΔ Goຊ൪ӡ༻ͷ࣮ݱ
Graceful restart 11 go-server-starter + http.Server#Shutdown() import "github.com/lestrrat/go-server-starter/listener" e :=
echo.New() listeners, err := listener.ListenAll() if err != nil && err != listener.ErrNoListeningTarget { fmt.Fprintln(os.Stderr, err) os.Exit(1) } e.Listener = netutil.LimitListener(listeners[0], opts.MaxConnections) Listen from start_server socket.
Graceful restart 12 go-server-starter + http.Server#Shutdown() go func() { if
err := e.Start(""); err != nil { e.Logger.Info("Shutting down the server") } }() sigCh := make(chan os.Signal, 1) signal.Notify(sigCh, sig) <-sigCh ctx, cancel := context.WithTimeout(context.Background(), time.Duration(opts.ShutDownTimeout)*time.Second) defer cancel() if err := e.Shutdown(ctx); err != nil { e.Logger.Fatal(err) } Start server as goroutines. Wait SIGTERM. Graceful shutdown using http.Server.
Daemonize 13 Systemd [Unit] Description=Gannoy DB ConditionPathExists=/usr/bin/start_server ConditionPathExists=/usr/bin/gannoy-db After=network.target [Service]
User=gannoy Group=gannoy Type=simple PIDFile=/var/run/gannoy/server_starter.pid ExecStart=/usr/bin/start_server --port 8080 --pid-file /var/run/gannoy/server_starter.pid -- / usr/bin/gannoy-db -c /etc/gannoy/gannoy-db.toml ExecReload=/bin/kill -HUP $MAINPID ExecStop=/bin/kill -TERM $MAINPID [Install] WantedBy=multi-user.target gannoy-db.service
Monitoring 14 health-check endpoint + Consul e.GET("/health", func(c echo.Context) error
{ return c.NoContent(http.StatusOK) }) { "service": { (snip) "port": 80, "checks": [ { "script": "/usr/bin/check-http -u http://localhost/health", "interval": "30s" } ] } } health-check endpoint consul.d/gannoy.json
Log ratation 15 logrotate /var/log/gannoy/gannoy-db.log { daily rotate 7 missingok
notifempty compress postrotate kill -HUP `cat /var/run/gannoy/server_starter.pid` endscript } gannoy-db.logrotate reopen logfile.
Configuration 16 monochromegane/conflag // Parse option from args and configuration
file. conflag.LongHyphen = true conflag.BoolValue = false parser := flags.NewParser(&opts, flags.Default) _, err := parser.ParseArgs(os.Args[1:]) if err != nil { os.Exit(1) } if opts.Config != "" { if args, err := conflag.ArgsFrom(opts.Config); err == nil { if _, err := parser.ParseArgs(args); err != nil { os.Exit(1) } } } _, err = parser.ParseArgs(os.Args[1:]) if err != nil { os.Exit(1) } Read option from config. Overwrite option from args. data-dir = "/var/lib/gannoy" log-dir = "/var/log/gannoy" lock-dir = "/var/run/gannoy" server-starter = true auto-save = true gannoy-db —data-dir=“data”
Configuration 17 monochromegane/conflag http://blog.monochromegane.com/blog/2015/05/30/conflag/
Build 18 rpmbuild + docker-compose . !"" Dockerfile.rpmbuild !"" docker-compose.yml
!"" rpmbuild # !"" RPMS # # $"" x86_64 # # $"" gannoy-0.0.1-8.x86_64.rpm # !"" SOURCES # # !"" gannoy-db.logrotate # # !"" gannoy-db.service # # $"" gannoy-db.toml # $"" SPECS # $"" gannoy.spec $"" version.go version: '2' services: gannoy-rpmbuild: build: context: . dockerfile: Dockerfile.rpmbuild command: cp /root/rpmbuild/RPMS/x86_64/ gannoy-0.0.1-8.x86_64.rpm /tmp/rpmbuild/RPMS/ x86_64/. volumes: - .:/tmp:rw RUN rpmbuild -bb /root/rpmbuild/SPECS/gannoy.spec $ docker-compose build gannoy-rpmbuild $ docker-compose run gannoy-rpmbuild docker-compose.yml Dockerfile.rpmbuild
Deploy (and test) 19 Puppet + Serverspec class gannoy::install {
include ::go_start_server $rpm = 'gannoy-0.0.1-8.x86_64.rpm' file { "/usr/local/src/${rpm}": ensure => present, source => "puppet:///modules/gannoy/${rpm}", mode => '0755', owner => 'root', notify => Package['gannoy'], } -> package { 'gannoy': ensure => '0.0.1-8', source => "/usr/local/src/${rpm}", provider => rpm, } (snip) } (snip) describe service('gannoy-db') do it { should be_enabled } end describe package('gannoy') do it { should be_installed.with_version('0.0.1-8') } end (snip)
·ͱΊ
• σʔϞϯԽ͕ඞཁͳGoΞϓϦͷຊ൪ӡ༻όΠφϦҎ֎ʹඞཁͳ ϑΝΠϧ͕͋ΔͷͰύοέʔδϯάͨ͠΄͏͕औΓճָ͕͠ • ΦϓγϣϯͱઃఆϑΝΠϧͷڞ௨Խʹconflag͕ศར • ϕετϓϥΫςΟεࡧதɻσΟεΧογϣϯ͠·͠ΐ͏ɻ 21 ·ͱΊ
ݚڀһɺੵۃతʹืूதʂ http://rand.pepabo.com/