Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
Search
FUJIWARA Shunichiro
June 22, 2019
Technology
12
6.9k
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
kamakura.go #5
FUJIWARA Shunichiro
June 22, 2019
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
300
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
9
1.1k
「最高のチューニング」をしないために / hack@delta 24.10
fujiwara3
21
4k
AWS Lambdaで実現するスケーラブルで低コストなWebサービス構築/YAPC::Hakodate2024
fujiwara3
10
4.5k
CEL(Common Expression Language)で書いた条件にマッチしたIAM Policyを見つける / iam-policy-finder
fujiwara3
2
1.5k
awslim - Goで実装された高速なAWS CLIの代替品を作った/layerx.go#1
fujiwara3
6
760
AWS CLIの起動が重くてつらいので aws-sdk-client-go を書いた / kamakura.go#6
fujiwara3
7
10k
コードを書く隙間を見つけて生きていく技術/Findy 思考の現在地
fujiwara3
31
7.1k
fujiwara-ware OSSをひたすら紹介する/ya8-2024
fujiwara3
8
810
Other Decks in Technology
See All in Technology
Snowflake女子会#3 Snowpipeの良さを5分で語るよ
lana2548
0
230
権威ドキュメントで振り返る2024 #年忘れセキュリティ2024
hirotomotaguchi
2
740
AI時代のデータセンターネットワーク
lycorptech_jp
PRO
1
280
TSKaigi 2024 の登壇から広がったコミュニティ活動について
tsukuha
0
160
Qiita埋め込み用スライド
naoki_0531
0
4.8k
コンテナセキュリティのためのLandlock入門
nullpo_head
2
320
あの日俺達が夢見たサーバレスアーキテクチャ/the-serverless-architecture-we-dreamed-of
tomoki10
0
450
LINEスキマニにおけるフロントエンド開発
lycorptech_jp
PRO
0
330
プロダクト開発を加速させるためのQA文化の築き方 / How to build QA culture to accelerate product development
mii3king
1
260
Postman と API セキュリティ / Postman and API Security
yokawasa
0
200
Amazon SageMaker Unified Studio(Preview)、Lakehouse と Amazon S3 Tables
ishikawa_satoru
0
150
【re:Invent 2024 アプデ】 Prompt Routing の紹介
champ
0
140
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.6k
Rails Girls Zürich Keynote
gr2m
94
13k
Designing for Performance
lara
604
68k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
95
17k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
26
1.5k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
StorybookのUI Testing Handbookを読んだ
zakiyama
27
5.3k
BBQ
matthewcrist
85
9.4k
Mobile First: as difficult as doing things right
swwweet
222
9k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
The Pragmatic Product Professional
lauravandoore
32
6.3k
Transcript
ݎ࿚ͳTCPαʔόΛ࡞ΔͨΊʹ ʙkatsubushiͷݟ͔Βʙ 2019-06-22 Kamakura.go #5 @fujiwara
@fujiwara SRE(૯෦) github.com/fujiwara sfujiwara.hatenablog.com WEB+DB Press vol.111 SREνʔϜ࿈ࡌ Perl Hackers
HubͰAWS X-RayͷΛॻ ͖·ͨ͠
katsubushi - ࢄϢχʔΫID࠾൪ػ • ϢχʔΫͳ࣌ܥྻॱͷInt64 IDΛൃߦ͢Δ • ࢄڥ(ෳ)ͰඃΒͣಈ͘ • Memcached
ProtocolΛ࣮͍ͯ͠Δ • Goͷಠ࣮ࣗϛυϧΣΞ github.com/kayac/go-katsubushi
ݎ࿚ͳαʔόʁ ݎ࿚ is ... • ϦιʔεϦʔΫ͠ͳ͍ • མͪͳ͍ • (ϓϩμΫτͷੑ্࣭)
ID͕ܾͯ͠ඃΒͳ͍ • →γϟʔσΟϯά͞ΕͨผDBʹಉҰID͕ൃߦ͞ΕͨΒࢮ
memcached protocol ͷ࠾༻ཧ༝ • ܰྔ • HTTPͷΑ͏ʹ༨ܭͳϔομ͕ͳ͍ • Int64 Λ1ݸฦ͢ͷʹϔομ͕Կඦbyteͱ͔ͪΐͬͱ…
• ΫϥΠΞϯτ͕ߴ • ࣌ओʹ༻͍ͯͨ͠PerlʹCache::Memcached::Fastͱ ͍͏ߴͳ࣮ IDେྔʹൃߦ͢Δ͜ͱ͕͋ΔͷͰ͍ܰͷ͕Α͍
HTTPͳΒͱ͔͘memcached protocolΛॲཧ͢Δαʔό طଘͷϑϨʔϜϫʔΫ͕ͳ͍ ॻ͚͍͍ΑͶɺ؆୯ͩ͠
Go Ͱ TCP αʔό net package Ͱ؆୯ʹͰ͖Δ Listen, Acceptͯ͠ConnΛgoroutineͰॲཧ͢Δ͚ͩ ln,
err := net.Listen("tcp", ":8080") if err != nil { // handle error } for { conn, err := ln.Accept() if err != nil { // handle error } go handleConnection(conn) }
Go Ͱ TCP Echo αʔόɺ؆୯ func handleConnection(conn net.Conn) { defer
conn.Close() // ൈ͚Δͱ͖ʹஅ b := bufio.NewReader(conn) for { line, err := b.ReadString('\n') // 1ߦಡΉ if err != nil { return } _, err = io.WriteString(conn, line) // ΦϜฦ͠ if err != nil { return } } }
͔͠͠ݱ࣮໘
ϦιʔεϦʔΫ͠ͳ͍
ແ௨৴λΠϜΞτ • ܨ͗ͬͺͳ͠Ͱ͍ͳ͘ͳΔΫϥΠΞϯτରࡦ • ΫϥΠΞϯτ͕TCPΛ͖Ε͍ʹஅ͢ΔͱݶΒͳ͍ • αʔόଆͰΒͳ͍ͱίωΫγϣϯ(goroutine)͕ϦʔΫ͢Δ
net/Conn.SetDeadline ίωΫγϣϯʹࢦఆͨ࣌͠ࠁ·Ͱ௨৴͕ͳ͔ͬͨΒ ΤϥʔʹͳΔΑ͏ઃఆ͢Δ ௨৴ཱ֬ޙɺಡΈॻ͖Ͱ͖ͨλΠϛϯάͰ࠶ઃఆ → ແ௨৴λΠϜΞτ͕࣮Ͱ͖Δ type Conn interface {
SetDeadline(t time.Time) error }
for { conn, err := ln.Accept() if err != nil
{ // handle error } conn.SetDeadline(time.Now().Add(idleTimeout)) go handleConnection(conn) } func handleConnection(conn net.Conn) { defer conn.Close() buf := bufio.NewReader(conn) for { line, _ := buf.ReadString('\n') // 1ߦಡΉ conn.SetDeadline(time.Now().Add(idleTimeout)) // ^λΠϜΞτԆ
ϓϩηεऴྃ࣌ʹஅ αʔόͪΌΜͱஅ͠ͳ͍ͱߦّ͕ѱ͍ΑͶ? func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
͜͜ͰͰ͢ ͜ͷίʔυͰͳʹ͔͕ϦʔΫ͠·͢ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
goroutineϦʔΫ handleConn͕returnͯ͠goroutine͕ࢮͳͳ͍ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ͜Ε͕ݺΕΔ(=ϓϩηεऴྃ)·Ͱ conn.Close() // goroutine ͕ੜ͖Δ }() // ...
मਖ਼ํ๏1 ࢠ context Λ࡞ͬͯ defer cancel() ࣗൃతऴྃ࣌ɺऴྃ࣌ͲͪΒͰίωΫγϣϯஅͱ goroutineऴ͕ྃͰ͖ΔͷͰϦʔΫ͠ͳ͍ func (app
*App) handleConn(ctx context.Context, conn net.Conn) { ctx2, cancel := context.WithCancel(ctx) defer cancel() go func() { <-ctx2.Done() conn.Close() }() // ... 1 Fix goroutine leak. #28 https://github.com/kayac/go-katsubushi/pull/28/files
མͪͳ͍
ʮ͜ͷ katsubushi ಈ͍ͯΔ͔ͳ?ʯ $ curl localhost:11212 VALUE / 0 18
591644124189298688 VALUE HTTP/1.1 0 18 591644124189298689 END ERROR ERROR ERROR ʮIDฦ͖͚ͬͯͨͲͳʹ͜Ε??ʯ ʮ͋ɺcurlͰୟ͍ͯͨɻɻɻʯ
Կ͕ى͖͍͔ͯͨ curl katsubushi ʹ HTTP ͰϦΫΤετͨ͠(વ) $ curl -v
127.0.0.1:11212 > GET / HTTP/1.1 > Host: 127.0.0.1:11212 > User-Agent: curl/7.65.1 > Accept: */* > (ۭߦ) HTTPͷϦΫΤετ memcached protocol ͰղऍͰ͖Δʂ memcached protocol Ͱͷऔಘ: GET key1 key2 HTTP Ͱ GET ϦΫΤετͷ1ߦ: GET / HTTP/1.1
ͭ·Γ͜͏͍͏Ϩεϙϯε͕ฦΔ VALUE / 0 18 # / ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298688
VALUE HTTP/1.1 0 18 # HTTP/1.1 ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298689 END # HTTPͷ1ߦʹϨεϙϯε͕ฦͤͨΑʂ ERROR # Host: ͱ͔Βͳ͍! ERROR # User-Agnet: ͱ͔Βͳ͍!! ERROR # Accept: ͱ͔Βͳ͍!!!
Ұํͦͷ͜Ζ HTTP Λ৯ͨ katsubushi panic: runtime error: invalid memory
address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x134807f] goroutine 20 [running]: github.com/kayac/go-katsubushi.(*App).handleConn(0xc00012a000, 0x14a3de0, 0xc0000ac038) /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:160 +0x2cf created by github.com/kayac/go-katsubushi.(*App).Listen /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:131 +0x2ed ʮ͠ɺࢮΜͰΔ…ʯ
Կ͕ى͖͍͔ͯͨ ಡΈऔͬͨϦΫΤετߦ([]byte)ΛίϚϯυʹ࣮ͯ͠ߦ HTTPϔομ͓ΘΓͷۭߦΛಡΜͩͱ͜ΖͰ… cmd, err := app.BytesToCmd(scanner.Bytes()) if err !=
nil { return } err = cmd.Execute(app, conn) // ͜͜Ͱࢮ
൜ਓ func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { // ۭߦͳͷͰ 0 byte return nil, nil // ← ͍ͭ͜ } foo, err := Foo() ͱ͍͏ΠϯλʔϑΣʔεΛΈͨ ී௨ͷGopher͕ߟ͑Δ͜ͱ ʮerr != nil ͳΒ foo == nil ͔͔ͩΒؾΛ͚ͭΑ͏ʯ ʮerr == nil ͳΒ foo != nil ͩΑͶʂʯ
nil, nil Λฦ͢ͷ͕ѱ͍ ॻ͍ͨͷͲ͚ͬͪࣗͩͲ݀ʹམͪͨ ௨ৗmemcached protocolͰϦΫΤετʹۭߦདྷͳ͍ ຊ൪ӡ༻தͷϓϩηε͕ଈࢮͯ͠ͼͬ͘Γͨ͠ katsubushiΫϥΠΞϯτϥΠϒϥϦfailoverΛ࣮͍ͯͨ͠ daemontools Ͱ͙͢ʹ࠶ىಈ͞Εͨ
ͷͰக໋ইʹͳΒͳ͔ͬͨ γεςϜશମͰݎ࿚ʹ͍͖ͯ͠·͠ΐ͏…
ରԠ2 func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { - return nil, nil + return nil, errors.New("No command") Βͳ͍ίϚϯυ͕དྷͨΒଈ࠲ʹଓΛͬͨ΄͏͕͍͍͔ 2 Never die #8 https://github.com/kayac/go-katsubushi/pull/8
༨ஊ1 ࣮ curl telnet Ͱ͖·͢ $ curl telnet://127.0.0.1:11212 GET
id VALUE id 0 18 591655476635111424 END quit
༨ஊ2 ʮ1ߦΛղऍͯ͠key͕ / HTTP/1.[01] ͳΒHTTPɻɻɻʯ ʮͭ·Γಉ͡ϙʔτͰ memcached ͱ HTTP ྆ରԠ
αʔό͕ॻ͚ΔͷͰ!??ʯ ్த·Ͱॻ͍ͯਖ਼ؾʹͬͯʹ
(ϓϩμΫτͷੑ্࣭) ID͕ܾͯ͠ඃΒͳ͍
katsubushi ID ࣌ࠁϕʔε +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID |
Sequence | +-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z ࣌ࠁ͕Δͱಉ͡ID͕ൃߦ͞ΕΔՄೳੑ͕͋Δ
monotonic time https://golang.org/doc/go1.9#monotonic-time Go 1.9ͰOS࣌ࠁ͕ͬͯ time.Now() Βͳ͍ monotonic time͕࣮͞Εͨ 2017−01−01ͷӞඵൃੜ࣌
Cloudflare͕࣌ࠁͷר͖ΓͰো3 ࣌ࠁ͕ͬͨ݁Ռ math/rand.Int63n ʹෛͷ͕Γpanic 3 How and why the leap second affected Cloudflare DNS https://blog.cloudflare.com/how-and-why-the-leap-second-affected- cloudflare-dns/
monotonic time Ҏલͷ katsubushi ts := g.timestamp() // for rewind
of server clock if ts < g.lastTimestamp { return 0, errors.New("system clock was rollbacked") } // ུ g.lastTimestamp = ts ࣌ࠁ͕ר͖ͬͨΒΤϥʔʹͯ͠ޚ ID͕ඃΔ͙Β͍ͳΒൃߦͰ͖ͳ͍ํ͕Ϛγ
monotonic time ରԠ4 katsubushi ID ͷ࣌ࠁ෦ 2015-01-01T00:00:00 UTC Λىݯͱͨ͠ܦաඵ var
Epoch = time.Date(2015, 1, 1, 0, 0, 0, 0, time.UTC) d := time.Now().Sub(Epoch) ͜ͷΑ͏ʹɺ࣌ࢦఆͰ࡞͔ͬͨΒܭࢉ͢Δͱmonotonicʹ ͳΒͳ͍ 4 use Monotonic time #21 https://github.com/kayac/go-katsubushi/pull/21
monotonic time ରԠ now := time.Now() startedAt := now offset
:= now.Sub(Epoch) // ىಈ࣌ͷ࣌ࠁͱepochͷࠩΛܭࢉ͓ͯ͘͠ // ... d := time.Now().Sub(startedAt) + offset time.Now() Ͱ࡞ͬͨಉ࢜Λൺֱͯ͠offsetΛௐ͢Ε ר͖Βͳ͍
࣌ࠁͷଞʹඃΔՄೳੑ͕͋Δ෦͕… +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID | Sequence |
+-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z WorkerID = ಉ࣌ʹಈ࡞͍ͯ͠ΔϓϩηεؒͰҰҙͷID
RedisΛͬͯҰҙੑΛ୲อ͢Δ Ҏલͷൃදʹ͋ΔͷͰͲ͏ͧ https://speakerdeck.com/fujiwara3/katsubushi?slide=79
ࢹେࣄ • github.com/fukata/golang-stats-api-handler • GoͷϝτϦΫεΛऔΔ • mackerel-plugin-gostats ͋ΔΑ • net/http/pprof
ͰϓϩϑΝΠϧऔಘͷޱΛ։͚Δ
·ͱΊ • GoͰTCPαʔόΛ࡞Δͷ؆୯ • Ͱ࣮ӡ༻Ͱݎ࿚ͳͷΛ࡞Δʹ͍Ζ͍Ζ͋Δ • Ұݟ؆୯ʹݟ͑ΔͷͰɺࣗલ࣮পͷୈҰา