Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
Search
FUJIWARA Shunichiro
June 22, 2019
Technology
12
7k
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
kamakura.go #5
FUJIWARA Shunichiro
June 22, 2019
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
2
260
監視のこれまでとこれから/sakura monitoring seminar 2025
fujiwara3
11
4.2k
k6による負荷試験 入門から日常的な実践まで/Re:TechTalk #01
fujiwara3
2
82
困難を「一般解」で解く
fujiwara3
10
3.7k
「隙間家具OSS」に至る道/Fujiwara Tech Conference 2025
fujiwara3
7
12k
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
1k
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
11
1.4k
「最高のチューニング」をしないために / hack@delta 24.10
fujiwara3
21
4.4k
AWS Lambdaで実現するスケーラブルで低コストなWebサービス構築/YAPC::Hakodate2024
fujiwara3
10
6.4k
Other Decks in Technology
See All in Technology
Tech-Verse 2025 Keynote
lycorptech_jp
PRO
0
1.3k
本が全く読めなかった過去の自分へ
genshun9
0
710
生成AI時代の開発組織・技術・プロセス 〜 ログラスの挑戦と考察 〜
itohiro73
1
380
Amazon Bedrockで実現する 新たな学習体験
kzkmaeda
2
680
Tech-Verse 2025 Global CTO Session
lycorptech_jp
PRO
0
1.2k
OpenHands🤲にContributeしてみた
kotauchisunsun
1
500
自律的なスケーリング手法FASTにおけるVPoEとしてのアカウンタビリティ / dev-productivity-con-2025
yoshikiiida
0
710
Github Copilot エージェントモードで試してみた
ochtum
0
140
250627 関西Ruby会議08 前夜祭 RejectKaigi「DJ on Ruby Ver.0.1」
msykd
PRO
2
370
Yamla: Rustでつくるリアルタイム性を追求した機械学習基盤 / Yamla: A Rust-Based Machine Learning Platform Pursuing Real-Time Capabilities
lycorptech_jp
PRO
4
180
Should Our Project Join the CNCF? (Japanese Recap)
whywaita
PRO
0
300
KubeCon + CloudNativeCon Japan 2025 に行ってきた! & containerd の新機能紹介
honahuku
0
120
Featured
See All Featured
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.8k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.5k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
How to train your dragon (web standard)
notwaldorf
94
6.1k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Being A Developer After 40
akosma
90
590k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
680
Balancing Empowerment & Direction
lara
1
390
Music & Morning Musume
bryan
46
6.6k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Transcript
ݎ࿚ͳTCPαʔόΛ࡞ΔͨΊʹ ʙkatsubushiͷݟ͔Βʙ 2019-06-22 Kamakura.go #5 @fujiwara
@fujiwara SRE(૯෦) github.com/fujiwara sfujiwara.hatenablog.com WEB+DB Press vol.111 SREνʔϜ࿈ࡌ Perl Hackers
HubͰAWS X-RayͷΛॻ ͖·ͨ͠
katsubushi - ࢄϢχʔΫID࠾൪ػ • ϢχʔΫͳ࣌ܥྻॱͷInt64 IDΛൃߦ͢Δ • ࢄڥ(ෳ)ͰඃΒͣಈ͘ • Memcached
ProtocolΛ࣮͍ͯ͠Δ • Goͷಠ࣮ࣗϛυϧΣΞ github.com/kayac/go-katsubushi
ݎ࿚ͳαʔόʁ ݎ࿚ is ... • ϦιʔεϦʔΫ͠ͳ͍ • མͪͳ͍ • (ϓϩμΫτͷੑ্࣭)
ID͕ܾͯ͠ඃΒͳ͍ • →γϟʔσΟϯά͞ΕͨผDBʹಉҰID͕ൃߦ͞ΕͨΒࢮ
memcached protocol ͷ࠾༻ཧ༝ • ܰྔ • HTTPͷΑ͏ʹ༨ܭͳϔομ͕ͳ͍ • Int64 Λ1ݸฦ͢ͷʹϔομ͕Կඦbyteͱ͔ͪΐͬͱ…
• ΫϥΠΞϯτ͕ߴ • ࣌ओʹ༻͍ͯͨ͠PerlʹCache::Memcached::Fastͱ ͍͏ߴͳ࣮ IDେྔʹൃߦ͢Δ͜ͱ͕͋ΔͷͰ͍ܰͷ͕Α͍
HTTPͳΒͱ͔͘memcached protocolΛॲཧ͢Δαʔό طଘͷϑϨʔϜϫʔΫ͕ͳ͍ ॻ͚͍͍ΑͶɺ؆୯ͩ͠
Go Ͱ TCP αʔό net package Ͱ؆୯ʹͰ͖Δ Listen, Acceptͯ͠ConnΛgoroutineͰॲཧ͢Δ͚ͩ ln,
err := net.Listen("tcp", ":8080") if err != nil { // handle error } for { conn, err := ln.Accept() if err != nil { // handle error } go handleConnection(conn) }
Go Ͱ TCP Echo αʔόɺ؆୯ func handleConnection(conn net.Conn) { defer
conn.Close() // ൈ͚Δͱ͖ʹஅ b := bufio.NewReader(conn) for { line, err := b.ReadString('\n') // 1ߦಡΉ if err != nil { return } _, err = io.WriteString(conn, line) // ΦϜฦ͠ if err != nil { return } } }
͔͠͠ݱ࣮໘
ϦιʔεϦʔΫ͠ͳ͍
ແ௨৴λΠϜΞτ • ܨ͗ͬͺͳ͠Ͱ͍ͳ͘ͳΔΫϥΠΞϯτରࡦ • ΫϥΠΞϯτ͕TCPΛ͖Ε͍ʹஅ͢ΔͱݶΒͳ͍ • αʔόଆͰΒͳ͍ͱίωΫγϣϯ(goroutine)͕ϦʔΫ͢Δ
net/Conn.SetDeadline ίωΫγϣϯʹࢦఆͨ࣌͠ࠁ·Ͱ௨৴͕ͳ͔ͬͨΒ ΤϥʔʹͳΔΑ͏ઃఆ͢Δ ௨৴ཱ֬ޙɺಡΈॻ͖Ͱ͖ͨλΠϛϯάͰ࠶ઃఆ → ແ௨৴λΠϜΞτ͕࣮Ͱ͖Δ type Conn interface {
SetDeadline(t time.Time) error }
for { conn, err := ln.Accept() if err != nil
{ // handle error } conn.SetDeadline(time.Now().Add(idleTimeout)) go handleConnection(conn) } func handleConnection(conn net.Conn) { defer conn.Close() buf := bufio.NewReader(conn) for { line, _ := buf.ReadString('\n') // 1ߦಡΉ conn.SetDeadline(time.Now().Add(idleTimeout)) // ^λΠϜΞτԆ
ϓϩηεऴྃ࣌ʹஅ αʔόͪΌΜͱஅ͠ͳ͍ͱߦّ͕ѱ͍ΑͶ? func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
͜͜ͰͰ͢ ͜ͷίʔυͰͳʹ͔͕ϦʔΫ͠·͢ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
goroutineϦʔΫ handleConn͕returnͯ͠goroutine͕ࢮͳͳ͍ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ͜Ε͕ݺΕΔ(=ϓϩηεऴྃ)·Ͱ conn.Close() // goroutine ͕ੜ͖Δ }() // ...
मਖ਼ํ๏1 ࢠ context Λ࡞ͬͯ defer cancel() ࣗൃతऴྃ࣌ɺऴྃ࣌ͲͪΒͰίωΫγϣϯஅͱ goroutineऴ͕ྃͰ͖ΔͷͰϦʔΫ͠ͳ͍ func (app
*App) handleConn(ctx context.Context, conn net.Conn) { ctx2, cancel := context.WithCancel(ctx) defer cancel() go func() { <-ctx2.Done() conn.Close() }() // ... 1 Fix goroutine leak. #28 https://github.com/kayac/go-katsubushi/pull/28/files
མͪͳ͍
ʮ͜ͷ katsubushi ಈ͍ͯΔ͔ͳ?ʯ $ curl localhost:11212 VALUE / 0 18
591644124189298688 VALUE HTTP/1.1 0 18 591644124189298689 END ERROR ERROR ERROR ʮIDฦ͖͚ͬͯͨͲͳʹ͜Ε??ʯ ʮ͋ɺcurlͰୟ͍ͯͨɻɻɻʯ
Կ͕ى͖͍͔ͯͨ curl katsubushi ʹ HTTP ͰϦΫΤετͨ͠(વ) $ curl -v
127.0.0.1:11212 > GET / HTTP/1.1 > Host: 127.0.0.1:11212 > User-Agent: curl/7.65.1 > Accept: */* > (ۭߦ) HTTPͷϦΫΤετ memcached protocol ͰղऍͰ͖Δʂ memcached protocol Ͱͷऔಘ: GET key1 key2 HTTP Ͱ GET ϦΫΤετͷ1ߦ: GET / HTTP/1.1
ͭ·Γ͜͏͍͏Ϩεϙϯε͕ฦΔ VALUE / 0 18 # / ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298688
VALUE HTTP/1.1 0 18 # HTTP/1.1 ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298689 END # HTTPͷ1ߦʹϨεϙϯε͕ฦͤͨΑʂ ERROR # Host: ͱ͔Βͳ͍! ERROR # User-Agnet: ͱ͔Βͳ͍!! ERROR # Accept: ͱ͔Βͳ͍!!!
Ұํͦͷ͜Ζ HTTP Λ৯ͨ katsubushi panic: runtime error: invalid memory
address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x134807f] goroutine 20 [running]: github.com/kayac/go-katsubushi.(*App).handleConn(0xc00012a000, 0x14a3de0, 0xc0000ac038) /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:160 +0x2cf created by github.com/kayac/go-katsubushi.(*App).Listen /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:131 +0x2ed ʮ͠ɺࢮΜͰΔ…ʯ
Կ͕ى͖͍͔ͯͨ ಡΈऔͬͨϦΫΤετߦ([]byte)ΛίϚϯυʹ࣮ͯ͠ߦ HTTPϔομ͓ΘΓͷۭߦΛಡΜͩͱ͜ΖͰ… cmd, err := app.BytesToCmd(scanner.Bytes()) if err !=
nil { return } err = cmd.Execute(app, conn) // ͜͜Ͱࢮ
൜ਓ func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { // ۭߦͳͷͰ 0 byte return nil, nil // ← ͍ͭ͜ } foo, err := Foo() ͱ͍͏ΠϯλʔϑΣʔεΛΈͨ ී௨ͷGopher͕ߟ͑Δ͜ͱ ʮerr != nil ͳΒ foo == nil ͔͔ͩΒؾΛ͚ͭΑ͏ʯ ʮerr == nil ͳΒ foo != nil ͩΑͶʂʯ
nil, nil Λฦ͢ͷ͕ѱ͍ ॻ͍ͨͷͲ͚ͬͪࣗͩͲ݀ʹམͪͨ ௨ৗmemcached protocolͰϦΫΤετʹۭߦདྷͳ͍ ຊ൪ӡ༻தͷϓϩηε͕ଈࢮͯ͠ͼͬ͘Γͨ͠ katsubushiΫϥΠΞϯτϥΠϒϥϦfailoverΛ࣮͍ͯͨ͠ daemontools Ͱ͙͢ʹ࠶ىಈ͞Εͨ
ͷͰக໋ইʹͳΒͳ͔ͬͨ γεςϜશମͰݎ࿚ʹ͍͖ͯ͠·͠ΐ͏…
ରԠ2 func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { - return nil, nil + return nil, errors.New("No command") Βͳ͍ίϚϯυ͕དྷͨΒଈ࠲ʹଓΛͬͨ΄͏͕͍͍͔ 2 Never die #8 https://github.com/kayac/go-katsubushi/pull/8
༨ஊ1 ࣮ curl telnet Ͱ͖·͢ $ curl telnet://127.0.0.1:11212 GET
id VALUE id 0 18 591655476635111424 END quit
༨ஊ2 ʮ1ߦΛղऍͯ͠key͕ / HTTP/1.[01] ͳΒHTTPɻɻɻʯ ʮͭ·Γಉ͡ϙʔτͰ memcached ͱ HTTP ྆ରԠ
αʔό͕ॻ͚ΔͷͰ!??ʯ ్த·Ͱॻ͍ͯਖ਼ؾʹͬͯʹ
(ϓϩμΫτͷੑ্࣭) ID͕ܾͯ͠ඃΒͳ͍
katsubushi ID ࣌ࠁϕʔε +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID |
Sequence | +-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z ࣌ࠁ͕Δͱಉ͡ID͕ൃߦ͞ΕΔՄೳੑ͕͋Δ
monotonic time https://golang.org/doc/go1.9#monotonic-time Go 1.9ͰOS࣌ࠁ͕ͬͯ time.Now() Βͳ͍ monotonic time͕࣮͞Εͨ 2017−01−01ͷӞඵൃੜ࣌
Cloudflare͕࣌ࠁͷר͖ΓͰো3 ࣌ࠁ͕ͬͨ݁Ռ math/rand.Int63n ʹෛͷ͕Γpanic 3 How and why the leap second affected Cloudflare DNS https://blog.cloudflare.com/how-and-why-the-leap-second-affected- cloudflare-dns/
monotonic time Ҏલͷ katsubushi ts := g.timestamp() // for rewind
of server clock if ts < g.lastTimestamp { return 0, errors.New("system clock was rollbacked") } // ུ g.lastTimestamp = ts ࣌ࠁ͕ר͖ͬͨΒΤϥʔʹͯ͠ޚ ID͕ඃΔ͙Β͍ͳΒൃߦͰ͖ͳ͍ํ͕Ϛγ
monotonic time ରԠ4 katsubushi ID ͷ࣌ࠁ෦ 2015-01-01T00:00:00 UTC Λىݯͱͨ͠ܦաඵ var
Epoch = time.Date(2015, 1, 1, 0, 0, 0, 0, time.UTC) d := time.Now().Sub(Epoch) ͜ͷΑ͏ʹɺ࣌ࢦఆͰ࡞͔ͬͨΒܭࢉ͢Δͱmonotonicʹ ͳΒͳ͍ 4 use Monotonic time #21 https://github.com/kayac/go-katsubushi/pull/21
monotonic time ରԠ now := time.Now() startedAt := now offset
:= now.Sub(Epoch) // ىಈ࣌ͷ࣌ࠁͱepochͷࠩΛܭࢉ͓ͯ͘͠ // ... d := time.Now().Sub(startedAt) + offset time.Now() Ͱ࡞ͬͨಉ࢜Λൺֱͯ͠offsetΛௐ͢Ε ר͖Βͳ͍
࣌ࠁͷଞʹඃΔՄೳੑ͕͋Δ෦͕… +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID | Sequence |
+-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z WorkerID = ಉ࣌ʹಈ࡞͍ͯ͠ΔϓϩηεؒͰҰҙͷID
RedisΛͬͯҰҙੑΛ୲อ͢Δ Ҏલͷൃදʹ͋ΔͷͰͲ͏ͧ https://speakerdeck.com/fujiwara3/katsubushi?slide=79
ࢹେࣄ • github.com/fukata/golang-stats-api-handler • GoͷϝτϦΫεΛऔΔ • mackerel-plugin-gostats ͋ΔΑ • net/http/pprof
ͰϓϩϑΝΠϧऔಘͷޱΛ։͚Δ
·ͱΊ • GoͰTCPαʔόΛ࡞Δͷ؆୯ • Ͱ࣮ӡ༻Ͱݎ࿚ͳͷΛ࡞Δʹ͍Ζ͍Ζ͋Δ • Ұݟ؆୯ʹݟ͑ΔͷͰɺࣗલ࣮পͷୈҰา