Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
Search
FUJIWARA Shunichiro
June 22, 2019
Technology
12
6.6k
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
kamakura.go #5
FUJIWARA Shunichiro
June 22, 2019
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
AWS CLIの起動が重くてつらいので aws-sdk-client-go を書いた / kamakura.go#6
fujiwara3
7
8.9k
コードを書く隙間を見つけて生きていく技術/Findy 思考の現在地
fujiwara3
28
6.6k
fujiwara-ware OSSをひたすら紹介する/ya8-2024
fujiwara3
7
560
Amazon ECSで好きなだけ検証環境を起動できるOSSの設計・実装・運用 / YAPC::Hiroshima 2024
fujiwara3
22
7.5k
リアル事例から読み解くWebパフォーマンスチューニングの勘所/Offers web performance tuning
fujiwara3
4
1.6k
隙間家具OSS開発で『自分の庭』をつくる / kayac-andpad-event
fujiwara3
0
760
ISUCON作問入門/ ISUCON Summer Fes 2023
fujiwara3
2
1.6k
隙間家具職人が考えること/ecspresso meetup
fujiwara3
5
4.6k
MackerelとGrafana OnCallを連携してみた
fujiwara3
0
1.7k
Other Decks in Technology
See All in Technology
エムスリーMR君ファミリー開発チーム紹介資料 / Introduction of M3 MRkun Family Dev Team
m3_engineering
0
110
OpenTelemetry Meetup 2024-06 - ABEMA と分散トレーシングのあゆみ
tetsuya28
0
250
OSS活動のススメ / oss-activities
gotok365
4
480
サイボウズのOSPO
sat
PRO
3
170
私たちはなぜ事業責任者にならないといけないのか ~Webサービスを作る茨の道~
gimupop
6
2.3k
はてなのチーム開発一巡り / Hatena Engineer Seminar 30
daiksy
0
200
こんなに違うよ MySQLとPostgreSQL /
sakaik
18
11k
アウトプット エンジニアリング 〜 インプット偏重から脱却して飛躍するための発想 /20240621-AWS-Summit-hatano-output
opelab
5
420
20240626_PLATEAU_AWARD説明会
nokonoko1203
0
170
Next.js の fetch 拡張とキャッシュ機構の違いを理解する
ryo_manba
2
530
Macインスタンスを活用したアプリE2Eテストの実現
coconala_engineer
1
130
Comment nous avons transformé les Restos du Coeur en Cloud Provider
ju_hnny5
1
330
Featured
See All Featured
Speed Design
sergeychernyshev
3
150
How to Ace a Technical Interview
jacobian
273
22k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
27
2.4k
Docker and Python
trallard
36
2.8k
A Tale of Four Properties
chriscoyier
154
22k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
22
1.8k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
Building a Modern Day E-commerce SEO Strategy
aleyda
22
6.6k
Design by the Numbers
sachag
276
18k
The Pragmatic Product Professional
lauravandoore
28
6k
Ruby is Unlike a Banana
tanoku
96
10k
Adopting Sorbet at Scale
ufuk
69
8.7k
Transcript
ݎ࿚ͳTCPαʔόΛ࡞ΔͨΊʹ ʙkatsubushiͷݟ͔Βʙ 2019-06-22 Kamakura.go #5 @fujiwara
@fujiwara SRE(૯෦) github.com/fujiwara sfujiwara.hatenablog.com WEB+DB Press vol.111 SREνʔϜ࿈ࡌ Perl Hackers
HubͰAWS X-RayͷΛॻ ͖·ͨ͠
katsubushi - ࢄϢχʔΫID࠾൪ػ • ϢχʔΫͳ࣌ܥྻॱͷInt64 IDΛൃߦ͢Δ • ࢄڥ(ෳ)ͰඃΒͣಈ͘ • Memcached
ProtocolΛ࣮͍ͯ͠Δ • Goͷಠ࣮ࣗϛυϧΣΞ github.com/kayac/go-katsubushi
ݎ࿚ͳαʔόʁ ݎ࿚ is ... • ϦιʔεϦʔΫ͠ͳ͍ • མͪͳ͍ • (ϓϩμΫτͷੑ্࣭)
ID͕ܾͯ͠ඃΒͳ͍ • →γϟʔσΟϯά͞ΕͨผDBʹಉҰID͕ൃߦ͞ΕͨΒࢮ
memcached protocol ͷ࠾༻ཧ༝ • ܰྔ • HTTPͷΑ͏ʹ༨ܭͳϔομ͕ͳ͍ • Int64 Λ1ݸฦ͢ͷʹϔομ͕Կඦbyteͱ͔ͪΐͬͱ…
• ΫϥΠΞϯτ͕ߴ • ࣌ओʹ༻͍ͯͨ͠PerlʹCache::Memcached::Fastͱ ͍͏ߴͳ࣮ IDେྔʹൃߦ͢Δ͜ͱ͕͋ΔͷͰ͍ܰͷ͕Α͍
HTTPͳΒͱ͔͘memcached protocolΛॲཧ͢Δαʔό طଘͷϑϨʔϜϫʔΫ͕ͳ͍ ॻ͚͍͍ΑͶɺ؆୯ͩ͠
Go Ͱ TCP αʔό net package Ͱ؆୯ʹͰ͖Δ Listen, Acceptͯ͠ConnΛgoroutineͰॲཧ͢Δ͚ͩ ln,
err := net.Listen("tcp", ":8080") if err != nil { // handle error } for { conn, err := ln.Accept() if err != nil { // handle error } go handleConnection(conn) }
Go Ͱ TCP Echo αʔόɺ؆୯ func handleConnection(conn net.Conn) { defer
conn.Close() // ൈ͚Δͱ͖ʹஅ b := bufio.NewReader(conn) for { line, err := b.ReadString('\n') // 1ߦಡΉ if err != nil { return } _, err = io.WriteString(conn, line) // ΦϜฦ͠ if err != nil { return } } }
͔͠͠ݱ࣮໘
ϦιʔεϦʔΫ͠ͳ͍
ແ௨৴λΠϜΞτ • ܨ͗ͬͺͳ͠Ͱ͍ͳ͘ͳΔΫϥΠΞϯτରࡦ • ΫϥΠΞϯτ͕TCPΛ͖Ε͍ʹஅ͢ΔͱݶΒͳ͍ • αʔόଆͰΒͳ͍ͱίωΫγϣϯ(goroutine)͕ϦʔΫ͢Δ
net/Conn.SetDeadline ίωΫγϣϯʹࢦఆͨ࣌͠ࠁ·Ͱ௨৴͕ͳ͔ͬͨΒ ΤϥʔʹͳΔΑ͏ઃఆ͢Δ ௨৴ཱ֬ޙɺಡΈॻ͖Ͱ͖ͨλΠϛϯάͰ࠶ઃఆ → ແ௨৴λΠϜΞτ͕࣮Ͱ͖Δ type Conn interface {
SetDeadline(t time.Time) error }
for { conn, err := ln.Accept() if err != nil
{ // handle error } conn.SetDeadline(time.Now().Add(idleTimeout)) go handleConnection(conn) } func handleConnection(conn net.Conn) { defer conn.Close() buf := bufio.NewReader(conn) for { line, _ := buf.ReadString('\n') // 1ߦಡΉ conn.SetDeadline(time.Now().Add(idleTimeout)) // ^λΠϜΞτԆ
ϓϩηεऴྃ࣌ʹஅ αʔόͪΌΜͱஅ͠ͳ͍ͱߦّ͕ѱ͍ΑͶ? func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
͜͜ͰͰ͢ ͜ͷίʔυͰͳʹ͔͕ϦʔΫ͠·͢ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
goroutineϦʔΫ handleConn͕returnͯ͠goroutine͕ࢮͳͳ͍ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ͜Ε͕ݺΕΔ(=ϓϩηεऴྃ)·Ͱ conn.Close() // goroutine ͕ੜ͖Δ }() // ...
मਖ਼ํ๏1 ࢠ context Λ࡞ͬͯ defer cancel() ࣗൃతऴྃ࣌ɺऴྃ࣌ͲͪΒͰίωΫγϣϯஅͱ goroutineऴ͕ྃͰ͖ΔͷͰϦʔΫ͠ͳ͍ func (app
*App) handleConn(ctx context.Context, conn net.Conn) { ctx2, cancel := context.WithCancel(ctx) defer cancel() go func() { <-ctx2.Done() conn.Close() }() // ... 1 Fix goroutine leak. #28 https://github.com/kayac/go-katsubushi/pull/28/files
མͪͳ͍
ʮ͜ͷ katsubushi ಈ͍ͯΔ͔ͳ?ʯ $ curl localhost:11212 VALUE / 0 18
591644124189298688 VALUE HTTP/1.1 0 18 591644124189298689 END ERROR ERROR ERROR ʮIDฦ͖͚ͬͯͨͲͳʹ͜Ε??ʯ ʮ͋ɺcurlͰୟ͍ͯͨɻɻɻʯ
Կ͕ى͖͍͔ͯͨ curl katsubushi ʹ HTTP ͰϦΫΤετͨ͠(વ) $ curl -v
127.0.0.1:11212 > GET / HTTP/1.1 > Host: 127.0.0.1:11212 > User-Agent: curl/7.65.1 > Accept: */* > (ۭߦ) HTTPͷϦΫΤετ memcached protocol ͰղऍͰ͖Δʂ memcached protocol Ͱͷऔಘ: GET key1 key2 HTTP Ͱ GET ϦΫΤετͷ1ߦ: GET / HTTP/1.1
ͭ·Γ͜͏͍͏Ϩεϙϯε͕ฦΔ VALUE / 0 18 # / ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298688
VALUE HTTP/1.1 0 18 # HTTP/1.1 ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298689 END # HTTPͷ1ߦʹϨεϙϯε͕ฦͤͨΑʂ ERROR # Host: ͱ͔Βͳ͍! ERROR # User-Agnet: ͱ͔Βͳ͍!! ERROR # Accept: ͱ͔Βͳ͍!!!
Ұํͦͷ͜Ζ HTTP Λ৯ͨ katsubushi panic: runtime error: invalid memory
address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x134807f] goroutine 20 [running]: github.com/kayac/go-katsubushi.(*App).handleConn(0xc00012a000, 0x14a3de0, 0xc0000ac038) /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:160 +0x2cf created by github.com/kayac/go-katsubushi.(*App).Listen /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:131 +0x2ed ʮ͠ɺࢮΜͰΔ…ʯ
Կ͕ى͖͍͔ͯͨ ಡΈऔͬͨϦΫΤετߦ([]byte)ΛίϚϯυʹ࣮ͯ͠ߦ HTTPϔομ͓ΘΓͷۭߦΛಡΜͩͱ͜ΖͰ… cmd, err := app.BytesToCmd(scanner.Bytes()) if err !=
nil { return } err = cmd.Execute(app, conn) // ͜͜Ͱࢮ
൜ਓ func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { // ۭߦͳͷͰ 0 byte return nil, nil // ← ͍ͭ͜ } foo, err := Foo() ͱ͍͏ΠϯλʔϑΣʔεΛΈͨ ී௨ͷGopher͕ߟ͑Δ͜ͱ ʮerr != nil ͳΒ foo == nil ͔͔ͩΒؾΛ͚ͭΑ͏ʯ ʮerr == nil ͳΒ foo != nil ͩΑͶʂʯ
nil, nil Λฦ͢ͷ͕ѱ͍ ॻ͍ͨͷͲ͚ͬͪࣗͩͲ݀ʹམͪͨ ௨ৗmemcached protocolͰϦΫΤετʹۭߦདྷͳ͍ ຊ൪ӡ༻தͷϓϩηε͕ଈࢮͯ͠ͼͬ͘Γͨ͠ katsubushiΫϥΠΞϯτϥΠϒϥϦfailoverΛ࣮͍ͯͨ͠ daemontools Ͱ͙͢ʹ࠶ىಈ͞Εͨ
ͷͰக໋ইʹͳΒͳ͔ͬͨ γεςϜશମͰݎ࿚ʹ͍͖ͯ͠·͠ΐ͏…
ରԠ2 func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { - return nil, nil + return nil, errors.New("No command") Βͳ͍ίϚϯυ͕དྷͨΒଈ࠲ʹଓΛͬͨ΄͏͕͍͍͔ 2 Never die #8 https://github.com/kayac/go-katsubushi/pull/8
༨ஊ1 ࣮ curl telnet Ͱ͖·͢ $ curl telnet://127.0.0.1:11212 GET
id VALUE id 0 18 591655476635111424 END quit
༨ஊ2 ʮ1ߦΛղऍͯ͠key͕ / HTTP/1.[01] ͳΒHTTPɻɻɻʯ ʮͭ·Γಉ͡ϙʔτͰ memcached ͱ HTTP ྆ରԠ
αʔό͕ॻ͚ΔͷͰ!??ʯ ్த·Ͱॻ͍ͯਖ਼ؾʹͬͯʹ
(ϓϩμΫτͷੑ্࣭) ID͕ܾͯ͠ඃΒͳ͍
katsubushi ID ࣌ࠁϕʔε +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID |
Sequence | +-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z ࣌ࠁ͕Δͱಉ͡ID͕ൃߦ͞ΕΔՄೳੑ͕͋Δ
monotonic time https://golang.org/doc/go1.9#monotonic-time Go 1.9ͰOS࣌ࠁ͕ͬͯ time.Now() Βͳ͍ monotonic time͕࣮͞Εͨ 2017−01−01ͷӞඵൃੜ࣌
Cloudflare͕࣌ࠁͷר͖ΓͰো3 ࣌ࠁ͕ͬͨ݁Ռ math/rand.Int63n ʹෛͷ͕Γpanic 3 How and why the leap second affected Cloudflare DNS https://blog.cloudflare.com/how-and-why-the-leap-second-affected- cloudflare-dns/
monotonic time Ҏલͷ katsubushi ts := g.timestamp() // for rewind
of server clock if ts < g.lastTimestamp { return 0, errors.New("system clock was rollbacked") } // ུ g.lastTimestamp = ts ࣌ࠁ͕ר͖ͬͨΒΤϥʔʹͯ͠ޚ ID͕ඃΔ͙Β͍ͳΒൃߦͰ͖ͳ͍ํ͕Ϛγ
monotonic time ରԠ4 katsubushi ID ͷ࣌ࠁ෦ 2015-01-01T00:00:00 UTC Λىݯͱͨ͠ܦաඵ var
Epoch = time.Date(2015, 1, 1, 0, 0, 0, 0, time.UTC) d := time.Now().Sub(Epoch) ͜ͷΑ͏ʹɺ࣌ࢦఆͰ࡞͔ͬͨΒܭࢉ͢Δͱmonotonicʹ ͳΒͳ͍ 4 use Monotonic time #21 https://github.com/kayac/go-katsubushi/pull/21
monotonic time ରԠ now := time.Now() startedAt := now offset
:= now.Sub(Epoch) // ىಈ࣌ͷ࣌ࠁͱepochͷࠩΛܭࢉ͓ͯ͘͠ // ... d := time.Now().Sub(startedAt) + offset time.Now() Ͱ࡞ͬͨಉ࢜Λൺֱͯ͠offsetΛௐ͢Ε ר͖Βͳ͍
࣌ࠁͷଞʹඃΔՄೳੑ͕͋Δ෦͕… +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID | Sequence |
+-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z WorkerID = ಉ࣌ʹಈ࡞͍ͯ͠ΔϓϩηεؒͰҰҙͷID
RedisΛͬͯҰҙੑΛ୲อ͢Δ Ҏલͷൃදʹ͋ΔͷͰͲ͏ͧ https://speakerdeck.com/fujiwara3/katsubushi?slide=79
ࢹେࣄ • github.com/fukata/golang-stats-api-handler • GoͷϝτϦΫεΛऔΔ • mackerel-plugin-gostats ͋ΔΑ • net/http/pprof
ͰϓϩϑΝΠϧऔಘͷޱΛ։͚Δ
·ͱΊ • GoͰTCPαʔόΛ࡞Δͷ؆୯ • Ͱ࣮ӡ༻Ͱݎ࿚ͳͷΛ࡞Δʹ͍Ζ͍Ζ͋Δ • Ұݟ؆୯ʹݟ͑ΔͷͰɺࣗલ࣮পͷୈҰา