Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
マイクロサービス時代 のHTTPクライアントの作り方 / how we build our http client for microservice
LINE Developers
PRO
July 13, 2019
Technology
3
1.3k
マイクロサービス時代 のHTTPクライアントの作り方 / how we build our http client for microservice
Go Conference'19 Summer in Fukuokaでの登壇資料です
https://fukuoka.gocon.jp/ja/
LINE Developers
PRO
July 13, 2019
Tweet
Share
More Decks by LINE Developers
See All by LINE Developers
LINE Technology Selection at New Live Commerce Service
line_developers
PRO
0
18
The application of formal methods in Kafka reliability engineering
line_developers
PRO
1
210
How to send feedback on a document
line_developers
PRO
1
47
LINEのB2Bプラットフォームにおけるトラブルシューティング2選
line_developers
PRO
4
310
The role of the data organization as a business progresses
line_developers
PRO
3
870
シン・スウィフト 5.7 ~ Swift5.7を振り返る ~ / New Swift 5.7
line_developers
PRO
0
900
LINEポイントクラブにおける PerlからKotlinへの移行を振り返る / The migration from Perl to Kotlin at LINE Point Club
line_developers
PRO
0
290
Server-side Kotlin in LINE Messaging API
line_developers
PRO
0
280
LIFF Deep Dive 2022
line_developers
PRO
1
920
Other Decks in Technology
See All in Technology
GeoLocationAnchor and MKTileOverlay
toyship
0
110
データエンジニアと作るデータ文化
yuki_saito
4
1.7k
Build 2022で発表されたWindowsアプリ開発のあれこれ振り返ろう
hatsunea
1
390
今どきのLinux事情
tokida
44
36k
サイボウズの アジャイル・クオリティ / Agile Quality at Cybozu
cybozuinsideout
PRO
4
2.4k
Retca Cloud
bau
0
530
JUnit5.7, 5.8の新機能紹介 #jjug_ccc #jjug_ccc_b / junit 5.7, 5.8 new features
kyonmm
PRO
2
420
Apa itu DevOps & Kenapa perlu belajar DevOps?
dicodingevent
0
110
Azure Arc Virtual MachineとAzure Arc Resource Bridge / VM provisioning through Azure portal on Azure Stack HCI (preview)
sashizaki
0
160
Oracle Cloud Infrastructure:2022年6月度サービス・アップデート
oracle4engineer
PRO
0
160
MySQL HeatWaveの新しいトピック! / MySQL HeatWave Update 202206
yoshiakiyamasaki
0
100
What's new in Vision
satotakeshi
0
220
Featured
See All Featured
GraphQLとの向き合い方2022年版
quramy
16
8.3k
Imperfection Machines: The Place of Print at Facebook
scottboms
253
12k
Typedesign – Prime Four
hannesfritz
34
1.4k
Principles of Awesome APIs and How to Build Them.
keavy
113
15k
Designing for Performance
lara
597
63k
Building a Scalable Design System with Sketch
lauravandoore
448
30k
Product Roadmaps are Hard
iamctodd
34
6.5k
JazzCon 2018 Closing Keynote - Leadership for the Reluctant Leader
reverentgeek
172
8.4k
Six Lessons from altMBA
skipperchong
14
1.4k
Become a Pro
speakerdeck
PRO
3
840
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
29
4.3k
Clear Off the Table
cherdarchuk
79
280k
Transcript
ϚΠΫϩαʔϏε࣌ ͷHTTPΫϥΠΞϯτͷ࡞Γํ @dxhuy Observability Team / LINE Corp
@dxhuy https://github.com/huydx
None
Developer
• LINEͷObservabilityνʔϜʹ͍ͭͯ • ϚΠΫϩαʔϏεͷམͱ݀͠ • ෦ͰΧελϜͨ͠HTTPΫϥΠΞϯτ ͷઃܭͱհ ࠓͷ
Engineering LINE’S OBSERVABILITY TEAM
Engineering OBSERVABILITY IS NEW WAVE
Πϝʔδతʹ ࣾDATADOGΛ࡞͍ͬͯΔ
Engineering Log ج൫ Metric ج൫ ࢄ τϨʔγ ϯάج൫ ZipkinͷOSS ίϯτϦϏϡʔτ
Engineering GO͍ͬͯΔͷʁ Metric Agent Time Series Storage Metric API Server
͜Ε͔Β࡞Δͷ ͲΜͲΜGO࠾༻ʂ CLI tool Metric Alert Engine
> 1.000.000 metrics per sec εέʔϧνϟϨϯδ
> 2.000 read per sec εέʔϧνϟϨϯδ
Engineering Metrics૿Ճ Ϗδωεεέʔϧ
Engineering GOPHERੵۃ࠾༻தʂ ιϑτΣΞΤϯδχΞ (Software EngineerɺEngineering Efficiency)ʲLINEϓϥοτϑΥʔϜʳ
Engineering ࠓͷຊ
Engineering ϚΠΫϩαʔϏε ͍ͬͯ·͔͢
Engineering ίϛϡχέʔγϣϯ(RPC) ͕େࣄ
Engineering twitch/Twirp GRPC Thrift REST Http RPCख๏͕༷ʑ ࠓޙ૿͑ͦ͏
Engineering GRPC, THRIFTͳͲશͯͷ γεςϜΛม͑ͳ͍ͱߦ͚ͳ͍
Engineering ͢Ͱʹଘࡏ͍ͯ͠Δ APIαʔϏε͕ࠞࡏ net/http͕ఆ൪
Engineering NET/HTTPͪΐͬͱ Γͳ͍
Engineering ϚΠΫϩαʔϏε௨৴ͷ ΫϥΠΞϯτඞཁͳੑ࣭
Engineering αʔϏεͷελοΫ ωοτϫʔΫԆ ϦΫΤετࣦഊ αʔϏεಥવؾઈ https://www.flickr.com/photos/chrish_99/8526019374/
Engineering https://www.flickr.com/photos/chrish_99/8526019374/
Engineering ϦϞʔτίʔϧ͕ࣦഊ͠ ͯճ෮Ͱ͖ΔΑ͏ʹͳ Δ ࣮ύλʔϯ - Retry - CircuitBreaker ճ෮ੑ
ίʔϧઌ͕ࢄͰ͖ΔΑ ͏ʹͳΔ ࣮ύλʔϯ - Load Balancing - Service Discovery ߴՄ༻ੑ ࢲ͕ཉ͍͠ HTTPΫϥΠΞϯτ
Engineering • CircuitBreaker, RetryͳͲ͕ἧ͏ύοέʔδͳͲଘࡏ͢Δ͕ • Ұͭͷػೳ͔͍࣋ͬͯ͠ͳ͍ͷ͕ଟ͍ • ඪ४ͷnet/httpͷΠϯλʔϑΣʔεʹ૬ੑѱ͍ɺίʔυϕʔεͷม ߋ͕ଟ͘ͳΔ •
BoilerPlate͕ଟ͍ • ύϑΥʔϚϯεྑ͘ͳ͍ • ֎෦dependencies͕ଟ͍ • ྫɿgojek/heimdall, go-kitܥͷϥΠϒϥΠͳͲ ͭ·ΓΧελϜNET/HTTP ΫϥΠΞϯτཉ͍͠
Engineering ࣾͰΧελϜNET/HTTP ࡞Γ·ͨ͠
Engineering XHTTPύοέʔδ ઃܭͱ͍ํͷհ
Engineering • ඪ४ͷnet/httpͷΠϯλʔϑΣʔεͰ͖Δ͚ͩै͏ • ͏ଆʢϢʔβʣͷมߋΛ࠷খݶʹ͢Δ • net/httpͷϥούϥΠϒϥϦͱͯ͠ • ඞཁͳͷશ෦ἧ͏ •
ճ෮ੑɺߴՄ༻ੑͷػೳΛἧ͏ • ϝτϦΫεɺTracingͷػೳΛἧ͏ • ଥͳσϑΥϧτʢλΠϜΞτͳͲʣ ઃܭࢥߟ
Engineering package xhttp type ClientConfig struct {…} func NewClient(config *ClientConfig)
*Client {} func NewRequestX(method string, addr *AddressGroup, path string, body io.Reader) (*Request, error) func (c *Client) DoX(req *Request, option ...CallOption) (*Response, error) func (c *Client) PostX(addr *AddressGroup, path string, contentType string, body io.Reader, options ...CallOption) (resp *Response, err error) { func (c *Client) PostFormX(addr *AddressGroup, path string, data url.Values, options ...CallOption) (resp *Response, err error) func (c *Client) GetX(addr *AddressGroup, path string, options ...CallOption) (resp *Response, err error) { NET/HTTPͱಉ͡ΠϯλʔϑΣΠε
Engineering addrGroup := xhttp.MustNewAddressGroup( //AddressGroup͋ͱ΄Ͳհ͠·͢ “dns: //txt:your-domain.com:8080”, xhttp.WithLoadBalancer(xhttp.RoundRobin), ) reqP,
_ := xhttp.NewRequestX("POST", addrGroup, "/", nil) res, err = client.DoX(reqG) res, err = client.PostFormX(addrGroup, "/", url.Values{}) res, err = client.GetX(addrGroup, "/") ͍ํͷΠϝʔδ
Engineering ߴՄ༻ੑͷ
Upstream 2 die xhttp client Upstream 1 αʔό͍ͭͰࢮ͵Մೳੑ Upstream 3
Engineering • ීஈ Reverse Proxy (VIP)ͳͲར༻͕ଟ͍ • ৽͍͠αʔϏεՃΛ͢Δͨͼʹ৽͍͠ Reverse Proxyಋೖ͢Δ
ͷ͕໘ • Solution • Client Side Load Balancing • ΫϥΠΞϯτίʔϧઌͷαʔόάϧʔϓΛࣗͰίϯτϩʔ ϧ͢Δ͜ͱ • ΫϥΠΞϯτίʔϧઌͷαʔόάϧʔϓɺͲΕΛݺͼग़͢ͷ ͔ͷϩʔυόϥϯγϯάϩδοΫΛ࣮͢Δ͜ͱ αʔόάϧʔϓ͕ඞཁ
Engineering type Address struct { Port string Host string }
type AddressGroup struct { scheme Scheme name string resolver Resolver loadBalancer LoadBalancer cacheAddress []Address closeCh chan interface{} } AddressGroup ʢαʔόάϧʔϓͷநԽʣ
Engineering type Resolver interface { Scheme() Scheme Init(address string) error
UpdateCh(context.Context) <-chan []Address Close() } type LoadBalancer interface { Pick() (Address, error) HandleStateChange(list []Address) error } Resolver + LoadBalancer
Engineering type DnsResolver struct { domain string updateCh chan []Address
doneCh chan interface{} } func (d *DnsResolver) Scheme() Scheme { return Scheme("dns") } func (d *DnsResolver) Init(name string) error { d.resolve() } func (d *DnsResolver) resolve() { go func() { // resolve d.domain d.updateCh <- addresses time.Sleep(TTL) } } Example: DnsResolve
Engineering type roundRobinLB struct { list []Address next uint32 mu
sync.RWMutex } func (rr *roundRobinLB) Pick() (Address, error) { l := len(rr.list) if l == 0 { return Address{}, fmt.Errorf("empty list of address") } v := atomic.AddUint32(&rr.next, 1) return rr.list[int(v)%len(rr.list)], nil } func (rr *roundRobinLB) HandleStateChange(list []Address) error { rr.mu.Lock() defer rr.mu.Unlock() rr.list = list return nil } Example: Round Robin LB
Engineering ͏ଆBOILER PLATE͕΄ͱΜͲͳ͍ addrGroup := xhttp.MustNewAddressGroup( //AddressGroup͋ͱ΄Ͳհ͠·͢ “dns: //txt:your-domain.com:8080”, xhttp.WithLoadBalancer(xhttp.RoundRobin),
) reqP, _ := xhttp.NewRequestX("POST", addrGroup, "/", nil) ϥΠϒϥϦଆͰ schemeύʔεͯ͠Resolver ΛΞαΠϯ͢Δ
Engineering • Load Balancing is impossible • https://www.youtube.com/watch?v=kpvbOzHUakA • Round
Robin Upstreamϗετͷঢ়ଶʢLatency, Φʔόϩʔυঢ় گʣͳͲΛҙࣝͰ͖ͣɺͣͬͱಉ͡ѱ͍ঢ়ଶͷϗετʹಉ͡ϦΫΤε τΛ͛Δɹ • P2C ΞϧΰϦζϜ • Power of 2 Random Choice • Upstreamͷঢ়ଶΛୡඞཁ͋Δ • https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf ༨ஊ ϩʔυόϥγϯάΞϧΰϦζϜ͍͠
Engineering ճ෮ੑͷ
Engineering Upstream 2 die xhttp client Upstream 1 αʔό͍ͭͰࢮ͵Մೳੑ Upstream
3
Engineering Upstream 2 die xhttp client Upstream 1 die (શʣαʔό͍ͭͰࢮ͵Մೳੑ
Upstream 3 die
Engineering • શʹࢮ͵͡Όͳͯ͘ɺ(GCͳͲͷཧ༝ͰʣLatency͕Ͷ্͕Δ • దͳσϑΥϧτλΠϜΞτ͕ඞཁ • ଈ࣌తͳϦΫΤετࣦഊ (500ΤϥʔͳͲʣ • దͳϦτϥΠ
• ܧଓతͳϦΫΤετࣦഊ (ωοτϫʔΫࣦഊͳͲʣ • Ͱ͖Δ͚ͩૣ͍ஈ֊ͰϦΫΤετΛυϩοϓ͢Δ • CircuitBreakerύʔλϯɿFail Fast Error αʔό͕ࢮ͵ͱ͖Ͳ͏͠·͢ʁ
Engineering • net/httpσϑΥϧτλΠϜΞτ͋Γ·ͤΜʂ • ແݶʹϦΫΤετͭ • xhttpͰσϑΥϧτͰ30sઃఆՄೳɺ֎͔Β͢ͷՄೳ λΠϜΞτʹ͍ͭͯ func NewClient(config
*ClientConfig) *Client { client.client = &http.Client{ Timeout: config.RequestTimeout, Transport: newRoundTripper(config.RoundTripper, config.MetricRegistry), }… }
Engineering Dial TLS Handshake Request Resp Headers Resp Body Idle
Client.Do http.Client.Timeout net.Dialer.Timeout http.Transport.TLSHandshake.Timeout http.Transport.ResponseHeader.Timeout http.Transport.IdleConn.Timeout https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/ ߹ʹΑΓࡉ͔͘ίϯτϩʔϧՄೳ
Engineering • Լखʹretry͢Δ͜ͱʹࣾαʔϏεΛDDOS͢Δ͜ͱ • ඞͣ Backoff ( Exponential )͢Δ͜ͱ •
RetryϦΫΤετείʔϓͷOption (ClientείʔϓͰ͋Γ·ͤΜʣ RETRYʹ͍ͭͯ res, err = client.DoX(reqG, xhttp.WithRetry(&xhttp.RetryOptions{ MaxAttempts: 2, RetryOn: func(response *http.Response) bool { return response.StatusCode != 200 }, TimeoutPerAttempt: time.Second, }))
Engineering • ૣ͍ஈ֊Ͱυϩοϓ͢Δख๏ͷҰͭͱͯ͠ CircuitBreakerύλʔϯɹ • https://engineering.linecorp.com/ja/blog/circuit-breakers-for- distributed-services/ • ϗετͷঢ়ଶΛʮࣦഊ͢ΔϦΫΤετͷʯͰஅ •
ʮࣦഊʯఆٛࣗମϢʔβʹͤΔ • ঢ়ଶʹΑΓʮϦΫΤετՄೳʯͱʮϦΫΤετෆՄೳʯͰ͚Δ • ϦΫΤετෆՄೳͷ߹୯७ʹ FailFastErr ฦ͢ FAIL FASTରࡦʹ͍ͭͯ
Engineering • ׂͱύϥϝʔλ͕ଟ͍ͷͰɺΞϧΰϦζϜࣗମΛཧղ͢Δ্ઃఆ͢Δ ඞཁ͕͋Δ CIRCUIT BREAKER type CircuitBreakerConfig struct {
noop bool FailureRateThreshold float64 MinimumRequestThreshold int64 TrialRequestInterval time.Duration CircuitOpenWindow time.Duration CounterSlidingWindow time.Duration CounterUpdateInterval time.Duration IsFailed FailedJudger KeyBuilder KeyBuilder } client := xhttp.NewClient(&xhttp.ClientConfig{ CircuitBreakerConfig: &xhttp.CircuitBreakerConfig{}, })
Engineering • > ϗετͷঢ়ଶΛʮࣦഊ͢ΔϦΫΤετͷʯͰஅ • Ͳ͏ͬͯૣ͑͘Δʁ ༨ஊɹHIGH PERFORMANCE CIRCUITBREAKERͷͨΊʹ var
c int64 c ++ var c atomic.Int64 c.Add(1) • AtomicͬͯcontentionมΘΒͳ͍ʢಉ͡มʹࢀর͢Δ͜ͱʣ • https://github.com/linxGnu/go-adder • Java LongAdderͷΞΠσΟΞʢatomicΛsharding) • ϚϧνϧʔνϯڥͰatomic.Int64ΑΓ2~4ഒૣ͍
Engineering METRICS/TRACINGͷ
Engineering • ObservabilityνʔϜͳͷͰͦͷॏཁੑΛ͘͢͝ཧղ͍ͯ͠Δ • ͑ͳ͍ͱ͍͚ͳ͍ͷ • ϦΫΤετLatencyߴ͍Ͱ͔͢ʁ • Latencyߴ͍࣌ͳͥʁ OBSERVABILITYͷॏཁੑ
• Latencyߴ͍͔Ͳ͏͔MetricsͰ͑Δ • Metrics Instrumentation prometheus client ͏ • ͳͥʁΛ͑Δͷ Metrics + TracingͰ͑Δ • Tracing Instrumentation zipkin-go
Engineering ROUNDTRIPPERར༻ͯ͠INSTRUMENT type ClientConfig struct { MetricRegistry *prometheus.Registry Tracer *zipkin.Tracer
…… } func newRoundTripper(originRt http.RoundTripper, mr *prometheus.Registry, tc *zipkin.Tracer) *roundtripper { … .. } client.client = &http.Client{ Timeout: config.RequestTimeout, Transport: newRoundTripper(config.RoundTripper, config.MetricRegistry, config.Tracer), }
Engineering • ΧελϚΠζ͢ΔͨΊͷϑοΫ • ϦΫΤετૹΔॲཧΛΧελϚΠζɿ RoundTripperΛ͢ • ίωΫγϣϯΛ࡞ΔॲཧΛΧελϚΠζɿDialContextΛ͢ ༨ஊɹNET/HTTPࢁ ϑοΫఏڙͯ͘͠ΕͯΔ
• ϝτϦΫεΛऔಘ͢ΔͨΊͷϑοΫ • http/ClientTraceेɹ • https://blog.golang.org/http-tracing
Engineering ࠷ޙ
Engineering • Ͱ͖Εࣾɺ͋Δ͍νʔϜڞ௨ͷϥΠϒϥϦἧͬͨ΄͏͕ྑ͍ • http client, log package, metrics instrument…
• ϚΠΫϩαʔϏε͍͠ɺͰ͖Εආ͚ͨ΄͏͕ྑ͍ • ͍ۙ͏ͪʹxhttpΛOSS͢Δ • go͖ͰɺେنγεςϜΛ࡞Γ͍ͨਓ͝࿈བྷ͓ͪͯ͠Γ·͢
Engineering ͝੩ௌ͋Γ͕ͱ͏͍͟͝·͢