マイクロサービス時代 のHTTPクライアントの作り方 / how we build our http client for microservice

Transcript

1. 1.

   ϚΠΫϩαʔϏε࣌୅ ͷHTTPΫϥΠΞϯτͷ࡞Γํ @dxhuy Observability Team / LINE Corp

2. 2.

   @dxhuy https://github.com/huydx

3. 3.

   None
4. 4.

   Developer

5. 5.

   • LINEͷObservabilityνʔϜʹ͍ͭͯ • ϚΠΫϩαʔϏεͷམͱ݀͠ • ಺෦ͰΧελϜͨ͠HTTPΫϥΠΞϯτ ͷઃܭͱ঺հ ࠓ೔ͷ࿩

6. 6.

   Engineering LINE’S OBSERVABILITY TEAM

7. 7.

   Engineering OBSERVABILITY IS NEW WAVE

8. 8.

   Πϝʔδతʹ͸ ࣾ಺DATADOGΛ࡞͍ͬͯΔ

9. 9.

   Engineering Log ج൫ Metric ج൫ ෼ࢄ τϨʔγ ϯάج൫ Zipkin΁ͷOSS ίϯτϦϏϡʔτ

10. 10.

    Engineering GO࢖͍ͬͯΔͷʁ Metric Agent Time Series Storage Metric API Server

    ͜Ε͔Β࡞Δ΋ͷ΋ ͲΜͲΜGO࠾༻ʂ CLI tool Metric Alert Engine
11. 11.

    > 1.000.000 metrics per sec εέʔϧνϟϨϯδ

12. 12.

    > 2.000 read per sec εέʔϧνϟϨϯδ

13. 13.

    Engineering Metrics਺૿Ճ Ϗδωεεέʔϧ

14. 14.

    Engineering GOPHERੵۃ࠾༻தʂ ιϑτ΢ΣΞΤϯδχΞ (Software EngineerɺEngineering Efficiency)ʲLINEϓϥοτϑΥʔϜʳ

15. 15.

    Engineering ࠓ೔ͷຊ୊΁

16. 16.

    Engineering ϚΠΫϩαʔϏε ࢖͍ͬͯ·͔͢

17. 17.

    Engineering ίϛϡχέʔγϣϯ(RPC) ͕େࣄ

18. 18.

    Engineering twitch/Twirp GRPC Thrift REST Http RPCख๏͕༷ʑ ࠓޙ΋૿͑ͦ͏

19. 19.

    Engineering GRPC, THRIFTͳͲ͸શͯͷ γεςϜΛม͑ͳ͍ͱߦ͚ͳ͍

20. 20.

    Engineering ͢Ͱʹଘࡏ͍ͯ͠Δ APIαʔϏε͕ࠞࡏ net/http͕ఆ൪

21. 21.

    Engineering NET/HTTP͸ͪΐͬͱ ෺଍Γͳ͍

22. 22.

    Engineering ϚΠΫϩαʔϏε௨৴ͷ ΫϥΠΞϯτඞཁͳੑ࣭

23. 23.

    Engineering αʔϏεͷελοΫ ωοτϫʔΫ஗Ԇ ϦΫΤετࣦഊ αʔϏεಥવؾઈ https://www.flickr.com/photos/chrish_99/8526019374/

24. 24.

    Engineering https://www.flickr.com/photos/chrish_99/8526019374/

25. 25.

    Engineering ϦϞʔτίʔϧ͕ࣦഊ͠ ͯ΋ճ෮Ͱ͖ΔΑ͏ʹͳ Δ ࣮૷ύλʔϯ - Retry - CircuitBreaker ճ෮ੑ

    ίʔϧઌ͕෼ࢄͰ͖ΔΑ ͏ʹͳΔ ࣮૷ύλʔϯ - Load Balancing - Service Discovery ߴՄ༻ੑ ࢲ͕ཉ͍͠ HTTPΫϥΠΞϯτ
26. 26.

    Engineering • CircuitBreaker, RetryͳͲ͕ἧ͏ύοέʔδͳͲ΋ଘࡏ͢Δ͕ • Ұͭͷػೳ͔͍࣋ͬͯ͠ͳ͍ͷ͕ଟ͍ • ඪ४ͷnet/httpͷΠϯλʔϑΣʔεʹ૬ੑѱ͍ɺίʔυϕʔεͷม ߋ͕ଟ͘ͳΔ •

    BoilerPlate͕ଟ͍ • ύϑΥʔϚϯεྑ͘ͳ͍ • ֎෦dependencies͕ଟ͍ • ྫɿgojek/heimdall, go-kitܥͷϥΠϒϥΠͳͲ ͭ·ΓΧελϜNET/HTTP ΫϥΠΞϯτཉ͍͠
27. 27.

    Engineering ࣾ಺ͰΧελϜNET/HTTP ࡞Γ·ͨ͠

28. 28.

    Engineering XHTTPύοέʔδ ઃܭͱ࢖͍ํͷ঺հ

29. 29.

    Engineering • ඪ४ͷnet/httpͷΠϯλʔϑΣʔεͰ͖Δ͚ͩै͏ • ࢖͏ଆʢϢʔβʣͷมߋΛ࠷খݶʹ͢Δ • net/httpͷϥούϥΠϒϥϦͱͯ͠ • ඞཁͳ΋ͷ͸શ෦ἧ͏ •

    ճ෮ੑɺߴՄ༻ੑͷػೳΛἧ͏ • ϝτϦΫεɺTracingͷػೳΛἧ͏ • ଥ౰ͳσϑΥϧτʢλΠϜΞ΢τͳͲʣ ઃܭࢥߟ
30. 30.

    Engineering package xhttp type ClientConfig struct {…} func NewClient(config *ClientConfig)

    *Client {} func NewRequestX(method string, addr *AddressGroup, path string, body io.Reader) (*Request, error) func (c *Client) DoX(req *Request, option ...CallOption) (*Response, error) func (c *Client) PostX(addr *AddressGroup, path string, contentType string, body io.Reader, options ...CallOption) (resp *Response, err error) { func (c *Client) PostFormX(addr *AddressGroup, path string, data url.Values, options ...CallOption) (resp *Response, err error) func (c *Client) GetX(addr *AddressGroup, path string, options ...CallOption) (resp *Response, err error) { NET/HTTPͱಉ͡ΠϯλʔϑΣΠε
31. 31.

    Engineering addrGroup := xhttp.MustNewAddressGroup( //AddressGroup͸͋ͱ΄Ͳ঺հ͠·͢ “dns: //txt:your-domain.com:8080”, xhttp.WithLoadBalancer(xhttp.RoundRobin), ) reqP,

    _ := xhttp.NewRequestX("POST", addrGroup, "/", nil) res, err = client.DoX(reqG) res, err = client.PostFormX(addrGroup, "/", url.Values{}) res, err = client.GetX(addrGroup, "/") ࢖͍ํͷΠϝʔδ
32. 32.

    Engineering ߴՄ༻ੑͷ࿩

33. 33.

    Upstream 2 die xhttp client Upstream 1 αʔό͸͍ͭͰ΋ࢮ͵Մೳੑ΋ Upstream 3

34. 34.

    Engineering • ීஈ͸ Reverse Proxy (VIP)ͳͲར༻͕ଟ͍ • ৽͍͠αʔϏε௥ՃΛ͢Δͨͼʹ৽͍͠ Reverse Proxyಋೖ͢Δ

    ͷ͕໘౗ • Solution • Client Side Load Balancing • ΫϥΠΞϯτ͸ίʔϧઌͷαʔόάϧʔϓΛࣗ෼Ͱίϯτϩʔ ϧ͢Δ͜ͱ • ΫϥΠΞϯτ͸ίʔϧઌͷαʔόάϧʔϓɺͲΕΛݺͼग़͢ͷ ͔ͷϩʔυόϥϯγϯάϩδοΫΛ࣮૷͢Δ͜ͱ αʔόάϧʔϓ͕ඞཁ
35. 35.

    Engineering type Address struct { Port string Host string }

    type AddressGroup struct { scheme Scheme name string resolver Resolver loadBalancer LoadBalancer cacheAddress []Address closeCh chan interface{} } AddressGroup ʢαʔόάϧʔϓͷந৅Խʣ
36. 36.

    Engineering type Resolver interface { Scheme() Scheme Init(address string) error

    UpdateCh(context.Context) <-chan []Address Close() } type LoadBalancer interface { Pick() (Address, error) HandleStateChange(list []Address) error } Resolver + LoadBalancer
37. 37.

    Engineering type DnsResolver struct { domain string updateCh chan []Address

    doneCh chan interface{} } func (d *DnsResolver) Scheme() Scheme { return Scheme("dns") } func (d *DnsResolver) Init(name string) error { d.resolve() } func (d *DnsResolver) resolve() { go func() { // resolve d.domain d.updateCh <- addresses time.Sleep(TTL) } } Example: DnsResolve
38. 38.

    Engineering type roundRobinLB struct { list []Address next uint32 mu

    sync.RWMutex } func (rr *roundRobinLB) Pick() (Address, error) { l := len(rr.list) if l == 0 { return Address{}, fmt.Errorf("empty list of address") } v := atomic.AddUint32(&rr.next, 1) return rr.list[int(v)%len(rr.list)], nil } func (rr *roundRobinLB) HandleStateChange(list []Address) error { rr.mu.Lock() defer rr.mu.Unlock() rr.list = list return nil } Example: Round Robin LB
39. 39.

    Engineering ࢖͏ଆ͸BOILER PLATE͕΄ͱΜͲͳ͍ addrGroup := xhttp.MustNewAddressGroup( //AddressGroup͸͋ͱ΄Ͳ঺հ͠·͢ “dns: //txt:your-domain.com:8080”, xhttp.WithLoadBalancer(xhttp.RoundRobin),

    ) reqP, _ := xhttp.NewRequestX("POST", addrGroup, "/", nil) ϥΠϒϥϦଆͰ schemeύʔεͯ͠Resolver ΛΞαΠϯ͢Δ
40. 40.

    Engineering • Load Balancing is impossible • https://www.youtube.com/watch?v=kpvbOzHUakA • Round

    Robin ͸ Upstreamϗετͷঢ়ଶʢLatency, Φʔόϩʔυঢ় گʣͳͲΛҙࣝͰ͖ͣɺͣͬͱಉ͡ѱ͍ঢ়ଶͷϗετʹಉ͡ϦΫΤε τ਺Λ౤͛Δɹ • P2C ΞϧΰϦζϜ • Power of 2 Random Choice • Upstreamͷঢ়ଶΛ఻ୡඞཁ͋Δ • https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf ༨ஊ ϩʔυόϥγϯάΞϧΰϦζϜ͸೉͍͠
41. 41.

    Engineering ճ෮ੑͷ࿩

42. 42.

    Engineering Upstream 2 die xhttp client Upstream 1 αʔό͸͍ͭͰ΋ࢮ͵Մೳੑ΋ Upstream

    3
43. 43.

    Engineering Upstream 2 die xhttp client Upstream 1 die (શʣαʔό͸͍ͭͰ΋ࢮ͵Մೳੑ΋

    Upstream 3 die
44. 44.

    Engineering • ׬શʹࢮ͵͡Όͳͯ͘ɺ(GCͳͲͷཧ༝ͰʣLatency͕௓Ͷ্͕Δ • ద੾ͳσϑΥϧτλΠϜΞ΢τ͕ඞཁ • ଈ࣌తͳϦΫΤετࣦഊ (500ΤϥʔͳͲʣ • ద੾ͳϦτϥΠ

    • ܧଓతͳϦΫΤετࣦഊ (ωοτϫʔΫࣦഊͳͲʣ • Ͱ͖Δ͚ͩૣ͍ஈ֊ͰϦΫΤετΛυϩοϓ͢Δ • CircuitBreakerύʔλϯɿFail Fast Error αʔό͕ࢮ͵ͱ͖Ͳ͏͠·͢ʁ
45. 45.

    Engineering • net/http͸σϑΥϧτλΠϜΞ΢τ͋Γ·ͤΜʂ • ແݶʹϦΫΤετ଴ͭ • xhttpͰ͸σϑΥϧτͰ30sઃఆՄೳɺ֎͔Β౉͢ͷ΋Մೳ λΠϜΞ΢τʹ͍ͭͯ func NewClient(config

    *ClientConfig) *Client { client.client = &http.Client{ Timeout: config.RequestTimeout, Transport: newRoundTripper(config.RoundTripper, config.MetricRegistry), }… }
46. 46.

    Engineering Dial TLS Handshake Request Resp Headers Resp Body Idle

    Client.Do http.Client.Timeout net.Dialer.Timeout http.Transport.TLSHandshake.Timeout http.Transport.ResponseHeader.Timeout http.Transport.IdleConn.Timeout https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/ ৔߹ʹΑΓࡉ͔͘ίϯτϩʔϧՄೳ
47. 47.

    Engineering • Լखʹretry͢Δ͜ͱʹࣾ಺αʔϏεΛDDOS͢Δ͜ͱ • ඞͣ Backoff ( Exponential )͢Δ͜ͱ •

    Retry͸ϦΫΤετείʔϓͷOption (ClientείʔϓͰ͸͋Γ·ͤΜʣ RETRYʹ͍ͭͯ res, err = client.DoX(reqG, xhttp.WithRetry(&xhttp.RetryOptions{ MaxAttempts: 2, RetryOn: func(response *http.Response) bool { return response.StatusCode != 200 }, TimeoutPerAttempt: time.Second, }))
48. 48.

    Engineering • ૣ͍ஈ֊Ͱυϩοϓ͢Δख๏ͷҰͭͱͯ͠͸ CircuitBreakerύλʔϯɹ • https://engineering.linecorp.com/ja/blog/circuit-breakers-for- distributed-services/ • ϗετͷঢ়ଶΛʮࣦഊ͢ΔϦΫΤετͷ਺ʯͰ൑அ •

    ʮࣦഊʯఆٛࣗମ͸Ϣʔβʹ೚ͤΔ • ঢ়ଶʹΑΓʮϦΫΤετՄೳʯͱʮϦΫΤετෆՄೳʯͰ෼͚Δ • ϦΫΤετෆՄೳͷ৔߹୯७ʹ FailFastErr ฦ͢ FAIL FASTରࡦʹ͍ͭͯ
49. 49.

    Engineering • ׂͱύϥϝʔλ͕ଟ͍ͷͰɺΞϧΰϦζϜࣗମΛཧղ͢Δ্ઃఆ͢Δ ඞཁ͕͋Δ CIRCUIT BREAKER type CircuitBreakerConfig struct {

    noop bool FailureRateThreshold float64 MinimumRequestThreshold int64 TrialRequestInterval time.Duration CircuitOpenWindow time.Duration CounterSlidingWindow time.Duration CounterUpdateInterval time.Duration IsFailed FailedJudger KeyBuilder KeyBuilder } client := xhttp.NewClient(&xhttp.ClientConfig{ CircuitBreakerConfig: &xhttp.CircuitBreakerConfig{}, })
50. 50.

    Engineering • > ϗετͷঢ়ଶΛʮࣦഊ͢ΔϦΫΤετͷ਺ʯͰ൑அ • Ͳ͏΍ͬͯૣ͘਺͑Δʁ ༨ஊɹHIGH PERFORMANCE CIRCUITBREAKERͷͨΊʹ var

    c int64 c ++ var c atomic.Int64 c.Add(1) • Atomic࢖ͬͯ΋contentionมΘΒͳ͍ʢಉ͡ม਺ʹࢀর͢Δ͜ͱʣ • https://github.com/linxGnu/go-adder • Java LongAdderͷΞΠσΟΞʢatomicΛsharding) • Ϛϧνϧʔνϯ؀ڥͰatomic.Int64ΑΓ2~4ഒૣ͍
51. 51.

    Engineering METRICS/TRACINGͷ࿩

52. 52.

    Engineering • ObservabilityνʔϜͳͷͰͦͷॏཁੑΛ͘͢͝ཧղ͍ͯ͠Δ • ౴͑ͳ͍ͱ͍͚ͳ͍ͷ͸ • ϦΫΤετLatencyߴ͍Ͱ͔͢ʁ • Latencyߴ͍࣌͸ͳͥʁ OBSERVABILITYͷॏཁੑ

    • Latencyߴ͍͔Ͳ͏͔͸MetricsͰ౴͑Δ • Metrics Instrumentation ͸ prometheus client ࢖͏ • ͳͥʁΛ౴͑Δͷ͸ Metrics + TracingͰ౴͑Δ • Tracing Instrumentation ͸ zipkin-go
53. 53.

    Engineering ROUNDTRIPPERར༻ͯ͠INSTRUMENT type ClientConfig struct { MetricRegistry *prometheus.Registry Tracer *zipkin.Tracer

    …… } func newRoundTripper(originRt http.RoundTripper, mr *prometheus.Registry, tc *zipkin.Tracer) *roundtripper { … .. } client.client = &http.Client{ Timeout: config.RequestTimeout, Transport: newRoundTripper(config.RoundTripper, config.MetricRegistry, config.Tracer), }
54. 54.

    Engineering • ΧελϚΠζ͢ΔͨΊͷϑοΫ • ϦΫΤετૹΔॲཧΛΧελϚΠζɿ RoundTripperΛ౉͢ • ίωΫγϣϯΛ࡞ΔॲཧΛΧελϚΠζɿDialContextΛ౉͢ ༨ஊɹNET/HTTP͸୔ࢁ ϑοΫఏڙͯ͘͠ΕͯΔ

    • ϝτϦΫεΛऔಘ͢ΔͨΊͷϑοΫ • http/ClientTrace͸े෼ɹ • https://blog.golang.org/http-tracing
55. 55.

    Engineering ࠷ޙ

56. 56.

    Engineering • Ͱ͖Ε͹ࣾ಺ɺ͋Δ͍͸νʔϜ಺ڞ௨ͷϥΠϒϥϦἧͬͨ΄͏͕ྑ͍ • http client, log package, metrics instrument…

    • ϚΠΫϩαʔϏε͸೉͍͠ɺͰ͖Ε͹ආ͚ͨ΄͏͕ྑ͍ • ͍ۙ͏ͪʹxhttpΛOSS͢Δ • go޷͖Ͱɺେن໛γεςϜΛ࡞Γ͍ͨਓ͸͝࿈བྷ଴͓ͪͯ͠Γ·͢
57. 57.

    Engineering ͝੩ௌ͋Γ͕ͱ͏͍͟͝·͢