Japan Container Days v18.04 *TUJPͱڞʹ.JDSPTFSWJDFT ʹཱͪ޲͔͑ Aya Igarashi @Ladicle

Software Engineer @Ladicle AYA IGARASHI

Istio x KubernetesͰ͸ ͦΕΒͷ໰୊ΛͲ͏ղܾ ͢Δͷ͔? ࿩͍ͨ͜͠ͱ Istioʹೖ໳͢Δʹ͸? ෳࡶʹͳͬͨ Microservicesύλʔϯ ͷγεςϜ͕࣋ͭ໰୊఺ 1 2 3 ͭΒ͍…. Αͦ͞͏ ΍ͬͯΈΑ͏

ฉ͍ͯ΄͍͜͠ͱ ͯ͠ ͍Δ MicroservicesͰ։ൃ/ӡ༻Λ… IstioʹΑΔղܾํ๏ͱࠓ΍͍ͬͯΔ ํ๏Λൺֱ MicroservicesͰ͋Γ͕ͪͳ໰୊Λؾʹ ͱΊ͓ͯ͘ ͜Ε ͔Β ͯ͠ ͍Δ

2014 Microservices https://martinfowler.com/articles/microservices.html James Lewis & Martin Fowler

Monolithic vs. Microservices ༻్ɾ໨త͝ͱʹ খ͞ͳΞϓϦέʔγϣϯΛ࡞Δ Ұຕؠ(Monolitic)ͷΑ͏ʹ ҰͭͷΞϓϦέʔγϣϯΛ࡞Δ

Scale-out ͠΍͍͢ ૿΍͍ͨ͠ΞϓϦέʔγϣϯ͚ͩ Scale-outͰ͖Δ શͯΛҰؾʹ૿΍͢ or Scale-up ͷΈʹରԠ͍ͯ͠Δ

Update มߋ͠΍͍͢ Update Ұ෦ͷΞϓϦέʔγϣϯ͚ͩ ߋ৽͢Δ͜ͱ͕Ͱ͖Δ ModuleͷҰ෦ͷΈͷमਖ਼Ͱ΋ ΞϓϦέʔγϣϯશମ͕ର৅

େਓ਺Ͱ։ൃ͠΍͍͢ API API νʔϜؒ͸APIΛఆٛ͠ ಺෦ͷมߋ͸ಠཱ࣮ͯ͠ࢪͰ͖Δ ίϛϡχέʔγϣϯίετ͕ߴ͘ ։ൃ଎౓͕Լ͕Δ

NO SILVER BULLET

ෳࡶʹͳΓଓ͚ΔγεςϜ 2014 2016 201

ෳࡶͳγεςϜ͕΋ͨΒ͢໰୊఺ 1 2 3 4 γεςϜͷશମ૾͕೺ѲͰ͖ͳ͍ ΞϓϦέʔγϣϯؒͷ௨৴Λ੍ޚ͖͠Εͳ͍ 伴ͱূ໌ॻΛ؅ཧ͖͠Εͳ͍ ো֐࣌ʹԿ͕ى͜Δ͔෼͔Βͳ͍ Traffic Management Security Visualization Chaos Engineering

Istio͕͜ΕΒͷ ໰୊Λղܾ͠·͢ʂ

But,

NO SILVER BULLET

Istioͱ͸? “ෳࡶͳαʔϏεϝογϡͷ؅ཧ͕௥͍͔ͭͳ͍” ͱ͍͏໰୊Λղܾ͢ΔͨΊIBM, Google, Lyftʹ Αͬͯ։ൃ͞ΕͨOSSͰ͢ɻ https://github.com/istio/istio (latest: v0.7) ※ Istio͸ΪϦγϟޠͰൕͱ͍͏ҙຯ

ΞϓϦέʔγϣϯؒͰ ϝογϡঢ়ʹ ௨৴͢ΔωοτϫʔΫͷ͜ͱ Service Meshͱ͸ʁ Mesh

ͲͷΑ͏ʹServiceMeshΛ؅ཧ͍ͯ͠Δͷ͔? Pod App Envoy App Pod Pilot Mixer Auth Istio Controle Plane Data Plane Config data ֤ΞϓϦέʔγϣϯͷखલʹϓ ϩΩγαʔόΛஔ͖ɺ͜ͷϓϩ ΩγͷઃఆΛControle Plane͔ ΒAPIܦ༝Ͱ੍ޚ͢Δ Envoy

Envoyͱ͸? App Envoy C++11Ͱॻ͔Εͨ L4/L7 proxy LyftʹΑͬͯ։ൃ͞ΕͨOSS APIܦ༝ͷઃఆมߋ͕Մೳ ઃఆมߋ࣌ʹrestart͕ෆཁ HTTP/2 & GRPC ʹ΋ରԠ Proxy in/out-bound Traffic

Problem 1 ServiceMeshΛ੍ޚ͖͠Εͳ͍

໰୊1 ServiceMeshΛ੍ޚ͖͠Εͳ͍ Complex ServiceMesh ΞϓϦέʔγϣϯͷίʔυʹServiceMesh ͷઃఆ͕ຒΊࠐ·Ε͍ͯΔ৔߹ ↪︎ मਖ਼ຖʹ࠶σϓϩΠ͕ඞཁ ↪︎ ӡ༻/։ൃνʔϜͷ૒ํʹෛՙ

ղܾࡦ1 ServiceMeshͷઃఆΛ෼཭ apiVersion: networking.istio.io/v1alpha3 kind: VirtualService …. spec: # ϦΫΤετͷѼઌΛࢦఆ hosts: - myapp http: - match: - uri: # URI ʹ /v1/oldHome Λ͍࣋ͬͯΔ # ϦΫΤετʹϚον exact: /v1/oldHome redirect: # /v1/newHome ʹϦμΠϨΫτ uri: /v1/newHome authority: myapp.default.svc.cluster.local ServiceMeshͷઃఆ͕ΞϓϦέʔγϣ ϯίʔυʹґଘ͠ͳ͍ L7LBʹΑͬͯॊೈͳϧʔςΟϯά͕ Ͱ͖Δ ։ൃऀͱӡ༻ऀ͕෼ۀ͠΍͍͢ ● ΧφϦΞʔϦϦʔεͰ҆৺ͳupgrade ● ϦΫΤετΛϓϩμΫγϣϯͱ εςʔδϯά؀ڥ΁ͷϛϥʔ ● pathʹΑͬͯΞϓϦέʔγϣϯΛ෼͚Δ ● ܞଳͱPCͳͲΫϥΠΞϯτʹԠͨ͡ৼΓ෼͚

TrafficΛEnvoyʹྲྀ͢࢓૊Έ Init Istio App K8s 1. Deploy App initContainers: - name: istio-init image: docker.io/istio/proxy_init …. containers: - name: istio-proxy image: docker.io/istio/proxy_debug securityContext: runAsUser: 1337 args: - -p - “15001” …. - name: App image: docker.io/ladicle/myapp ...... 3.Change iptables rules Redirect In&Out bound Traffic UID: 1337 2. Create istio-init 4. Create App and istio-proxy proxy Istio

istio-initʹΑΔiptablesͷมߋ istio-proxy@productpage-v1-5f9b797dfc-n6rn4:/$ sudo iptables -t nat -n -L -v Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ISTIO_REDIRECT all -- * * 0.0.0.0/0 0.0.0.0/0 /* istio/install-istio-prerouting */ …. Chain OUTPUT (policy ACCEPT 1024 packets, 96005 bytes) pkts bytes target prot opt in out source destination 21 1260 ISTIO_OUTPUT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* istio/install-istio-output */ …. Chain ISTIO_OUTPUT (1 references) pkts bytes target prot opt in out source destination 0 0 ISTIO_REDIRECT all -- * lo 0.0.0.0/0 !127.0.0.1 /* istio/redirect-implicit-loopback */ 19 1140 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1337 /* istio/bypass-envoy */ 0 0 RETURN all -- * * 0.0.0.0/0 127.0.0.1 /* istio/bypass-explicit-loopback */ 2 120 ISTIO_REDIRECT all -- * * 0.0.0.0/0 0.0.0.0/0 /* istio/redirect-default-outbound */ Chain ISTIO_REDIRECT (3 references) pkts bytes target prot opt in out source destination 2 120 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* istio/redirect-to-envoy-port */ redir ports 15001 Redirect all Traffic to Envoy Except own & lo

K8s ServiceMeshͷઃఆΛม͑Δ App 1. Call Rules API proxy Istio 3. Call Envoy API Pilot 2. Detect changes of service-account 4. Update Envoy Config listeners: - address: socket_address: address: 0.0.0.0 port_value: 15004 filter_chains: - filters: …. http_filters: - config: default_destination_service: istio-pol service_configs: istio-policy.{{ .PodNamespace }}.svc disable_check_calls: true mixer_attributes: attributes:

Problem 2 伴ͱূ໌ॻΛ؅ཧ͖͠Εͳ͍

໰୊2 伴ͱূ໌ॻΛ؅ཧ͖͠Εͳ͍ γεςϜʹΑͬͯ͸ΞϓϦέʔγϣϯؒ௨ ৴ͷ҉߸Խ͸ඞਢ ΞϓϦέʔγϣϯ਺͕ଟ͘ͳΔͱɺ Difficult Key & Certificate Management ҰͭҰͭ伴΍ূ໌ॻͷੜ੒ɺ഑෍ɺߋ৽ɺ ແޮԽʹରԠ͢Δ͕࣌ؒͳ͘ͳΔ

ղܾࡦ2 伴ͱূ໌ॻͷ؅ཧΛࣗಈԽ 伴ͱূ໌ॻͷੜ੒, ഑෍, ߋ৽, ഇࢭʹରԠ͍ͯ͠Δ K8s 1. Deploy App 3. Generate self signed key & cert 7. Push Secure Naming 5. Deploy Pod with Secret 2. Detect service account change CA Istio 4. Store to Secret Pilot 6. Watch Secure Naming Foo proxy Istio (SD) Mutual TLS(૬ޓTLS)Λαϙʔτ SAN: “spiffe://…/foo” mTLS Bar proxy Istio SAN: “spiffe://…/bar” ref: https://spiffe.io/

Problem 3 γεςϜͷશମ૾͕೺ѲͰ͖ͳ͍ Observability, observability, observability!

໰୊3 γεςϜͷશମ૾͕೺ѲͰ͖ͳ͍ ? ΞϓϦέʔγϣϯͷ਺͕େ͖͘ͳΓɺ೔ʑ Ͳ͔͜Ͱमਖ਼͕Ճ͑ΒΕ͍ͯΔ৔߹ɺ ↪︎ ߏ੒ਤͷखಈυΩϡϝϯτ͸஗Ε͕ͪ ↪︎ σόοά΍ো֐ղੳ͢Δʹ΋ϩά͚ͩͰ͸ Ͳ͜·Ͱ੒ޭͨ͠ͷ͔೺Ѳͮ͠Β͍

໰୊3 ՄࢹԽπʔϧͱͷ࿈ܞ Metrics Traces Prometheus × Grafana Zipkin ServiceGraph Graphviz x Prometheus

IstioͱPrometheus͕Ͳ͏࿈ܞ͢Δͷ͔? # Rule to send metric instances to a Prometheus handler apiVersion: "config.istio.io/v1alpha2" kind: rule metadata: name: mongoprom namespace: default spec: match: context.protocol == "tcp" && destination.service == "mongodb.default.svc.cluster.local" actions: - handler: mongohandler.prometheus instances: - mongoreceivedbytes.metric - mongosentbytes.metric # Configuration for a Prometheus han apiVersion: "config.istio.io/v1alpha2" kind: prometheus metadata: name: mongohandler namespace: default spec: metrics: - name: mongo_sent_bytes # Prometh instance_name: mongosentbytes.me kind: COUNTER label_names: - source_service - source_version - destination_version - name: mongo_received_bytes # Prom instance_name: mongoreceivedbytes ௨৴ܥͷMetrics͸উखʹ औͬͯ͘ΕΔ σϑΥϧτͷMetrics͔Β ServiceGraphΛੜ੒Ͱ͖Δ

Distributed Tracingͱ͸? ϦΫΤετΛड͚औ͔ͬͯΒॲཧ͕׬ྃ͢Δ·ͰͷظؒΛSpanͱ͠ ͯDAGͰදݱ͠ɺ͜ΕΛ࣌ܥྻʹฒ΂ͨ΋ͷΛτϨʔε݁Ռͱͯ͠ දࣔ͢ΔɻશͯͷSpan͸໊લͱ։࢝࣌ؒɺܧଓ࣌ؒɺଞͷSpanͱ ͷRelationΛ͍࣋ͬͯΔɻ ΞϓϦέʔγϣϯؒͷϦΫΤετͷྲྀΕ΍࣌ؒΛՄࢹԽ͢Δπʔϧ Client LB Auth App DB 1, 8 2, 3 4, 7 5, 6 Time

IstioͱTraces͕Ͳ͏࿈ܞ͢Δͷ͔? ࣗಈతʹεύϯΛૹ৴Ͱ͖Δ τϨʔεશମΛؔ࿈͚ͮΔ͜ͱ͸Ͱ͖ͳ͍ ҎԼͷϔομΛࢠʹ఻ൖͤ͞Δ͜ͱͰؔ࿈͚ͮͰ͖Δ • x-request-id • x-b3-traceid • x-b3-spanid • x-b3-parentspanid • x-b3-sampled • x-b3-flags • x-ot-span-context def getForwardHeaders(request): headers = {} user_cookie = request.cookies.get("user") if user_cookie: headers['Cookie'] = 'user=' + user_cookie incoming_headers = [ 'x-request-id', 'x-b3-traceid', 'x-b3-spanid', 'x-b3-parentspanid', 'x-b3-sampled', 'x-b3-flags', 'x-ot-span-context' ] for ihdr in incoming_headers: val = request.headers.get(ihdr) if val is not None: headers[ihdr] = val

Mixer͕Backendͱ࿈ܞ͢Δ࢓૊Έ App proxy Istio K8s Update Mixer Config Mixer Adapters Report attributes Backends ΞϓϦέʔγϣϯίʔυͱBackendΛ஥հ MixterΛڬΉ͜ͱͰӡ༻ऀ੍͕ޚͰ͖Δ AdapterΛ௥Ճͯ͠ಠࣗͷBackendΛ૿΍ͤΔ

MixtureͷϦΫΤετͷྲྀΕ

Problem 4 ো֐࣌ʹԿ͕ى͜Δ͔Θ͔Βͳ͍

໰୊4 ো֐࣌ʹԿ͕ى͜Δ͔Θ͔Βͳ͍ γεςϜʹো֐͸͖ͭ΋ͷɻ ͔͠͠ɺো֐Λى͜͢ࢼݧ͸खಈͰ΍Δ ͱଓ͔ͣɺಠࣗʹ࣮૷͢Δͱଟ͘ͷίετ ͕͔͔Δɻ݁Ռɺޙճ͠ʹ͞Ε͕ͪ !? ↪︎ ো֐͕ى͜Δ·Ͱରࡦ͕ଧͨΕͳ͍

ղܾࡦ4 Fault InjectionͰো֐ʹඋ͑Δ ର৅ൣғͱো֐಺༰Λࢦఆ͢Δ͚ͩͰ खܰʹো֐Λ࣮ݧͰ͖Δ apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: reviews-route spec: …. fault: delay: percent: 10 fixedDelay: 5s …. fault: abort: percent: 10 httpStatus: 400 ࢦఆ͞Εׂͨ߹ͷϦΫΤετʹରͯ͠ Delay: ࢦఆ͞Ε͚ͨ࣌ؒͩ஗ΒͤΔ Abort: ֘౰ͷεςʔλείʔυΛฦ͢ Chaos http2Error grpcStatus΋࣮૷༧ఆ

ղܾࡦ4 Circuit BrakerͰো֐Λ࠷খݶʹ཈͑Δ apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: ratings-delay spec: … httpFault: delay: percent: 10 fixedDelay: 5s apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule … trafficPolicy: connectionPool: tcp: maxConnections: 100 http: http2MaxRequests: 1000 maxRequestPerConnection: 10 outlinerDetection: http: consecutiveErrors: 7 Interval: 5m baseEjectionTime: 10 ίʔυΛॻ͔ͣʹΤϥʔ͕ଓ͘ΞϓϦ έʔγϣϯ΁ͷϦΫΤετΛःஅ͢Δ Add x-envoy-overloaded Block Block Break

͓͞Β͍ 1 2 3 4 γεςϜͷશମ૾͕೺ѲͰ͖ͳ͍ ServiceMeshΛ੍ޚ͖͠Εͳ͍ 伴ͱূ໌ॻΛ؅ཧ͖͠Εͳ͍ ো֐࣌ʹԿ͕ى͜Δ͔Θ͔Βͳ͍ ՄࢹԽπʔϧͱ࿈ܞ & ࣗಈԽΛαϙʔτ ServiceMeshͷઃఆΛ෼཭͢Δ ηΩϡΞ௨৴ͷͨΊͷ伴Λ؅ཧ͖͠Εͳ͍ FIT & CircuitBrakerͰো֐ʹඋ͑Δ

ଞʹ΋ ৭ʑͰ͖·͢! Policy Enforcement Role Base Access Control Rate Limit Control Ingress Traffic Control Egress Traffic Integrate Bare Metal Logging Request Timeout Deploy to Eureka ɹɹɹɹɹMesos, CF

Istioʹೖ໳͢Δ

ೖ໳νϟʔτ C B D Start ͜ͷϓϨθϯ Λฉ͍ͯ֓ཁ Λ೺Ѳͨ͠ Istio؀ڥΛ ηοτΞοϓ ͨ͠ ڭ͑ͯཉ͍͠ ͜ͷൃදͰ ؾʹͳΔ Topic͕͋Δ A YES YES YES Istio ॳ৺ऀͩ IstioΛ׬શʹ ཧղ͍ͯ͠Δ NO YES NO NO ৸ͯͨͱ͜Ζ͸ SpeakerDeckΛcheck! https://speakerdeck.com/ladicle YES E QuickStartΛࢀߟʹ IstioΛΠϯετʔϧ https://istio.io/docs/setup/ kubernetes/quick-start.html GKEͷQuick Start͕Φεεϝ ֘౰͢ΔTasksΛࢼ͢ https://istio.io/docs/tasks/ NO ໢ཏతͳGuidsͷ νϡʔτϦΞϧΛࢼ͢ https://istio.io/docs/guides/

ࠔͬͨ࣌͸ʁ 1 2 3 4 ͓ࢼ͠தʹτϥϒͬͨ ࣭໰͕͋Δ 2Ͱ΋ղܾ͠ͳ͍࣭໰͕͋Δ τϥγϡʔΨΠυΛࢀর https://istio.io/help/troubleshooting.html όάΛݟ͚ͭͨ ॆ࣮ͷFAQΛࢀর https://istio.io/help/faq ҆ఆͷStack Overflow https://stackoverflow.com/questions/tagged/istio GitHubͷIssue΁ https://github.com/istio/issues/issues/

WE ARE HIRING!

THANK YOU For you time & we’ll see you soon @ladicle