JJUG_CCC_2019_Spring.pdf

 JJUG_CCC_2019_Spring.pdf

0d1076d3a26279d1a2fbc2d02d140210?s=128

川上徹

May 18, 2019
Tweet

Transcript

  1. 2.

    川上 徹(かわかみ とおる) • オイシックス・ラ・大地株式会社 Joined in October 2018 •

    システム本部 システム基盤部 基盤刷新セクション • Twitter: @kawakamitor0312 About me #jjug_ccc #ccc_g6 最近社内では・・・ CircleCIおじさん/API Managementおじさん/Ingressおじさん ついでにSpring Bootおじさん
  2. 11.

    Our EC Site On-Premise Oracle RAC ・
 ・
 • 全てを司る神Oracle

    <通称:Core DB> ◦ 何かとボトルネック • とってもモノリシック ◦ ちょっといじると爆発する(こともある) • とってもステートフル ◦ みんな大好きスティッキーセッション #jjug_ccc #ccc_g6
  3. 12.

    Our EC Site EC Site
 
 
 
 
 


    
 
 
 
 
 Original Framework
 ・
 ・
 • いわゆるECサイト • 顧客/受注の管理等を行うバックオフィスのシステム ◦ バッチ処理が載っていたりする • 他にもいろいろあるがこの2つがメインターゲット On-Premise Oracle RAC Back Office
 
 
 
 
 
 
 
 
 
 
 Struts2
 #jjug_ccc #ccc_g6
  4. 15.

    Migration Overview Existing EC Site ・
 ・
 CDN Azure API

    Management Azure Kubernetes Service Migrate Functions #jjug_ccc #ccc_g6
  5. 20.

    Function & Database Separation Existing EC Site Domain Domain Table

    Function Client Table Azure Micro Service Function Function Existing EC Site Domain Domain Table Table Azure Micro Service Function Client Function Client Function Client Existing EC Site Domain Azure Micro Service Function Client Function Client Domain Table Table #jjug_ccc #ccc_g6
  6. 24.

    Azure Batch Scalability Up Existing EC Site Batch Get Record

    Thread Thread Existing EC Site Producer Service Get Record Queue (or Http) Consumer Service Queue (or Http) #jjug_ccc #ccc_g6
  7. 25.

    Batch Scalability Up Existing EC Site Batch Step1 Step2 Step3

    Step4 Step5 Step6 Azure Batch Step1 Step2 Step3 Step4 Step5 Step6 #jjug_ccc #ccc_g6
  8. 28.

    Stateless Application Server On-Premise Oracle RAC ・
 ・
 • State

    = サーバインスタンス固有の状態 • 代表的な要素としては ◦ セッション ◦ ファイル/ストレージ ◦ 設定ファイル State State State #jjug_ccc #ccc_g6
  9. 31.

    Our Microservices is REST API • OpenAPI(Swagger) • Spring Boot

    • Azure Kubernetes Service • Azure API Management #jjug_ccc #ccc_g6
  10. 34.

    Swagger Codegen • Controller Interface(Spring MVC) • Client API for

    Other Microservices ◦ RestTemplate(Spring 5.x with Spring Boot 2.x) • Client API for Existing Application ◦ RestTemplate(Spring 4.x) ◦ OkHttp Swagger Specから複数のクライアントをGenerate&Publish • 怖くてバージョンアップできないアプリケーション • でもTypeSafeなクライアントは求めたい Generate Multiple Client Library #jjug_ccc #ccc_g6
  11. 35.

    他にも • Client API for BFF • Client API for

    Front End ◦ typescript-axios 何かと便利なSwagger Spec • 様々な言語に対応 • 効率的なAPI呼び出しを • やっぱりTypeSafeなクライアントを求めたい Generate Multiple Client Library #jjug_ccc #ccc_g6
  12. 39.

    ログ収集・集積・解析ツールは必須!! • Fluentd • Logstash • Apache Flume • Elasticsearch

    • Datadog • New Relic • Zipkin • Kibana • Grafana • Coralogix 弊社ではPapertrailを使用 Distributed Tracing #jjug_ccc #ccc_g6
  13. 40.

    Distributed Tracing Existing EC Site(VM) App Server Application Log File

    Azure Kubernetes Service Node Pod stdout Application Papertrail Service Log Aggregation Dashboard Alerts PagerDuty Alerts Papertrail Agent Papertrail Agent #jjug_ccc #ccc_g6
  14. 44.

    Logging with Trace Id / Span Id • 赤字:Trace Id

    • 青字:Span Id 2019-04-22 00:20:11.813 INFO [some-service,3164645b88b820a5,c1c1ce2556bc0286,false] 760 --- [Some-Task-5] j.c.o.o.logging.ExecutionLogger : Start AAA 2019-04-22 00:20:11.814 INFO [some-service,3164645b88b820a5,c1c1ce2556bc0286,false] 760 --- [Some-Task-5] j.c.o.o.logging.ExecutionLogger : Start BBB 2019-04-22 00:20:11.815 INFO [some-service,3164645b88b820a5,c1c1ce2556bc0286,false] 760 --- [Some-Task-5] j.c.o.o.logging.ExecutionLogger : Start CCC Distributed Tracing #jjug_ccc #ccc_g6
  15. 45.

    ClientでTrace Id / Span Idを払い出す • 既存ECサイトログとの紐付け • 自動生成クライアントに組み込むライブラリとして実装 Distributed

    Tracing Existing EC Site Domain Azure Micro Service Function Client Function Client Domain Table Table #jjug_ccc #ccc_g6
  16. 46.

    Interceptor for OkHttpClient public class TraceIdInterceptor implements Interceptor { private

    final Logger logger = LoggerFactory.getLogger(TraceIdInterceptor.class); private final Random random = new Random(); @Override public Response intercept(Chain chain) throws IOException { long traceId = random.nextLong(); long spanId = traceId; String traceIdString = TraceContext.newBuilder().traceId(traceId).spanId(spanId).build().traceIdString(); String sessionId = MDC.get("sessionId"); logger.info(String.format("Trace ID %s mapped for Session ID %s", traceIdString, sessionId)); Request newRequest = chain .request() .newBuilder() .addHeader("X-B3-TraceId", traceIdString) .addHeader("X-B3-SpanId", traceIdString) .build(); return chain.proceed(newRequest); } } Distributed Tracing #jjug_ccc #ccc_g6
  17. 47.

    ClientHttpRequestInterceptor for RestTemplate Factoryクラスを用意して設定済みのインスタンスを生成 public class TraceIdInterceptor implements ClientHttpRequestInterceptor {

    private final Logger logger = LoggerFactory.getLogger(TraceIdInterceptor.class); private final Random random = new Random(); @Override public ClientHttpResponse intercept( HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException { long traceId = random.nextLong(); long spanId = traceId; String traceIdString = TraceContext.newBuilder().traceId(traceId).spanId(spanId).build().traceIdString(); String sessionId = MDC.get("sessionId"); logger.info(String.format("Trace ID %s mapped for Session ID %s", traceIdString, sessionId)); request.getHeaders().add("X-B3-TraceId", traceIdString); request.getHeaders().add("X-B3-SpanId", traceIdString); return execution.execute(request, body); } } Distributed Tracing #jjug_ccc #ccc_g6
  18. 48.

    Server SideでRequestMappingを補足してロギングするAOP & Swaggerで生成したControllerのInterfaceについた アノテーションを補足 参考: https://qiita.com/NetPenguin/items/a59fc8d84314b39424cd AutoConfigurationでstarterプロジェクト化 @Aspect public

    class RequestMappingLoggingAdvice { @Around("@annotation(org.springframework.web.bind.annotation.RequestMapping)") public Object adviceForRequestMapping(ProceedingJoinPoint proceedingJoinPoint) throws Throwable { return ExecutionLogger.log(proceedingJoinPoint); } @Around("@annotation(org.springframework.web.bind.annotation.GetMapping)") public Object adviceForGetMapping(ProceedingJoinPoint proceedingJoinPoint) throws Throwable { return ExecutionLogger.log(proceedingJoinPoint); } // omit. } Distributed Tracing #jjug_ccc #ccc_g6
  19. 50.

    with @Async / @Scheduled • In Spring Cloud Sleuth, we

    instrument async-related components so that the tracing information is passed between threads. You can disable this behavior by setting the value of spring.sleuth.async.enabled to false. • In Spring Cloud Sleuth, we instrument scheduled method execution so that the tracing information is passed between threads. You can disable this behavior by setting the value of spring.sleuth.scheduled.enabled to false. Distributed Tracing https://cloud.spring.io/spring-cloud-sleuth/single/spring-cloud-sleuth.html#_async_annotated_methods #jjug_ccc #ccc_g6
  20. 51.

    @Configuration @EnableAutoConfiguration @EnableAsync @Role(BeanDefinition.ROLE_INFRASTRUCTURE) static class CustomExecutorConfig extends AsyncConfigurerSupport {

    @Autowired BeanFactory beanFactory; @Override public Executor getAsyncExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); // CUSTOMIZE HERE executor.setCorePoolSize(7); executor.setMaxPoolSize(42); executor.setQueueCapacity(11); executor.setThreadNamePrefix("MyExecutor-"); // DON'T FORGET TO INITIALIZE executor.initialize(); return new LazyTraceExecutor(this.beanFactory, executor); } } with Custom Executors Distributed Tracing https://cloud.spring.io/spring-cloud-sleuth/single/spring-cloud-sleuth.html#_customization_of_executors #jjug_ccc #ccc_g6
  21. 52.

    with TaskDecorator(Appendix) MDCを非同期スレッドに引き継ぎたい場合はこんなパターンも public class MdcTaskDecorator implements TaskDecorator { @Override

    public Runnable decorate(Runnable runnable) { Map<String, String> mdcMap = MDC.getCopyOfContextMap(); return () -> { try { if (mdcMap != null) { MDC.setContextMap(mdcMap); } runnable.run(); } finally { MDC.clear(); } }; } } Distributed Tracing #jjug_ccc #ccc_g6
  22. 61.

    Spring Cloud Config with Azure ハマったポイントを2点ほど Configuration File Aggregation Azure

    API Management Azure Kubernetes Service Spring Cloud Config Server Github Repository Configuration File Configuration File Configuration File Existing EC Site ・
 ・
 #jjug_ccc #ccc_g6
  23. 62.

    Spring Cloud Config Server Endpoint @RequestMapping("/{name}/{profile}/{label}/**") OpenAPI(Swagger)で書けない API Managementに定義できない @RestController

    @RequestMapping(method = RequestMethod.GET, path = "${spring.cloud.config.server.prefix:}") public class ResourceController { // omit. @RequestMapping("/{name}/{profile}/{label}/**") public String retrieve(@PathVariable String name, @PathVariable String profile, @PathVariable String label, HttpServletRequest request, @RequestParam(defaultValue = "true") boolean resolvePlaceholders) throws IOException { String path = getFilePath(request, name, profile, label); return retrieve(name, profile, label, path, resolvePlaceholders); } } Configuration File Aggregation #jjug_ccc #ccc_g6
  24. 64.

    Ingress Controller rewrite-targetがリライトするときにデコード(してた)! apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target:

    /$1 name: some-ingress namespace: some-namespace spec: rules: - host: some-domain.japaneast.cloudapp.azure.com http: paths: - backend: serviceName: some-service servicePort: 80 path: /apis/some-service/(.*$) Configuration File Aggregation #jjug_ccc #ccc_g6 ←①このdomainの ←②このpathにきたrequestを ←③このservice/portの ←④このpathにルーティング  ($1:②で取得したpath)
  25. 71.

    production development master feature development-ns1 production-ns1 development-ns2 production-ns2 kustomize build

    → kubectl apply kustomize build HOLD kubectl apply ns1 ns1 ns2 ns2 PR PR Switch Kubernetes Cluster Github + CircleCI + KustomizeによるGitops #jjug_ccc #ccc_g6
  26. 72.

    Switch Kubernetes Cluster kubernetes manifest リポジトリ構成 • ディレクトリ名 + ブランチ名で対象環境にデプロイ

    • 新クラスタを立ち上げる場合はディレクトリ + ブランチ追加 root ├── base │ ├── deployment.yaml │ ├── service.yaml │ ├── kustomization.yaml │ └── secret.yaml ├── dev-cluster-v1_namespace1 ├── prod-cluster-v1_namespace1 ├── dev-cluster-v2_namespace1 └── prod-cluster-v2_namespace1 #jjug_ccc #ccc_g6
  27. 74.

    API Managementによるbackend切り替え Switch Kubernetes Cluster Azure API Management Azure Kubernetes

    Service v1 Azure Kubernetes Service v2 API Revison 1(Current) Revison 2(Next) set-backend-service policy set-backend-service policy Managed Services domain v1 domain v2 #jjug_ccc #ccc_g6
  28. 75.

    まとめ • 複数クライアントライブラリ生成 ⇨アプリケーションのライブラリに合わせて • 分散トレーシング ⇨ログ集約必須・Trace Idをつけよう • 設定ファイル集約

    ⇨API Management・Ingressのハマりどころ • Kubernetesクラスタ移行 ⇨バージョンアップはクラスタ新規作成で  移行しやすい仕組みづくりを #jjug_ccc #ccc_g6
  29. 78.