Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Development and Infrastructure for Microservice Architecture

3ceba985a8d8efb44ef9c7d74089efdb?s=47 Yosuke
September 21, 2016

Development and Infrastructure for Microservice Architecture

3ceba985a8d8efb44ef9c7d74089efdb?s=128

Yosuke

September 21, 2016
Tweet

More Decks by Yosuke

Other Decks in Technology

Transcript

  1. 1 Development and Infrastructure for Microservice Architecture PyConJP2016 Yosuke Suzuki

    Nikkei Inc.
  2. About me Engineer at Nikkei Inc. 2 Yosuke Suzuki @yosukep

    Python, Golang, JavaScript (Worked as an editor and staff writer also)
  3. Silver sponsor! 3

  4. About Nikkei.com 4 • Launched on Mar 2010 • More

    than 480,000 paid users • 300,000,000 pv / month
  5. Dev team in Nikkei.com 5 • API • Mobile App

    • Front end • Infra • Bigdata Hiring inhouse engineers, migrating to inhouse development
  6. Collaboration with FinancialTimes 6 • Acquired on Dec 2015 •

    Share development technique • English!! Some team use English for internal communication
  7. Main Topic 7

  8. Definition of Microservices 8 • Small in size✔ • Autnomously

    developed✔ • Independently deployable✔ • Built and released with automated processes✔ • Decentralized O’Reilly: Microservice Architecture
  9. API Gateway Architecture 9 Cache/router (Fastly) UI service UI service

    Internet API gateway Search Content Auth DNS (Internal / Route53) API service API service Image API gateway is not decentralized?
  10. Gateway pattern 10 • Authentication and Authorization ◦ Front-end layer

    sometime does authorization • Some services like image delivery are directly served • Wrapper for old APIs
  11. Small in size✔ 11 • Django based python application ◦

    Each application has a few folders ◦ But API gateway has many folders • NodeJS application adopts express as framework ◦ 200 - 300 lines of JavaScript
  12. Autonomously developed✔ 12 • Do not share resources like DB

    ◦ Use S3 to share data • Use web APIs to exchange data between services • statelss • generate a service from template
  13. Generate a service from template 13

  14. Template includes 14 • requirements.txt • tox.ini ◦ Test settings

    and format check by Flake8 • Django application • Dockerfile • ElasticBeanstalk config file • CircleCI config file
  15. Generate by cookiecutter 15 pip install cookiecutter cookiecutter gh:Nikkei/django-project-template Interactive

    command line • repo_name: GitHub repo name • project_name: Django project name • subdomin: domain name • short_description: description
  16. Follow template updates 16 pip install git+https://github.com/hirokiky/cookiepatcher cookiepatcher • Do

    not make service legacy • KY made cookiepatcher • cookiepatcher.json manages diffs
  17. Private PyPI 17 • Internal common codes ◦ Convert article

    id ◦ Replace CMS tags requirements.txt: --extra-index-url https://xxx.x/xxxxx/packages/simple/ nikkei-utils==0.6
  18. Private PyPI (2) 18 1. python setup.py sdist 2. Copy

    generated file like xx.0.1.tar.gz to specified directory 3. By pushing github repo for packages, CircleCI uploads packages automatically
  19. Private PyPI (3) 19 from nikkei_utils.kiji import extract_snippet from nikkei_utils.region

    import region_info def hoge(): snippet = extract_snippet(article.body)
  20. NodeJS app uses private npm 20 package.json: "dependencies": { "@nikkei/nikkei-ui":

    "^6.2.0", "@nikkei/rnikkei-express": "^2.32.2", "denodeify": "^1.2.1", "dom-delegate": "^2.0.3", "express-validator": "^2.20.8", "open-graph-scraper": "^2.2.2" } Get private npm packages from private npm server
  21. Local development environment 21 • It is easy for API

    development ◦ python manage.py runserver ◦ Not need to use Docker • Front-end development is not easy ◦ Routing some services to local computer
  22. Django Rest Framework 22 • Django Rest Framework • Django

    Rest Framework Swagger Productive framework for API development
  23. Rest Framework(2) 23 class FeedAnArticleView(APIView): http_method_names = ['get'] permission_classes =

    (IsAuthenticated,) def get(self, request, kiji_id_raw): """ # 検索エンジン用フィード --- parameters: - name: kiji_id paramType: path description: 対象の記事ID """ article = Article.objects.get(kiji_id_raw=kiji_id_raw, status=Article.POST) return Response(data={'article': feed_article_mapper(article)})
  24. Independently deployable✔ 24 • Completely different instances • ElasticBeanstalk+Docker •

    Easy scale out
  25. Released with automated process✔ 25 Master branch -> development env

    deployment/production -> production env
  26. Issues of ElasticBeanstalk command 26 • Cannot use different config

    file(Dockerrun.aws.json) • Blue/Green is not supported
  27. ebi command can do that! 27 pip install ebi ebi

    deploy <app_name> <env_name> • KY made it • Use different Dockerrun.aws.json for each env • Blue/Green deploy support
  28. CircleCI.yml 28 - sudo pip install --ignore-installed awsebcli - sudo

    pip install -U ebi - aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID - aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY - aws configure set default.region ap-northeast-1 - aws configure set default.output json - ebi bgdeploy app-name env-name-blue env-name-green cname_prefix --profile default --noswap --region ap-northeast-1 --dockerrun Dockerrun.aws.prod.json - sleep 10 - sudo pip install -r e2etests/requirements.txt - py.test e2etests/tests.py - eb swap env-name-blue --profile default --destination_name env-name-green
  29. Batch 29 Docker Container Docker Container Batch process by Rundeck.

    Use same Docker Image both for Rundeck and ElasticBeanstalk Spot instance Docker Container
  30. Rundeck 30 Run Docker container by scheduled task

  31. Monitoring, logging 31 • Fluentd in another container aggregate logs

    ◦ github.com/bungoume/log-sender • Send error logs to Sentry • Send metrics data to New Relic • CloudWatch
  32. Log aggregation 32 Each service Docker container Docker fluentd container

    Log aggregation
  33. Incident alert 33 Send notification to Slack and call by

    PagerDuty when emegency situation happens
  34. Generic infrastructure 34 • Django/Python application • NodeJS(express) application •

    WordPress • Rundeck • Jenkins • Kibana • Nginx+plugin
  35. Easy to migrate 35 NodeJS application of FinancialTimes runs on

    Heroku Same application runs on ElasticBeanstalk without any modification
  36. Current issues 36 • Scaling out is not so ideal

    ◦ If spike more than expectation happens, scaling out is not fast enough ◦ It is better to use scheduled scaling out • Continuas update of libraries ◦ Django version 1.7 -> 1.10 ◦ Periodically check and update of requirements.txt is needed • AWS is not perfect, shipped with bugs sometime.
  37. Thanks 37

  38. Thank you! 38

  39. We're hiring! s.nikkei.com/saiyo Python Engineer Machine Learning Engineer Front-end Engineer

    Smartphone App Engineer Designer 39