Slide 1

Slide 1 text

1 Development and Infrastructure for Microservice Architecture PyConJP2016 Yosuke Suzuki Nikkei Inc.

Slide 2

Slide 2 text

About me Engineer at Nikkei Inc. 2 Yosuke Suzuki @yosukep Python, Golang, JavaScript (Worked as an editor and staff writer also)

Slide 3

Slide 3 text

Silver sponsor! 3

Slide 4

Slide 4 text

About Nikkei.com 4 ● Launched on Mar 2010 ● More than 480,000 paid users ● 300,000,000 pv / month

Slide 5

Slide 5 text

Dev team in Nikkei.com 5 ● API ● Mobile App ● Front end ● Infra ● Bigdata Hiring inhouse engineers, migrating to inhouse development

Slide 6

Slide 6 text

Collaboration with FinancialTimes 6 ● Acquired on Dec 2015 ● Share development technique ● English!! Some team use English for internal communication

Slide 7

Slide 7 text

Main Topic 7

Slide 8

Slide 8 text

Definition of Microservices 8 ● Small in size✔ ● Autnomously developed✔ ● Independently deployable✔ ● Built and released with automated processes✔ ● Decentralized O’Reilly: Microservice Architecture

Slide 9

Slide 9 text

API Gateway Architecture 9 Cache/router (Fastly) UI service UI service Internet API gateway Search Content Auth DNS (Internal / Route53) API service API service Image API gateway is not decentralized?

Slide 10

Slide 10 text

Gateway pattern 10 ● Authentication and Authorization ○ Front-end layer sometime does authorization ● Some services like image delivery are directly served ● Wrapper for old APIs

Slide 11

Slide 11 text

Small in size✔ 11 ● Django based python application ○ Each application has a few folders ○ But API gateway has many folders ● NodeJS application adopts express as framework ○ 200 - 300 lines of JavaScript

Slide 12

Slide 12 text

Autonomously developed✔ 12 ● Do not share resources like DB ○ Use S3 to share data ● Use web APIs to exchange data between services ● statelss ● generate a service from template

Slide 13

Slide 13 text

Generate a service from template 13

Slide 14

Slide 14 text

Template includes 14 ● requirements.txt ● tox.ini ○ Test settings and format check by Flake8 ● Django application ● Dockerfile ● ElasticBeanstalk config file ● CircleCI config file

Slide 15

Slide 15 text

Generate by cookiecutter 15 pip install cookiecutter cookiecutter gh:Nikkei/django-project-template Interactive command line ● repo_name: GitHub repo name ● project_name: Django project name ● subdomin: domain name ● short_description: description

Slide 16

Slide 16 text

Follow template updates 16 pip install git+https://github.com/hirokiky/cookiepatcher cookiepatcher ● Do not make service legacy ● KY made cookiepatcher ● cookiepatcher.json manages diffs

Slide 17

Slide 17 text

Private PyPI 17 ● Internal common codes ○ Convert article id ○ Replace CMS tags requirements.txt: --extra-index-url https://xxx.x/xxxxx/packages/simple/ nikkei-utils==0.6

Slide 18

Slide 18 text

Private PyPI (2) 18 1. python setup.py sdist 2. Copy generated file like xx.0.1.tar.gz to specified directory 3. By pushing github repo for packages, CircleCI uploads packages automatically

Slide 19

Slide 19 text

Private PyPI (3) 19 from nikkei_utils.kiji import extract_snippet from nikkei_utils.region import region_info def hoge(): snippet = extract_snippet(article.body)

Slide 20

Slide 20 text

NodeJS app uses private npm 20 package.json: "dependencies": { "@nikkei/nikkei-ui": "^6.2.0", "@nikkei/rnikkei-express": "^2.32.2", "denodeify": "^1.2.1", "dom-delegate": "^2.0.3", "express-validator": "^2.20.8", "open-graph-scraper": "^2.2.2" } Get private npm packages from private npm server

Slide 21

Slide 21 text

Local development environment 21 ● It is easy for API development ○ python manage.py runserver ○ Not need to use Docker ● Front-end development is not easy ○ Routing some services to local computer

Slide 22

Slide 22 text

Django Rest Framework 22 ● Django Rest Framework ● Django Rest Framework Swagger Productive framework for API development

Slide 23

Slide 23 text

Rest Framework(2) 23 class FeedAnArticleView(APIView): http_method_names = ['get'] permission_classes = (IsAuthenticated,) def get(self, request, kiji_id_raw): """ # 検索エンジン用フィード --- parameters: - name: kiji_id paramType: path description: 対象の記事ID """ article = Article.objects.get(kiji_id_raw=kiji_id_raw, status=Article.POST) return Response(data={'article': feed_article_mapper(article)})

Slide 24

Slide 24 text

Independently deployable✔ 24 ● Completely different instances ● ElasticBeanstalk+Docker ● Easy scale out

Slide 25

Slide 25 text

Released with automated process✔ 25 Master branch -> development env deployment/production -> production env

Slide 26

Slide 26 text

Issues of ElasticBeanstalk command 26 ● Cannot use different config file(Dockerrun.aws.json) ● Blue/Green is not supported

Slide 27

Slide 27 text

ebi command can do that! 27 pip install ebi ebi deploy ● KY made it ● Use different Dockerrun.aws.json for each env ● Blue/Green deploy support

Slide 28

Slide 28 text

CircleCI.yml 28 - sudo pip install --ignore-installed awsebcli - sudo pip install -U ebi - aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID - aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY - aws configure set default.region ap-northeast-1 - aws configure set default.output json - ebi bgdeploy app-name env-name-blue env-name-green cname_prefix --profile default --noswap --region ap-northeast-1 --dockerrun Dockerrun.aws.prod.json - sleep 10 - sudo pip install -r e2etests/requirements.txt - py.test e2etests/tests.py - eb swap env-name-blue --profile default --destination_name env-name-green

Slide 29

Slide 29 text

Batch 29 Docker Container Docker Container Batch process by Rundeck. Use same Docker Image both for Rundeck and ElasticBeanstalk Spot instance Docker Container

Slide 30

Slide 30 text

Rundeck 30 Run Docker container by scheduled task

Slide 31

Slide 31 text

Monitoring, logging 31 ● Fluentd in another container aggregate logs ○ github.com/bungoume/log-sender ● Send error logs to Sentry ● Send metrics data to New Relic ● CloudWatch

Slide 32

Slide 32 text

Log aggregation 32 Each service Docker container Docker fluentd container Log aggregation

Slide 33

Slide 33 text

Incident alert 33 Send notification to Slack and call by PagerDuty when emegency situation happens

Slide 34

Slide 34 text

Generic infrastructure 34 ● Django/Python application ● NodeJS(express) application ● WordPress ● Rundeck ● Jenkins ● Kibana ● Nginx+plugin

Slide 35

Slide 35 text

Easy to migrate 35 NodeJS application of FinancialTimes runs on Heroku Same application runs on ElasticBeanstalk without any modification

Slide 36

Slide 36 text

Current issues 36 ● Scaling out is not so ideal ○ If spike more than expectation happens, scaling out is not fast enough ○ It is better to use scheduled scaling out ● Continuas update of libraries ○ Django version 1.7 -> 1.10 ○ Periodically check and update of requirements.txt is needed ● AWS is not perfect, shipped with bugs sometime.

Slide 37

Slide 37 text

Thanks 37

Slide 38

Slide 38 text

Thank you! 38

Slide 39

Slide 39 text

We're hiring! s.nikkei.com/saiyo Python Engineer Machine Learning Engineer Front-end Engineer Smartphone App Engineer Designer 39