1 Development and Infrastructure for Microservice Architecture PyConJP2016 Yosuke Suzuki Nikkei Inc.

About me Engineer at Nikkei Inc. 2 Yosuke Suzuki @yosukep Python, Golang, JavaScript (Worked as an editor and staff writer also)

Silver sponsor! 3

About Nikkei.com 4 ● Launched on Mar 2010 ● More than 480,000 paid users ● 300,000,000 pv / month

Dev team in Nikkei.com 5 ● API ● Mobile App ● Front end ● Infra ● Bigdata Hiring inhouse engineers, migrating to inhouse development

Collaboration with FinancialTimes 6 ● Acquired on Dec 2015 ● Share development technique ● English!! Some team use English for internal communication

Main Topic 7

Definition of Microservices 8 ● Small in size✔ ● Autnomously developed✔ ● Independently deployable✔ ● Built and released with automated processes✔ ● Decentralized O’Reilly: Microservice Architecture

API Gateway Architecture 9 Cache/router (Fastly) UI service UI service Internet API gateway Search Content Auth DNS (Internal / Route53) API service API service Image API gateway is not decentralized?

Gateway pattern 10 ● Authentication and Authorization ○ Front-end layer sometime does authorization ● Some services like image delivery are directly served ● Wrapper for old APIs

Small in size✔ 11 ● Django based python application ○ Each application has a few folders ○ But API gateway has many folders ● NodeJS application adopts express as framework ○ 200 - 300 lines of JavaScript

Autonomously developed✔ 12 ● Do not share resources like DB ○ Use S3 to share data ● Use web APIs to exchange data between services ● statelss ● generate a service from template

Generate a service from template 13

Template includes 14 ● requirements.txt ● tox.ini ○ Test settings and format check by Flake8 ● Django application ● Dockerfile ● ElasticBeanstalk config file ● CircleCI config file

Generate by cookiecutter 15 pip install cookiecutter cookiecutter gh:Nikkei/django-project-template Interactive command line ● repo_name: GitHub repo name ● project_name: Django project name ● subdomin: domain name ● short_description: description

Follow template updates 16 pip install git+https://github.com/hirokiky/cookiepatcher cookiepatcher ● Do not make service legacy ● KY made cookiepatcher ● cookiepatcher.json manages diffs

Private PyPI 17 ● Internal common codes ○ Convert article id ○ Replace CMS tags requirements.txt: --extra-index-url https://xxx.x/xxxxx/packages/simple/ nikkei-utils==0.6

Private PyPI (2) 18 1. python setup.py sdist 2. Copy generated file like xx.0.1.tar.gz to specified directory 3. By pushing github repo for packages, CircleCI uploads packages automatically

Private PyPI (3) 19 from nikkei_utils.kiji import extract_snippet from nikkei_utils.region import region_info def hoge(): snippet = extract_snippet(article.body)

NodeJS app uses private npm 20 package.json: "dependencies": { "@nikkei/nikkei-ui": "^6.2.0", "@nikkei/rnikkei-express": "^2.32.2", "denodeify": "^1.2.1", "dom-delegate": "^2.0.3", "express-validator": "^2.20.8", "open-graph-scraper": "^2.2.2" } Get private npm packages from private npm server

Local development environment 21 ● It is easy for API development ○ python manage.py runserver ○ Not need to use Docker ● Front-end development is not easy ○ Routing some services to local computer

Django Rest Framework 22 ● Django Rest Framework ● Django Rest Framework Swagger Productive framework for API development

Rest Framework(2) 23 class FeedAnArticleView(APIView): http_method_names = ['get'] permission_classes = (IsAuthenticated,) def get(self, request, kiji_id_raw): """ # 検索エンジン用フィード --- parameters: - name: kiji_id paramType: path description: 対象の記事ID """ article = Article.objects.get(kiji_id_raw=kiji_id_raw, status=Article.POST) return Response(data={'article': feed_article_mapper(article)})

Independently deployable✔ 24 ● Completely different instances ● ElasticBeanstalk+Docker ● Easy scale out

Released with automated process✔ 25 Master branch -> development env deployment/production -> production env

Issues of ElasticBeanstalk command 26 ● Cannot use different config file(Dockerrun.aws.json) ● Blue/Green is not supported

ebi command can do that! 27 pip install ebi ebi deploy ● KY made it ● Use different Dockerrun.aws.json for each env ● Blue/Green deploy support

CircleCI.yml 28 - sudo pip install --ignore-installed awsebcli - sudo pip install -U ebi - aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID - aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY - aws configure set default.region ap-northeast-1 - aws configure set default.output json - ebi bgdeploy app-name env-name-blue env-name-green cname_prefix --profile default --noswap --region ap-northeast-1 --dockerrun Dockerrun.aws.prod.json - sleep 10 - sudo pip install -r e2etests/requirements.txt - py.test e2etests/tests.py - eb swap env-name-blue --profile default --destination_name env-name-green

Batch 29 Docker Container Docker Container Batch process by Rundeck. Use same Docker Image both for Rundeck and ElasticBeanstalk Spot instance Docker Container

Rundeck 30 Run Docker container by scheduled task

Monitoring, logging 31 ● Fluentd in another container aggregate logs ○ github.com/bungoume/log-sender ● Send error logs to Sentry ● Send metrics data to New Relic ● CloudWatch

Log aggregation 32 Each service Docker container Docker fluentd container Log aggregation

Incident alert 33 Send notification to Slack and call by PagerDuty when emegency situation happens

Generic infrastructure 34 ● Django/Python application ● NodeJS(express) application ● WordPress ● Rundeck ● Jenkins ● Kibana ● Nginx+plugin

Easy to migrate 35 NodeJS application of FinancialTimes runs on Heroku Same application runs on ElasticBeanstalk without any modification

Current issues 36 ● Scaling out is not so ideal ○ If spike more than expectation happens, scaling out is not fast enough ○ It is better to use scheduled scaling out ● Continuas update of libraries ○ Django version 1.7 -> 1.10 ○ Periodically check and update of requirements.txt is needed ● AWS is not perfect, shipped with bugs sometime.

Thanks 37

Thank you! 38

We're hiring! s.nikkei.com/saiyo Python Engineer Machine Learning Engineer Front-end Engineer Smartphone App Engineer Designer 39