Development and Infrastructure for Microservice Architecture

Transcript

1. 1 Development and Infrastructure for Microservice Architecture PyConJP2016 Yosuke Suzuki Nikkei Inc.
2. About me Engineer at Nikkei Inc. 2 Yosuke Suzuki @yosukep Python, Golang, JavaScript (Worked as an editor and staff writer also)
3. Silver sponsor! 3
4. About Nikkei.com 4 • Launched on Mar 2010 • More than 480,000 paid users • 300,000,000 pv / month
5. Dev team in Nikkei.com 5 • API • Mobile App • Front end • Infra • Bigdata Hiring inhouse engineers, migrating to inhouse development
6. Collaboration with FinancialTimes 6 • Acquired on Dec 2015 • Share development technique • English!! Some team use English for internal communication
7. Main Topic 7
8. Definition of Microservices 8 • Small in size✔ • Autnomously developed✔ • Independently deployable✔ • Built and released with automated processes✔ • Decentralized O’Reilly: Microservice Architecture
9. API Gateway Architecture 9 Cache/router (Fastly) UI service UI service Internet API gateway Search Content Auth DNS (Internal / Route53) API service API service Image API gateway is not decentralized?
10. Gateway pattern 10 • Authentication and Authorization ◦ Front-end layer sometime does authorization • Some services like image delivery are directly served • Wrapper for old APIs
11. Small in size✔ 11 • Django based python application ◦ Each application has a few folders ◦ But API gateway has many folders • NodeJS application adopts express as framework ◦ 200 - 300 lines of JavaScript
12. Autonomously developed✔ 12 • Do not share resources like DB ◦ Use S3 to share data • Use web APIs to exchange data between services • statelss • generate a service from template
13. Generate a service from template 13
14. Template includes 14 • requirements.txt • tox.ini ◦ Test settings and format check by Flake8 • Django application • Dockerfile • ElasticBeanstalk config file • CircleCI config file
15. Generate by cookiecutter 15 pip install cookiecutter cookiecutter gh:Nikkei/django-project-template Interactive command line • repo_name: GitHub repo name • project_name: Django project name • subdomin: domain name • short_description: description
16. Follow template updates 16 pip install git+https://github.com/hirokiky/cookiepatcher cookiepatcher • Do not make service legacy • KY made cookiepatcher • cookiepatcher.json manages diffs
17. Private PyPI 17 • Internal common codes ◦ Convert article id ◦ Replace CMS tags requirements.txt: --extra-index-url https://xxx.x/xxxxx/packages/simple/ nikkei-utils==0.6
18. Private PyPI (2) 18 1. python setup.py sdist 2. Copy generated file like xx.0.1.tar.gz to specified directory 3. By pushing github repo for packages, CircleCI uploads packages automatically
19. Private PyPI (3) 19 from nikkei_utils.kiji import extract_snippet from nikkei_utils.region import region_info def hoge(): snippet = extract_snippet(article.body)
20. NodeJS app uses private npm 20 package.json: "dependencies": { "@nikkei/nikkei-ui": "^6.2.0", "@nikkei/rnikkei-express": "^2.32.2", "denodeify": "^1.2.1", "dom-delegate": "^2.0.3", "express-validator": "^2.20.8", "open-graph-scraper": "^2.2.2" } Get private npm packages from private npm server
21. Local development environment 21 • It is easy for API development ◦ python manage.py runserver ◦ Not need to use Docker • Front-end development is not easy ◦ Routing some services to local computer
22. Django Rest Framework 22 • Django Rest Framework • Django Rest Framework Swagger Productive framework for API development
23. Rest Framework(2) 23 class FeedAnArticleView(APIView): http_method_names = ['get'] permission_classes = (IsAuthenticated,) def get(self, request, kiji_id_raw): """ # 検索エンジン用フィード --- parameters: - name: kiji_id paramType: path description: 対象の記事ID """ article = Article.objects.get(kiji_id_raw=kiji_id_raw, status=Article.POST) return Response(data={'article': feed_article_mapper(article)})
24. Independently deployable✔ 24 • Completely different instances • ElasticBeanstalk+Docker • Easy scale out
25. Released with automated process✔ 25 Master branch -> development env deployment/production -> production env
26. Issues of ElasticBeanstalk command 26 • Cannot use different config file(Dockerrun.aws.json) • Blue/Green is not supported
27. ebi command can do that! 27 pip install ebi ebi deploy <app_name> <env_name> • KY made it • Use different Dockerrun.aws.json for each env • Blue/Green deploy support
28. CircleCI.yml 28 - sudo pip install --ignore-installed awsebcli - sudo pip install -U ebi - aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID - aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY - aws configure set default.region ap-northeast-1 - aws configure set default.output json - ebi bgdeploy app-name env-name-blue env-name-green cname_prefix --profile default --noswap --region ap-northeast-1 --dockerrun Dockerrun.aws.prod.json - sleep 10 - sudo pip install -r e2etests/requirements.txt - py.test e2etests/tests.py - eb swap env-name-blue --profile default --destination_name env-name-green
29. Batch 29 Docker Container Docker Container Batch process by Rundeck. Use same Docker Image both for Rundeck and ElasticBeanstalk Spot instance Docker Container
30. Rundeck 30 Run Docker container by scheduled task
31. Monitoring, logging 31 • Fluentd in another container aggregate logs ◦ github.com/bungoume/log-sender • Send error logs to Sentry • Send metrics data to New Relic • CloudWatch
32. Log aggregation 32 Each service Docker container Docker fluentd container Log aggregation
33. Incident alert 33 Send notification to Slack and call by PagerDuty when emegency situation happens
34. Generic infrastructure 34 • Django/Python application • NodeJS(express) application • WordPress • Rundeck • Jenkins • Kibana • Nginx+plugin
35. Easy to migrate 35 NodeJS application of FinancialTimes runs on Heroku Same application runs on ElasticBeanstalk without any modification
36. Current issues 36 • Scaling out is not so ideal ◦ If spike more than expectation happens, scaling out is not fast enough ◦ It is better to use scheduled scaling out • Continuas update of libraries ◦ Django version 1.7 -> 1.10 ◦ Periodically check and update of requirements.txt is needed • AWS is not perfect, shipped with bugs sometime.
37. Thanks 37
38. Thank you! 38
39. We're hiring! s.nikkei.com/saiyo Python Engineer Machine Learning Engineer Front-end Engineer Smartphone App Engineer Designer 39