Development and Infrastructure for Microservice Architecture

Transcript

1. 1
   Development and Infrastructure for
   Microservice Architecture
   PyConJP2016
   Yosuke Suzuki
   Nikkei Inc.
   

   View Slide

2. About me
   Engineer at Nikkei Inc.
   2
   Yosuke Suzuki @yosukep
   Python, Golang, JavaScript
   (Worked as an editor and staff writer also)
   

   View Slide

3. Silver sponsor!
   3
   

   View Slide

4. About Nikkei.com
   4
   ● Launched on Mar 2010
   ● More than 480,000 paid
   users
   ● 300,000,000 pv / month
   

   View Slide

5. Dev team in Nikkei.com
   5
   ● API
   ● Mobile App
   ● Front end
   ● Infra
   ● Bigdata
   Hiring inhouse engineers, migrating to inhouse development
   

   View Slide

6. Collaboration with FinancialTimes
   6
   ● Acquired on Dec
   2015
   ● Share development
   technique
   ● English!!
   Some team use English for internal communication
   

   View Slide

7. Main Topic
   7
   

   View Slide

8. Definition of Microservices
   8
   ● Small in size✔
   ● Autnomously developed✔
   ● Independently deployable✔
   ● Built and released with automated
   processes✔
   ● Decentralized
   O’Reilly: Microservice Architecture
   

   View Slide

9. API Gateway Architecture
   9
   Cache/router (Fastly)
   UI service UI service
   Internet
   API gateway
   Search Content Auth
   DNS (Internal / Route53)
   API service API service
   Image
   API gateway is not decentralized?
   

   View Slide

10. Gateway pattern
    10
    ● Authentication and Authorization
    ○ Front-end layer sometime does
    authorization
    ● Some services like image delivery are
    directly served
    ● Wrapper for old APIs
    

    View Slide

11. Small in size✔
    11
    ● Django based python application
    ○ Each application has a few folders
    ○ But API gateway has many folders
    ● NodeJS application adopts express as
    framework
    ○ 200 - 300 lines of JavaScript
    

    View Slide

12. Autonomously developed✔
    12
    ● Do not share resources like DB
    ○ Use S3 to share data
    ● Use web APIs to exchange data
    between services
    ● statelss
    ● generate a service from template
    

    View Slide

13. Generate a service from template
    13
    

    View Slide

14. Template includes
    14
    ● requirements.txt
    ● tox.ini
    ○ Test settings and format check by Flake8
    ● Django application
    ● Dockerfile
    ● ElasticBeanstalk config file
    ● CircleCI config file
    

    View Slide

15. Generate by cookiecutter
    15
    pip install cookiecutter
    cookiecutter gh:Nikkei/django-project-template
    Interactive command line
    ● repo_name: GitHub repo name
    ● project_name: Django project name
    ● subdomin: domain name
    ● short_description: description
    

    View Slide

16. Follow template updates
    16
    pip install git+https://github.com/hirokiky/cookiepatcher
    cookiepatcher
    ● Do not make service legacy
    ● KY made cookiepatcher
    ● cookiepatcher.json manages diffs
    

    View Slide

17. Private PyPI
    17
    ● Internal common codes
    ○ Convert article id
    ○ Replace CMS tags
    requirements.txt:
    --extra-index-url https://xxx.x/xxxxx/packages/simple/
    nikkei-utils==0.6
    

    View Slide

18. Private PyPI (2)
    18
    1. python setup.py sdist
    2. Copy generated file like xx.0.1.tar.gz to specified
    directory
    3. By pushing github repo for packages, CircleCI
    uploads packages automatically
    

    View Slide

19. Private PyPI (3)
    19
    from nikkei_utils.kiji import extract_snippet
    from nikkei_utils.region import region_info
    def hoge():
    snippet = extract_snippet(article.body)
    

    View Slide

20. NodeJS app uses private npm
    20
    package.json:
    "dependencies": {
    "@nikkei/nikkei-ui": "^6.2.0",
    "@nikkei/rnikkei-express": "^2.32.2",
    "denodeify": "^1.2.1",
    "dom-delegate": "^2.0.3",
    "express-validator": "^2.20.8",
    "open-graph-scraper": "^2.2.2"
    }
    Get private npm packages from private npm server
    

    View Slide

21. Local development environment
    21
    ● It is easy for API development
    ○ python manage.py runserver
    ○ Not need to use Docker
    ● Front-end development is not easy
    ○ Routing some services to local
    computer
    

    View Slide

22. Django Rest Framework
    22
    ● Django Rest Framework
    ● Django Rest Framework Swagger
    Productive framework for API development
    

    View Slide

23. Rest Framework(2)
    23
    class FeedAnArticleView(APIView):
    http_method_names = ['get']
    permission_classes = (IsAuthenticated,)
    def get(self, request, kiji_id_raw):
    """ # 検索エンジン用フィード
    ---
    parameters:
    - name: kiji_id
    paramType: path
    description: 対象の記事ID
    """
    article = Article.objects.get(kiji_id_raw=kiji_id_raw, status=Article.POST)
    return Response(data={'article': feed_article_mapper(article)})
    

    View Slide

24. Independently deployable✔
    24
    ● Completely different
    instances
    ● ElasticBeanstalk+Docker
    ● Easy scale out
    

    View Slide

25. Released with automated process✔
    25
    Master branch -> development env
    deployment/production -> production env
    

    View Slide

26. Issues of ElasticBeanstalk command
    26
    ● Cannot use different config
    file(Dockerrun.aws.json)
    ● Blue/Green is not supported
    

    View Slide

27. ebi command can do that!
    27
    pip install ebi
    ebi deploy
    ● KY made it
    ● Use different Dockerrun.aws.json for each
    env
    ● Blue/Green deploy support
    

    View Slide

28. CircleCI.yml
    28
    - sudo pip install --ignore-installed awsebcli
    - sudo pip install -U ebi
    - aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID
    - aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY
    - aws configure set default.region ap-northeast-1
    - aws configure set default.output json
    - ebi bgdeploy app-name env-name-blue env-name-green cname_prefix --profile
    default --noswap --region ap-northeast-1 --dockerrun Dockerrun.aws.prod.json
    - sleep 10
    - sudo pip install -r e2etests/requirements.txt
    - py.test e2etests/tests.py
    - eb swap env-name-blue --profile default --destination_name env-name-green
    

    View Slide

29. Batch
    29
    Docker
    Container
    Docker
    Container
    Batch process by
    Rundeck.
    Use same Docker Image
    both for Rundeck and
    ElasticBeanstalk
    Spot instance
    Docker
    Container
    

    View Slide

30. Rundeck
    30
    Run Docker container by scheduled task
    

    View Slide

31. Monitoring, logging
    31
    ● Fluentd in another container aggregate
    logs
    ○ github.com/bungoume/log-sender
    ● Send error logs to Sentry
    ● Send metrics data to New Relic
    ● CloudWatch
    

    View Slide

32. Log aggregation
    32
    Each service
    Docker
    container
    Docker
    fluentd container
    Log aggregation
    

    View Slide

33. Incident alert
    33
    Send notification to Slack and call by PagerDuty when emegency situation
    happens
    

    View Slide

34. Generic infrastructure
    34
    ● Django/Python application
    ● NodeJS(express) application
    ● WordPress
    ● Rundeck
    ● Jenkins
    ● Kibana
    ● Nginx+plugin
    

    View Slide

35. Easy to migrate
    35
    NodeJS application of FinancialTimes runs on Heroku
    Same application runs on ElasticBeanstalk without any modification
    

    View Slide

36. Current issues
    36
    ● Scaling out is not so ideal
    ○ If spike more than expectation happens, scaling out
    is not fast enough
    ○ It is better to use scheduled scaling out
    ● Continuas update of libraries
    ○ Django version 1.7 -> 1.10
    ○ Periodically check and update of requirements.txt is
    needed
    ● AWS is not perfect, shipped with bugs sometime.
    

    View Slide

37. Thanks
    37
    

    View Slide

38. Thank you!
    38
    

    View Slide

39. We're hiring!
    s.nikkei.com/saiyo
    Python Engineer
    Machine Learning Engineer
    Front-end Engineer
    Smartphone App Engineer
    Designer
    39
    

    View Slide