Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Development and Infrastructure for Microservice Architecture

Yosuke
September 21, 2016

Development and Infrastructure for Microservice Architecture

Yosuke

September 21, 2016
Tweet

More Decks by Yosuke

Other Decks in Technology

Transcript

  1. 1
    Development and Infrastructure for
    Microservice Architecture
    PyConJP2016
    Yosuke Suzuki
    Nikkei Inc.

    View Slide

  2. About me
    Engineer at Nikkei Inc.
    2
    Yosuke Suzuki @yosukep
    Python, Golang, JavaScript
    (Worked as an editor and staff writer also)

    View Slide

  3. Silver sponsor!
    3

    View Slide

  4. About Nikkei.com
    4
    ● Launched on Mar 2010
    ● More than 480,000 paid
    users
    ● 300,000,000 pv / month

    View Slide

  5. Dev team in Nikkei.com
    5
    ● API
    ● Mobile App
    ● Front end
    ● Infra
    ● Bigdata
    Hiring inhouse engineers, migrating to inhouse development

    View Slide

  6. Collaboration with FinancialTimes
    6
    ● Acquired on Dec
    2015
    ● Share development
    technique
    ● English!!
    Some team use English for internal communication

    View Slide

  7. Main Topic
    7

    View Slide

  8. Definition of Microservices
    8
    ● Small in size✔
    ● Autnomously developed✔
    ● Independently deployable✔
    ● Built and released with automated
    processes✔
    ● Decentralized
    O’Reilly: Microservice Architecture

    View Slide

  9. API Gateway Architecture
    9
    Cache/router (Fastly)
    UI service UI service
    Internet
    API gateway
    Search Content Auth
    DNS (Internal / Route53)
    API service API service
    Image
    API gateway is not decentralized?

    View Slide

  10. Gateway pattern
    10
    ● Authentication and Authorization
    ○ Front-end layer sometime does
    authorization
    ● Some services like image delivery are
    directly served
    ● Wrapper for old APIs

    View Slide

  11. Small in size✔
    11
    ● Django based python application
    ○ Each application has a few folders
    ○ But API gateway has many folders
    ● NodeJS application adopts express as
    framework
    ○ 200 - 300 lines of JavaScript

    View Slide

  12. Autonomously developed✔
    12
    ● Do not share resources like DB
    ○ Use S3 to share data
    ● Use web APIs to exchange data
    between services
    ● statelss
    ● generate a service from template

    View Slide

  13. Generate a service from template
    13

    View Slide

  14. Template includes
    14
    ● requirements.txt
    ● tox.ini
    ○ Test settings and format check by Flake8
    ● Django application
    ● Dockerfile
    ● ElasticBeanstalk config file
    ● CircleCI config file

    View Slide

  15. Generate by cookiecutter
    15
    pip install cookiecutter
    cookiecutter gh:Nikkei/django-project-template
    Interactive command line
    ● repo_name: GitHub repo name
    ● project_name: Django project name
    ● subdomin: domain name
    ● short_description: description

    View Slide

  16. Follow template updates
    16
    pip install git+https://github.com/hirokiky/cookiepatcher
    cookiepatcher
    ● Do not make service legacy
    ● KY made cookiepatcher
    ● cookiepatcher.json manages diffs

    View Slide

  17. Private PyPI
    17
    ● Internal common codes
    ○ Convert article id
    ○ Replace CMS tags
    requirements.txt:
    --extra-index-url https://xxx.x/xxxxx/packages/simple/
    nikkei-utils==0.6

    View Slide

  18. Private PyPI (2)
    18
    1. python setup.py sdist
    2. Copy generated file like xx.0.1.tar.gz to specified
    directory
    3. By pushing github repo for packages, CircleCI
    uploads packages automatically

    View Slide

  19. Private PyPI (3)
    19
    from nikkei_utils.kiji import extract_snippet
    from nikkei_utils.region import region_info
    def hoge():
    snippet = extract_snippet(article.body)

    View Slide

  20. NodeJS app uses private npm
    20
    package.json:
    "dependencies": {
    "@nikkei/nikkei-ui": "^6.2.0",
    "@nikkei/rnikkei-express": "^2.32.2",
    "denodeify": "^1.2.1",
    "dom-delegate": "^2.0.3",
    "express-validator": "^2.20.8",
    "open-graph-scraper": "^2.2.2"
    }
    Get private npm packages from private npm server

    View Slide

  21. Local development environment
    21
    ● It is easy for API development
    ○ python manage.py runserver
    ○ Not need to use Docker
    ● Front-end development is not easy
    ○ Routing some services to local
    computer

    View Slide

  22. Django Rest Framework
    22
    ● Django Rest Framework
    ● Django Rest Framework Swagger
    Productive framework for API development

    View Slide

  23. Rest Framework(2)
    23
    class FeedAnArticleView(APIView):
    http_method_names = ['get']
    permission_classes = (IsAuthenticated,)
    def get(self, request, kiji_id_raw):
    """ # 検索エンジン用フィード
    ---
    parameters:
    - name: kiji_id
    paramType: path
    description: 対象の記事ID
    """
    article = Article.objects.get(kiji_id_raw=kiji_id_raw, status=Article.POST)
    return Response(data={'article': feed_article_mapper(article)})

    View Slide

  24. Independently deployable✔
    24
    ● Completely different
    instances
    ● ElasticBeanstalk+Docker
    ● Easy scale out

    View Slide

  25. Released with automated process✔
    25
    Master branch -> development env
    deployment/production -> production env

    View Slide

  26. Issues of ElasticBeanstalk command
    26
    ● Cannot use different config
    file(Dockerrun.aws.json)
    ● Blue/Green is not supported

    View Slide

  27. ebi command can do that!
    27
    pip install ebi
    ebi deploy
    ● KY made it
    ● Use different Dockerrun.aws.json for each
    env
    ● Blue/Green deploy support

    View Slide

  28. CircleCI.yml
    28
    - sudo pip install --ignore-installed awsebcli
    - sudo pip install -U ebi
    - aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID
    - aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY
    - aws configure set default.region ap-northeast-1
    - aws configure set default.output json
    - ebi bgdeploy app-name env-name-blue env-name-green cname_prefix --profile
    default --noswap --region ap-northeast-1 --dockerrun Dockerrun.aws.prod.json
    - sleep 10
    - sudo pip install -r e2etests/requirements.txt
    - py.test e2etests/tests.py
    - eb swap env-name-blue --profile default --destination_name env-name-green

    View Slide

  29. Batch
    29
    Docker
    Container
    Docker
    Container
    Batch process by
    Rundeck.
    Use same Docker Image
    both for Rundeck and
    ElasticBeanstalk
    Spot instance
    Docker
    Container

    View Slide

  30. Rundeck
    30
    Run Docker container by scheduled task

    View Slide

  31. Monitoring, logging
    31
    ● Fluentd in another container aggregate
    logs
    ○ github.com/bungoume/log-sender
    ● Send error logs to Sentry
    ● Send metrics data to New Relic
    ● CloudWatch

    View Slide

  32. Log aggregation
    32
    Each service
    Docker
    container
    Docker
    fluentd container
    Log aggregation

    View Slide

  33. Incident alert
    33
    Send notification to Slack and call by PagerDuty when emegency situation
    happens

    View Slide

  34. Generic infrastructure
    34
    ● Django/Python application
    ● NodeJS(express) application
    ● WordPress
    ● Rundeck
    ● Jenkins
    ● Kibana
    ● Nginx+plugin

    View Slide

  35. Easy to migrate
    35
    NodeJS application of FinancialTimes runs on Heroku
    Same application runs on ElasticBeanstalk without any modification

    View Slide

  36. Current issues
    36
    ● Scaling out is not so ideal
    ○ If spike more than expectation happens, scaling out
    is not fast enough
    ○ It is better to use scheduled scaling out
    ● Continuas update of libraries
    ○ Django version 1.7 -> 1.10
    ○ Periodically check and update of requirements.txt is
    needed
    ● AWS is not perfect, shipped with bugs sometime.

    View Slide

  37. Thanks
    37

    View Slide

  38. Thank you!
    38

    View Slide

  39. We're hiring!
    s.nikkei.com/saiyo
    Python Engineer
    Machine Learning Engineer
    Front-end Engineer
    Smartphone App Engineer
    Designer
    39

    View Slide