Quipper on Containers

2016/09/21 JAWS UG コンテナ支部 #6

Transcript

1. on Containers
   )
   

   View Slide

2. @hakobera
   DevOps Engineer
   

   View Slide

3. View Slide

4. Mexico City
   London
   Tokyo
   Manila
   Jakarta
   Sales team only
   Sales and dev team
   International Offices
   No one here (yet).
   

   View Slide

5. Products
   

   View Slide

6. Container != Docker
   

   View Slide

7. Docker is one of the container impls.
   Ubuntu LXD
   Microsoft®
   Windows® Server Container
   Hyper-V® Container
   

   View Slide

8. When do you use container
   on your system?
   

   View Slide

9. Now!
   

   View Slide

10. Why do you use container?
    

    View Slide

11. Portability
    and
    Beyond Your Cloud Limits
    

    View Slide

12. Docker on Elastic Beanstalk
    ● For short batch using worker tier
    ● Avoid darkness of .ebextension
    ● Avoid Amazon Linux
    ○ We want to use Ubuntu
    

    View Slide

13. Load Test on Elastic Container Service
    ● Easy to setup locust cluster
    ○ Consists of 2 Tasks
    ■ 1 master
    ■ Multiple slave
    ○ Manage instances using Spot Fleet
    ■ Cheap but scalable
    

    View Slide

14. nginx-build on CircleCI
    ● Easy to build custom nginx build
    ○ Bundle custom modules such as ngx_mruby
    ○ Patch for security issue
    ■ Update only openssl but not nginx
    

    View Slide

15. Ansible playbook test on CircleCI
    ● Run playbook to clean environment
    ● Run playboot test in parallel
    

    View Slide

16. Do you also use container
    for Web application?
    

    View Slide

17. Yes!
    But it’s not so easy
    

    View Slide

18. History of
    Quipper and Container
    

    View Slide

19. Quipper is
    Container Native Company
    

    View Slide

20. Quipper use container
    from beginning
    

    View Slide

21. Back to 2012
    First Product
    First Container is ...
    

    View Slide

22. View Slide

23. It’s too early
    

    View Slide

24. No source repository
    Only CTO only knows
    

    View Slide

25. From 2013
    Second Container Choice is ...
    

    View Slide

26. View Slide

27. Heroku is great!
    

    View Slide

28. Awesome features of Heroku
    ● Deploy via git
    ● Buildpack
    ● Multiple Dyno Size
    ● One-off dyno
    ● Review Apps
    

    View Slide

29. At April, 2015
    We are acquired by Recruit Co. Ltd
    

    View Slide

30. What problem do we have to resolve?
    ● Performance
    ○ Latency from Japan and USA (400ms/req)
    ○ Needed more powerful dyno
    ● Security
    ○ At least we need fixed outbound IP address
    ● Cost
    

    View Slide

31. We want to keep awesome features
    ● Deploy using git from CircleCI
    ● Buildpack
    ● Multiple Dyno Size
    ● One-off dyno
    ● CLI compatibility
    

    View Slide

32. View Slide

33. Your PaaS, Your Rules
    Open Source
    Application Platform
    

    View Slide

34. Similarity to Heroku
    ● Deploy via git
    ● Buildpack
    ● CLI sub command
    ○ Ex: heroku conifg:set <-> deis config:set
    ● Interactive console not yet supported
    ○ Create own tool using Docker API
    

    View Slide

35. Similarity to Heroku is important
    Because we’re deploying one source
    

    View Slide

36. Source: http://docs.deis.io/en/latest/understanding_deis/architecture/
    

    View Slide

37. View Slide

38. View Slide

39. View Slide

40. Components of PaaS for Web app
    ● Router
    ● Controller
    ● Slug Builder
    ● Container Registry
    ● Object Storage
    ● Scheduler
    ● Key Value Store
    ● Database
    ● Git Repository
    ● Logger
    

    View Slide

41. Where is the best
    place to deploy
    each component?
    

    View Slide

42. Late 2015
    

    View Slide

43. Elastic Load
    Balancing
    Router
    Controller
    Builder
    Applications
    etcd
    AZ #1
    AZ #2
    Production Cluster 1st Gen. PosggreSQL
    On Amazon
    RDS
    Docker Registry
    On Amazon S3
    

    View Slide

44. Hard to Scale Out
    

    View Slide

45. April 2016
    

    View Slide

46. Elastic Load
    Balancing
    Router/Applications
    AZ #1
    AZ #2
    Controller
    /Builder
    etcd
    Production Cluster 2nd Gen. PosggreSQL
    On Amazon
    RDS
    Docker Registry
    On Amazon S3
    

    View Slide

47. Noisy Neighbor
    AND
    Cost Increase
    

    View Slide

48. Fleet is too simple
    ● Fleet is not resource aware scheduler
    ● Fleet deploy new container in instance
    which has the least containers
    ● Sometimes make it unbalanced workload
    of clusters
    

    View Slide

49. Bad scheduling of fleet
    High
    High Normal Low
    Low
    Low
    Instance 1 Instance 2 Instance 3
    

    View Slide

50. We hope resource aware scheduling
    High High
    Normal
    Low
    Low
    Instance 1 Instance 2 Instance 3
    Low
    

    View Slide

51. Aug 2016 〜
    

    View Slide

52. Elastic Load
    Balancing
    Router/Apps (Shared)
    AZ #1
    AZ #2
    Controller
    /Builder
    High-Load Apps (Dedicated) etcd
    Production Cluster 3rd Gen PosggreSQL
    On Amazon
    RDS
    Docker Registry
    On Amazon S3
    

    View Slide

53. Tag based Deployment
    High HighI
    Normal
    Low
    Low
    Low
    high=false
    high=true
    

    View Slide

54. Tag based Deployment
    ● Tag = Fleet metadata
    ● Set fleet metadata by cloud-init
    #cloud-config
    Coreos:
    ...
    fleet:
    public-ip: $private_ipv4
    metadata: dataPlane=true,high=true
    #cloud-config
    Coreos:
    ...
    fleet:
    public-ip: $private_ipv4
    metadata: dataPlane=true,routerMesh=true,high=false
    For dedicated For shared
    

    View Slide

55. Elastic Load
    Balancing
    Router
    AZ #1
    AZ #2
    Controller
    /Builder
    Applications etcd
    Staging Cluster PosggreSQL
    On Amazon
    RDS
    Docker Registry
    On Amazon S3
    

    View Slide

56. Other Awesome Features
    

    View Slide

57. Blue Green Deployment
    1. Start new containers
    2. Notify to etcd
    3. Update nginx.conf by confd
    4. Reload router (nginx)
    5. Update routing
    6. Destroy old containers
    

    View Slide

58. Use nginx as reverse proxy for PaaS
    ● Easy to handle naked domain
    ● Resolve slow client issue
    ● Can support HTTP/2
    ● Flexible routing
    ○ A/B test, 10% release, etc ...
    

    View Slide

59. Problem as of now
    

    View Slide

60. Fleet is too simple (again)
    ● Fleet is not resource aware scheduler
    ● Fleet deploy new container in instance
    which has the least containers
    ● Sometimes make it unbalanced workload
    of clusters
    

    View Slide

61. etcd stability
    ● etcd sometimes get high CPU load
    

    View Slide

62. In the future ...
    

    View Slide

63. Deis Workflow (Deis v2)
    ● https://github.com/deis/workflow
    ● Based on Kubernates
    

    View Slide

64. End
    

    View Slide

65. We’re hiring!
    https://www.quipper.com/career/Japan/
    https://www.wantedly.com/companies/quipper
    

    View Slide