Quipper on Containers

Transcript

1. on Containers )

2. @hakobera DevOps Engineer

3. None

4. Mexico City London Tokyo Manila Jakarta Sales team only Sales

   and dev team International Offices No one here (yet).

5. Products

6. Container != Docker

7. Docker is one of the container impls. Ubuntu LXD Microsoft®

   Windows® Server Container Hyper-V® Container

8. When do you use container on your system?

9. Now!

10. Why do you use container?

11. Portability and Beyond Your Cloud Limits

12. Docker on Elastic Beanstalk • For short batch using worker

    tier • Avoid darkness of .ebextension • Avoid Amazon Linux ◦ We want to use Ubuntu

13. Load Test on Elastic Container Service • Easy to setup

    locust cluster ◦ Consists of 2 Tasks ▪ 1 master ▪ Multiple slave ◦ Manage instances using Spot Fleet ▪ Cheap but scalable

14. nginx-build on CircleCI • Easy to build custom nginx build

    ◦ Bundle custom modules such as ngx_mruby ◦ Patch for security issue ▪ Update only openssl but not nginx

15. Ansible playbook test on CircleCI • Run playbook to clean

    environment • Run playboot test in parallel

16. Do you also use container for Web application?

17. Yes! But it’s not so easy

18. History of Quipper and Container

19. Quipper is Container Native Company

20. Quipper use container from beginning

21. Back to 2012 First Product First Container is ...

22. None

23. It’s too early

24. No source repository Only CTO only knows

25. From 2013 Second Container Choice is ...

26. None

27. Heroku is great!

28. Awesome features of Heroku • Deploy via git • Buildpack

    • Multiple Dyno Size • One-off dyno • Review Apps

29. At April, 2015 We are acquired by Recruit Co. Ltd

30. What problem do we have to resolve? • Performance ◦

    Latency from Japan and USA (400ms/req) ◦ Needed more powerful dyno • Security ◦ At least we need fixed outbound IP address • Cost

31. We want to keep awesome features • Deploy using git

    from CircleCI • Buildpack • Multiple Dyno Size • One-off dyno • CLI compatibility
32. None

33. Your PaaS, Your Rules Open Source Application Platform

34. Similarity to Heroku • Deploy via git • Buildpack •

    CLI sub command ◦ Ex: heroku conifg:set <-> deis config:set • Interactive console not yet supported ◦ Create own tool using Docker API

35. Similarity to Heroku is important Because we’re deploying one source

36. Source: http://docs.deis.io/en/latest/understanding_deis/architecture/

37. None
38. None
39. None

40. Components of PaaS for Web app • Router • Controller

    • Slug Builder • Container Registry • Object Storage • Scheduler • Key Value Store • Database • Git Repository • Logger

41. Where is the best place to deploy each component?

42. Late 2015

43. Elastic Load Balancing Router Controller Builder Applications etcd AZ #1

    AZ #2 Production Cluster 1st Gen. PosggreSQL On Amazon RDS Docker Registry On Amazon S3

44. Hard to Scale Out

45. April 2016

46. Elastic Load Balancing Router/Applications AZ #1 AZ #2 Controller /Builder

    etcd Production Cluster 2nd Gen. PosggreSQL On Amazon RDS Docker Registry On Amazon S3

47. Noisy Neighbor AND Cost Increase

48. Fleet is too simple • Fleet is not resource aware

    scheduler • Fleet deploy new container in instance which has the least containers • Sometimes make it unbalanced workload of clusters

49. Bad scheduling of fleet High High Normal Low Low Low

    Instance 1 Instance 2 Instance 3

50. We hope resource aware scheduling High High Normal Low Low

    Instance 1 Instance 2 Instance 3 Low

51. Aug 2016 〜

52. Elastic Load Balancing Router/Apps (Shared) AZ #1 AZ #2 Controller

    /Builder High-Load Apps (Dedicated) etcd Production Cluster 3rd Gen PosggreSQL On Amazon RDS Docker Registry On Amazon S3

53. Tag based Deployment High HighI Normal Low Low Low high=false

    high=true

54. Tag based Deployment • Tag = Fleet metadata • Set

    fleet metadata by cloud-init #cloud-config Coreos: ... fleet: public-ip: $private_ipv4 metadata: dataPlane=true,high=true #cloud-config Coreos: ... fleet: public-ip: $private_ipv4 metadata: dataPlane=true,routerMesh=true,high=false For dedicated For shared

55. Elastic Load Balancing Router AZ #1 AZ #2 Controller /Builder

    Applications etcd Staging Cluster PosggreSQL On Amazon RDS Docker Registry On Amazon S3

56. Other Awesome Features

57. Blue Green Deployment 1. Start new containers 2. Notify to

    etcd 3. Update nginx.conf by confd 4. Reload router (nginx) 5. Update routing 6. Destroy old containers

58. Use nginx as reverse proxy for PaaS • Easy to

    handle naked domain • Resolve slow client issue • Can support HTTP/2 • Flexible routing ◦ A/B test, 10% release, etc ...

59. Problem as of now

60. Fleet is too simple (again) • Fleet is not resource

    aware scheduler • Fleet deploy new container in instance which has the least containers • Sometimes make it unbalanced workload of clusters

61. etcd stability • etcd sometimes get high CPU load

62. In the future ...

63. Deis Workflow (Deis v2) • https://github.com/deis/workflow • Based on Kubernates

64. End

65. We’re hiring! https://www.quipper.com/career/Japan/ https://www.wantedly.com/companies/quipper