Alkemics CI & CD with Jenkins and Docker

Presented by Florent Paulais & Guillaume Morin
SysadminDays #7 : https://sysadmindays.fr/

Transcript

1. Continuous Integration and Delivery with Jenkins and Docker
2. Summary • Our platform • QA at Alkemics • CI/CD workflow • Tests • Build chain • SQL schema changes • What we have learnt & next steps
3. Alkemics platform Data sharing platform for retail
5. Alkemics platform • Automation ◦ No more excel files ◦ Import/export system ◦ Data quality control ◦ Accelerate content updates • Network ◦ Connect to retailers, manufacturers and third parties ◦ France, aiming for the world
6. Infrastructure • 29 micro services in python and go • Data storage ◦ Mysql ◦ Cassandra ◦ ElasticSearch ◦ Couchbase
7. Infrastructure • 4 environments ◦ Production ◦ Preproduction ◦ Integration ◦ Development
8. The team • 42 engineers ◦ Developers ◦ Data scientists ◦ Category managers ◦ SRE
9. SRE • Sysadmin / Devops / QA • Scope ◦ Infrastructure ◦ Production ◦ Tooling • Automation
10. QA at Alkemics, a brief history 2015 • Single server 2016 • Multiple docker instances • Browserstack 2017 • Multiple platforms with “Pillar”
11. Workflow
12. Workflow
13. Workflow
14. Workflow
15. Jenkins definition • Pipeline job is a job described with a Jenkinsfile for each integration branch • Jenkinsfile ◦ Define a workflow to run tests and deployment in groovy ◦ Stored in service repository • Dedicated groovy lib to share functions between all Jenkinsfile
16. “Unit” tests • Docker controlled by Jenkins • Central MySQL on a dedicated server
17. Docker setup in Jenkins
18. Docker templates
19. Database management
20. Pull Request demo
21. Pull Request demo
22. Pull Request demo
23. Pull Request demo
24. Pull Request demo
25. Pull Request demo
26. Pull Request demo
27. Pull Request demo
28. Pull Request demo
29. Pull Request demo
30. Introducing Pillar • Python tool designed to ◦ Create complete environment from scratch ◦ Orchestrate docker instances ◦ Seed data according to pre made scenarios
31. Integration & End to end tests
32. Pillar demo mysql: container_name: teste2e-mysql image: alkreg:5000/alkemics/mysql:staging service-application: command: bash -c "sed -i ..." container_name: teste2e-service-application image: alkreg:5000/alkemics/service-application:staging volumes: - service-application-logs:/var/log working_dir: /var/www/service-application
33. Pillar demo core: accounts: retailer-admin: products: '03663215733878': assigned: - 'Alkemics Test - Manufacturer' user: permissions: 'ReadWrite': ['product.show', 'product.update']
34. Pillar demo
35. Pillar demo Jenkins pipeline example
36. Pillar demo Browserstack example
37. Build chain • Docker images stored and built on the registry • Same base image for all tests • Inheritance
38. Build chain tree
39. SQL schema changes • No manual ALTER • shmig ◦ Included in the repository ◦ Run when deploying in any environment ◦ Periodic merge to speed up tests execution
40. shmig example
41. Results Average time to build and test Bugs identified by customers between 2016 and 2017 3 min 35 sec - 28 % “Pillar made my skin 51 % softer.” Youcef Mammar, Software Engineer, Alkemics
42. What we have learnt • Building an existing platform from scratch is long and sometimes complex • External services are complicated to manage ◦ Use Mock when possible • Backward compatibility is mandatory
43. Next steps • Scalability • Docker in production ◦ On demand instances • Environment for developers
44. Tools • Jenkins • Docker • Shmig • Ansible • Github • Browserstack
45. • Florent Paulais ◦ fpaulais@alkemics.com • Guillaume “arachne” Morin ◦ gmorin@alkemics.com ◦ @gc_morin on twitter About us