Alkemics CI & CD with Jenkins and Docker

Presented by Florent Paulais & Guillaume Morin
SysadminDays #7 : https://sysadmindays.fr/

Transcript

1. Continuous Integration and Delivery
   with Jenkins and Docker
   

   View Slide

2. Summary
   ● Our platform
   ● QA at Alkemics
   ● CI/CD workflow
   ● Tests
   ● Build chain
   ● SQL schema changes
   ● What we have learnt & next steps
   

   View Slide

3. Alkemics platform
   Data sharing platform for retail
   

   View Slide

4. View Slide

5. Alkemics platform
   ● Automation
   ○ No more excel files
   ○ Import/export system
   ○ Data quality control
   ○ Accelerate content updates
   ● Network
   ○ Connect to retailers, manufacturers and third parties
   ○ France, aiming for the world
   

   View Slide

6. Infrastructure
   ● 29 micro services in python and go
   ● Data storage
   ○ Mysql
   ○ Cassandra
   ○ ElasticSearch
   ○ Couchbase
   

   View Slide

7. Infrastructure
   ● 4 environments
   ○ Production
   ○ Preproduction
   ○ Integration
   ○ Development
   

   View Slide

8. The team
   ● 42 engineers
   ○ Developers
   ○ Data scientists
   ○ Category managers
   ○ SRE
   

   View Slide

9. SRE
   ● Sysadmin / Devops / QA
   ● Scope
   ○ Infrastructure
   ○ Production
   ○ Tooling
   ● Automation
   

   View Slide

10. QA at Alkemics, a brief history
    2015 ● Single server
    2016
    ● Multiple docker instances
    ● Browserstack
    2017 ● Multiple platforms with “Pillar”
    

    View Slide

11. Workflow
    

    View Slide

12. Workflow
    

    View Slide

13. Workflow
    

    View Slide

14. Workflow
    

    View Slide

15. Jenkins definition
    ● Pipeline job is a job described with a
    Jenkinsfile for each integration branch
    ● Jenkinsfile
    ○ Define a workflow to run tests and
    deployment in groovy
    ○ Stored in service repository
    ● Dedicated groovy lib to share functions
    between all Jenkinsfile
    

    View Slide

16. “Unit” tests
    ● Docker controlled by Jenkins
    ● Central MySQL on a dedicated server
    

    View Slide

17. Docker setup in Jenkins
    

    View Slide

18. Docker templates
    

    View Slide

19. Database management
    

    View Slide

20. Pull Request demo
    

    View Slide

21. Pull Request demo
    

    View Slide

22. Pull Request demo
    

    View Slide

23. Pull Request demo
    

    View Slide

24. Pull Request demo
    

    View Slide

25. Pull Request demo
    

    View Slide

26. Pull Request demo
    

    View Slide

27. Pull Request demo
    

    View Slide

28. Pull Request demo
    

    View Slide

29. Pull Request demo
    

    View Slide

30. Introducing Pillar
    ● Python tool designed to
    ○ Create complete environment from scratch
    ○ Orchestrate docker instances
    ○ Seed data according to pre made scenarios
    

    View Slide

31. Integration & End to end tests
    

    View Slide

32. Pillar demo
    mysql:
    container_name: teste2e-mysql
    image: alkreg:5000/alkemics/mysql:staging
    service-application:
    command: bash -c "sed -i ..."
    container_name: teste2e-service-application
    image: alkreg:5000/alkemics/service-application:staging
    volumes:
    - service-application-logs:/var/log
    working_dir: /var/www/service-application
    

    View Slide

33. Pillar demo
    core:
    accounts:
    retailer-admin:
    products:
    '03663215733878':
    assigned:
    - 'Alkemics Test - Manufacturer'
    user:
    permissions:
    'ReadWrite': ['product.show', 'product.update']
    

    View Slide

34. Pillar demo
    

    View Slide

35. Pillar demo
    Jenkins pipeline example
    

    View Slide

36. Pillar demo
    Browserstack example
    

    View Slide

37. Build chain
    ● Docker images stored and built on the registry
    ● Same base image for all tests
    ● Inheritance
    

    View Slide

38. Build chain tree
    

    View Slide

39. SQL schema changes
    ● No manual ALTER
    ● shmig
    ○ Included in the repository
    ○ Run when deploying in any environment
    ○ Periodic merge to speed up tests
    execution
    

    View Slide

40. shmig example
    

    View Slide

41. Results
    Average time to build and test
    Bugs identified by customers
    between 2016 and 2017
    3 min 35 sec - 28 %
    “Pillar made my skin 51 % softer.”
    Youcef Mammar, Software Engineer, Alkemics
    

    View Slide

42. What we have learnt
    ● Building an existing platform from scratch is
    long and sometimes complex
    ● External services are complicated to manage
    ○ Use Mock when possible
    ● Backward compatibility is mandatory
    

    View Slide

43. Next steps
    ● Scalability
    ● Docker in production
    ○ On demand instances
    ● Environment for developers
    

    View Slide

44. Tools
    ● Jenkins
    ● Docker
    ● Shmig
    ● Ansible
    ● Github
    ● Browserstack
    

    View Slide

45. ● Florent Paulais
    ○ fpaulais@alkemics.com
    ● Guillaume “arachne” Morin
    ○ gmorin@alkemics.com
    ○ @gc_morin on twitter
    About us
    

    View Slide