Alkemics CI & CD with Jenkins and Docker

Alkemics CI & CD with Jenkins and Docker

Presented by Florent Paulais & Guillaume Morin
SysadminDays #7 : https://sysadmindays.fr/

415efaa445ed983307231341eaa4be55?s=128

Renaud Chaput

October 19, 2017
Tweet

Transcript

  1. Continuous Integration and Delivery with Jenkins and Docker

  2. Summary • Our platform • QA at Alkemics • CI/CD

    workflow • Tests • Build chain • SQL schema changes • What we have learnt & next steps
  3. Alkemics platform Data sharing platform for retail

  4. None
  5. Alkemics platform • Automation ◦ No more excel files ◦

    Import/export system ◦ Data quality control ◦ Accelerate content updates • Network ◦ Connect to retailers, manufacturers and third parties ◦ France, aiming for the world
  6. Infrastructure • 29 micro services in python and go •

    Data storage ◦ Mysql ◦ Cassandra ◦ ElasticSearch ◦ Couchbase
  7. Infrastructure • 4 environments ◦ Production ◦ Preproduction ◦ Integration

    ◦ Development
  8. The team • 42 engineers ◦ Developers ◦ Data scientists

    ◦ Category managers ◦ SRE
  9. SRE • Sysadmin / Devops / QA • Scope ◦

    Infrastructure ◦ Production ◦ Tooling • Automation
  10. QA at Alkemics, a brief history 2015 • Single server

    2016 • Multiple docker instances • Browserstack 2017 • Multiple platforms with “Pillar”
  11. Workflow

  12. Workflow

  13. Workflow

  14. Workflow

  15. Jenkins definition • Pipeline job is a job described with

    a Jenkinsfile for each integration branch • Jenkinsfile ◦ Define a workflow to run tests and deployment in groovy ◦ Stored in service repository • Dedicated groovy lib to share functions between all Jenkinsfile
  16. “Unit” tests • Docker controlled by Jenkins • Central MySQL

    on a dedicated server
  17. Docker setup in Jenkins

  18. Docker templates

  19. Database management

  20. Pull Request demo

  21. Pull Request demo

  22. Pull Request demo

  23. Pull Request demo

  24. Pull Request demo

  25. Pull Request demo

  26. Pull Request demo

  27. Pull Request demo

  28. Pull Request demo

  29. Pull Request demo

  30. Introducing Pillar • Python tool designed to ◦ Create complete

    environment from scratch ◦ Orchestrate docker instances ◦ Seed data according to pre made scenarios
  31. Integration & End to end tests

  32. Pillar demo mysql: container_name: teste2e-mysql image: alkreg:5000/alkemics/mysql:staging service-application: command: bash

    -c "sed -i ..." container_name: teste2e-service-application image: alkreg:5000/alkemics/service-application:staging volumes: - service-application-logs:/var/log working_dir: /var/www/service-application
  33. Pillar demo core: accounts: retailer-admin: products: '03663215733878': assigned: - 'Alkemics

    Test - Manufacturer' user: permissions: 'ReadWrite': ['product.show', 'product.update']
  34. Pillar demo

  35. Pillar demo Jenkins pipeline example

  36. Pillar demo Browserstack example

  37. Build chain • Docker images stored and built on the

    registry • Same base image for all tests • Inheritance
  38. Build chain tree

  39. SQL schema changes • No manual ALTER • shmig ◦

    Included in the repository ◦ Run when deploying in any environment ◦ Periodic merge to speed up tests execution
  40. shmig example

  41. Results Average time to build and test Bugs identified by

    customers between 2016 and 2017 3 min 35 sec - 28 % “Pillar made my skin 51 % softer.” Youcef Mammar, Software Engineer, Alkemics
  42. What we have learnt • Building an existing platform from

    scratch is long and sometimes complex • External services are complicated to manage ◦ Use Mock when possible • Backward compatibility is mandatory
  43. Next steps • Scalability • Docker in production ◦ On

    demand instances • Environment for developers
  44. Tools • Jenkins • Docker • Shmig • Ansible •

    Github • Browserstack
  45. • Florent Paulais ◦ fpaulais@alkemics.com • Guillaume “arachne” Morin ◦

    gmorin@alkemics.com ◦ @gc_morin on twitter About us