Alkemics CI & CD with Jenkins and Docker

Alkemics CI & CD with Jenkins and Docker

Presented by Florent Paulais & Guillaume Morin
SysadminDays #7 : https://sysadmindays.fr/

415efaa445ed983307231341eaa4be55?s=128

Renaud Chaput

October 19, 2017
Tweet

Transcript

  1. 2.

    Summary • Our platform • QA at Alkemics • CI/CD

    workflow • Tests • Build chain • SQL schema changes • What we have learnt & next steps
  2. 4.
  3. 5.

    Alkemics platform • Automation ◦ No more excel files ◦

    Import/export system ◦ Data quality control ◦ Accelerate content updates • Network ◦ Connect to retailers, manufacturers and third parties ◦ France, aiming for the world
  4. 6.

    Infrastructure • 29 micro services in python and go •

    Data storage ◦ Mysql ◦ Cassandra ◦ ElasticSearch ◦ Couchbase
  5. 9.

    SRE • Sysadmin / Devops / QA • Scope ◦

    Infrastructure ◦ Production ◦ Tooling • Automation
  6. 10.

    QA at Alkemics, a brief history 2015 • Single server

    2016 • Multiple docker instances • Browserstack 2017 • Multiple platforms with “Pillar”
  7. 11.
  8. 12.
  9. 13.
  10. 14.
  11. 15.

    Jenkins definition • Pipeline job is a job described with

    a Jenkinsfile for each integration branch • Jenkinsfile ◦ Define a workflow to run tests and deployment in groovy ◦ Stored in service repository • Dedicated groovy lib to share functions between all Jenkinsfile
  12. 30.

    Introducing Pillar • Python tool designed to ◦ Create complete

    environment from scratch ◦ Orchestrate docker instances ◦ Seed data according to pre made scenarios
  13. 32.

    Pillar demo mysql: container_name: teste2e-mysql image: alkreg:5000/alkemics/mysql:staging service-application: command: bash

    -c "sed -i ..." container_name: teste2e-service-application image: alkreg:5000/alkemics/service-application:staging volumes: - service-application-logs:/var/log working_dir: /var/www/service-application
  14. 33.

    Pillar demo core: accounts: retailer-admin: products: '03663215733878': assigned: - 'Alkemics

    Test - Manufacturer' user: permissions: 'ReadWrite': ['product.show', 'product.update']
  15. 37.

    Build chain • Docker images stored and built on the

    registry • Same base image for all tests • Inheritance
  16. 39.

    SQL schema changes • No manual ALTER • shmig ◦

    Included in the repository ◦ Run when deploying in any environment ◦ Periodic merge to speed up tests execution
  17. 41.

    Results Average time to build and test Bugs identified by

    customers between 2016 and 2017 3 min 35 sec - 28 % “Pillar made my skin 51 % softer.” Youcef Mammar, Software Engineer, Alkemics
  18. 42.

    What we have learnt • Building an existing platform from

    scratch is long and sometimes complex • External services are complicated to manage ◦ Use Mock when possible • Backward compatibility is mandatory
  19. 43.

    Next steps • Scalability • Docker in production ◦ On

    demand instances • Environment for developers
  20. 45.

    • Florent Paulais ◦ fpaulais@alkemics.com • Guillaume “arachne” Morin ◦

    gmorin@alkemics.com ◦ @gc_morin on twitter About us