$30 off During Our Annual Pro Sale. View Details »

Alkemics CI & CD with Jenkins and Docker

Alkemics CI & CD with Jenkins and Docker

Presented by Florent Paulais & Guillaume Morin
SysadminDays #7 : https://sysadmindays.fr/

Renaud Chaput

October 19, 2017
Tweet

More Decks by Renaud Chaput

Other Decks in Technology

Transcript

  1. Continuous Integration and Delivery
    with Jenkins and Docker

    View Slide

  2. Summary
    ● Our platform
    ● QA at Alkemics
    ● CI/CD workflow
    ● Tests
    ● Build chain
    ● SQL schema changes
    ● What we have learnt & next steps

    View Slide

  3. Alkemics platform
    Data sharing platform for retail

    View Slide

  4. View Slide

  5. Alkemics platform
    ● Automation
    ○ No more excel files
    ○ Import/export system
    ○ Data quality control
    ○ Accelerate content updates
    ● Network
    ○ Connect to retailers, manufacturers and third parties
    ○ France, aiming for the world

    View Slide

  6. Infrastructure
    ● 29 micro services in python and go
    ● Data storage
    ○ Mysql
    ○ Cassandra
    ○ ElasticSearch
    ○ Couchbase

    View Slide

  7. Infrastructure
    ● 4 environments
    ○ Production
    ○ Preproduction
    ○ Integration
    ○ Development

    View Slide

  8. The team
    ● 42 engineers
    ○ Developers
    ○ Data scientists
    ○ Category managers
    ○ SRE

    View Slide

  9. SRE
    ● Sysadmin / Devops / QA
    ● Scope
    ○ Infrastructure
    ○ Production
    ○ Tooling
    ● Automation

    View Slide

  10. QA at Alkemics, a brief history
    2015 ● Single server
    2016
    ● Multiple docker instances
    ● Browserstack
    2017 ● Multiple platforms with “Pillar”

    View Slide

  11. Workflow

    View Slide

  12. Workflow

    View Slide

  13. Workflow

    View Slide

  14. Workflow

    View Slide

  15. Jenkins definition
    ● Pipeline job is a job described with a
    Jenkinsfile for each integration branch
    ● Jenkinsfile
    ○ Define a workflow to run tests and
    deployment in groovy
    ○ Stored in service repository
    ● Dedicated groovy lib to share functions
    between all Jenkinsfile

    View Slide

  16. “Unit” tests
    ● Docker controlled by Jenkins
    ● Central MySQL on a dedicated server

    View Slide

  17. Docker setup in Jenkins

    View Slide

  18. Docker templates

    View Slide

  19. Database management

    View Slide

  20. Pull Request demo

    View Slide

  21. Pull Request demo

    View Slide

  22. Pull Request demo

    View Slide

  23. Pull Request demo

    View Slide

  24. Pull Request demo

    View Slide

  25. Pull Request demo

    View Slide

  26. Pull Request demo

    View Slide

  27. Pull Request demo

    View Slide

  28. Pull Request demo

    View Slide

  29. Pull Request demo

    View Slide

  30. Introducing Pillar
    ● Python tool designed to
    ○ Create complete environment from scratch
    ○ Orchestrate docker instances
    ○ Seed data according to pre made scenarios

    View Slide

  31. Integration & End to end tests

    View Slide

  32. Pillar demo
    mysql:
    container_name: teste2e-mysql
    image: alkreg:5000/alkemics/mysql:staging
    service-application:
    command: bash -c "sed -i ..."
    container_name: teste2e-service-application
    image: alkreg:5000/alkemics/service-application:staging
    volumes:
    - service-application-logs:/var/log
    working_dir: /var/www/service-application

    View Slide

  33. Pillar demo
    core:
    accounts:
    retailer-admin:
    products:
    '03663215733878':
    assigned:
    - 'Alkemics Test - Manufacturer'
    user:
    permissions:
    'ReadWrite': ['product.show', 'product.update']

    View Slide

  34. Pillar demo

    View Slide

  35. Pillar demo
    Jenkins pipeline example

    View Slide

  36. Pillar demo
    Browserstack example

    View Slide

  37. Build chain
    ● Docker images stored and built on the registry
    ● Same base image for all tests
    ● Inheritance

    View Slide

  38. Build chain tree

    View Slide

  39. SQL schema changes
    ● No manual ALTER
    ● shmig
    ○ Included in the repository
    ○ Run when deploying in any environment
    ○ Periodic merge to speed up tests
    execution

    View Slide

  40. shmig example

    View Slide

  41. Results
    Average time to build and test
    Bugs identified by customers
    between 2016 and 2017
    3 min 35 sec - 28 %
    “Pillar made my skin 51 % softer.”
    Youcef Mammar, Software Engineer, Alkemics

    View Slide

  42. What we have learnt
    ● Building an existing platform from scratch is
    long and sometimes complex
    ● External services are complicated to manage
    ○ Use Mock when possible
    ● Backward compatibility is mandatory

    View Slide

  43. Next steps
    ● Scalability
    ● Docker in production
    ○ On demand instances
    ● Environment for developers

    View Slide

  44. Tools
    ● Jenkins
    ● Docker
    ● Shmig
    ● Ansible
    ● Github
    ● Browserstack

    View Slide

  45. ● Florent Paulais
    [email protected]
    ● Guillaume “arachne” Morin
    [email protected]
    ○ @gc_morin on twitter
    About us

    View Slide