Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Travisify - Project life cycle in the own repository

Tuenti
November 09, 2014

Travisify - Project life cycle in the own repository

Miguel Ángel's (@magmax9) talk at PyConEs'14

Tuenti

November 09, 2014
Tweet

More Decks by Tuenti

Other Decks in Technology

Transcript

  1. TRAVISIFY
    PROJECT LIFE CICLE IN THE
    OWN REPOSITORY

    View Slide

  2. ABOUT ME
    Miguel Ángel García
    SRE (Systems Reliability Engineer) at
    @magmax9
    http://magmax.org

    View Slide

  3. ONCE UPON A
    TIME...

    View Slide

  4. JUST ONE BIG REPOSITORY

    View Slide

  5. TESTING EVOLUTION
    No testing at all
    Some manual testing performed by QA team
    Some Acceptation tests written by QA team
    Unit, Integration and Acceptation tests
    written by developers (no QA team)

    View Slide

  6. The bad guy at the begining
    Time saver later
    Deployments center right now

    View Slide

  7. TESTING
    Ad hoc testing environment
    One team to manage the testing environment:
    Testing Framework (TFW)

    View Slide

  8. PROBLEMS
    It doesn't scale to other repositories/projects
    The team was not able to deal with problems
    developers are used to.
    Developers want to improve to that environment

    View Slide

  9. THE PRESENT

    View Slide

  10. A BUNCH OF REPOSITORIES

    View Slide

  11. PROBLEMS
    Impossible to be managed by just one team
    Too different technologies:
    PHP: fpm, hphpi
    Java: tomcat, jboss, android
    Erlang, Python, Ruby, ...
    make, rake, ant, maven, gradle, ...

    View Slide

  12. Another aproach is required:

    View Slide

  13. TRAVIS IDEA
    The repository contains more than code:
    The definition of the testing environment
    The test execution commands

    View Slide

  14. OUR REQUIREMENTS
    One job per branch
    Dynamic reconfiguration
    Special jobs for releases
    Just one testing environment per project
    Some projects require several environments (client-
    server)

    View Slide

  15. OUR WISHLIST
    Reuse Jenkins if possible...
    ... and avoid coupling if possible

    View Slide

  16. View Slide

  17. APROACH
    Jenkins has a Python API
    Jenkins job definition is made in XML

    View Slide

  18. APROACH
    1. Create a XML template
    2. Parameterize the XML template
    3. Use a YAML file in the repo to store values

    View Slide

  19. Later on, we found that our library must communicate
    with Jenkins in order to:
    ERRORS
    Check if a plugin is installed
    Check a plugin version
    Still to be done...

    View Slide

  20. USE CASE 1: CREATION
    We created the "tu-ci" (Tuenti Continuous Integration)
    command:
    Reads information about working copy
    Sends a message to Flow to create the repository
    Flow reads the .ci.yml and reconfigures the Jenkins
    Job

    View Slide

  21. FLOW
    Is our centralized deployments tool
    Django
    You must be tired to hear us talking about it

    View Slide

  22. USE CASE 2: UPDATE
    We added a hook in our git and mercurial repositories:
    It just sends a curl to Flow
    Flow regenerates the Jenkins Job

    View Slide

  23. USE CASE 3: LAUNCH
    The Jenkins Job:
    1. Generates a VirtualEnv environment
    2. Installs cigen
    3. Uses the cigen runner to parse(again) the .ci.yml

    View Slide

  24. TELL ME
    WHAT TO DO
    And not where, when or how
    Write meta-commands in order to help developers:
    Create resources for servers
    (BBDDs, memcache, ...)
    Deploy the server side for clients
    Launch VMs

    View Slide

  25. META FEATURES
    We discovered that:
    Some commands must be executed to setup a job
    Some commands should be executed on
    success/failure

    View Slide

  26. EASY TO IMPLEMENT WITH
    CURRENT DESIGN
    Just added new mandatory stages!

    View Slide

  27. SHOW ME
    THE CODE!
    Not yet :(

    View Slide

  28. SCHEMA
    Platform where it should run
    Workflows are lists of Stages
    Stages are lists of Steps
    Steps are shell scripts
    Plugins are... a best effort :D

    View Slide

  29. SCHEMA
    PYTHON RULES!!
    d
    e
    f p
    r
    o
    j
    e
    c
    t
    _
    f
    r
    o
    m
    _
    s
    t
    r
    e
    a
    m
    (
    s
    t
    r
    e
    a
    m
    , p
    r
    o
    j
    e
    c
    t
    =
    N
    o
    n
    e
    , y
    a
    m
    l
    _
    l
    i
    b
    =
    N
    o
    n
    e
    )
    :
    y
    a
    m
    l
    _
    l
    i
    b = y
    a
    m
    l
    _
    l
    i
    b o
    r y
    a
    m
    l
    p
    r
    o
    j
    e
    c
    t = p
    r
    o
    j
    e
    c
    t o
    r P
    r
    o
    j
    e
    c
    t
    (
    )
    c
    o
    n
    t
    e
    n
    t = y
    a
    m
    l
    _
    l
    i
    b
    .
    l
    o
    a
    d
    (
    s
    t
    r
    e
    a
    m
    )
    p
    r
    o
    j
    e
    c
    t
    .
    p
    l
    a
    t
    f
    o
    r
    m = c
    o
    n
    t
    e
    n
    t
    .
    g
    e
    t
    (
    '
    p
    l
    a
    t
    f
    o
    r
    m
    '
    , '
    d
    e
    f
    a
    u
    l
    t
    '
    )
    p
    r
    o
    j
    e
    c
    t
    .
    t
    i
    m
    e
    o
    u
    t = c
    o
    n
    t
    e
    n
    t
    .
    g
    e
    t
    (
    '
    t
    i
    m
    e
    o
    u
    t
    '
    , p
    r
    o
    j
    e
    c
    t
    .
    t
    i
    m
    e
    o
    u
    t
    )
    p
    r
    o
    j
    e
    c
    t
    .
    p
    a
    r
    a
    m
    e
    t
    e
    r
    s = c
    o
    n
    t
    e
    n
    t
    .
    g
    e
    t
    (
    '
    p
    a
    r
    a
    m
    e
    t
    e
    r
    s
    '
    , {
    }
    )
    p
    r
    o
    j
    e
    c
    t
    .
    p
    l
    u
    g
    i
    n
    s = c
    o
    n
    t
    e
    n
    t
    .
    g
    e
    t
    (
    '
    p
    l
    u
    g
    i
    n
    s
    '
    , {
    }
    )
    p
    r
    o
    j
    e
    c
    t
    .
    s
    t
    a
    g
    e
    s = c
    o
    n
    t
    e
    n
    t
    .
    g
    e
    t
    (
    '
    s
    t
    a
    g
    e
    s
    '
    , {
    }
    )
    p
    r
    o
    j
    e
    c
    t
    .
    w
    o
    r
    k
    f
    l
    o
    w
    s = c
    o
    n
    t
    e
    n
    t
    .
    g
    e
    t
    (
    '
    w
    o
    r
    k
    f
    l
    o
    w
    s
    '
    , {
    }
    )
    r
    e
    t
    u
    r
    n p
    r
    o
    j
    e
    c
    t

    View Slide

  30. SCHEMA EXAMPLE
    p
    l
    a
    t
    f
    o
    r
    m
    : a
    n
    d
    r
    o
    i
    d
    w
    o
    r
    k
    f
    l
    o
    w
    s
    :
    f
    a
    s
    t
    : [
    c
    o
    m
    p
    i
    l
    e
    , u
    n
    i
    t
    ]
    f
    u
    l
    l
    : [
    c
    o
    m
    p
    i
    l
    e
    , u
    n
    i
    t
    , i
    n
    t
    e
    g
    r
    a
    t
    i
    o
    n
    ]
    r
    e
    l
    e
    a
    s
    e
    : [
    c
    o
    m
    p
    i
    l
    e
    , u
    n
    i
    t
    , p
    u
    b
    l
    i
    s
    h
    ]
    s
    t
    a
    g
    e
    s
    :
    c
    o
    m
    p
    i
    l
    e
    :
    - f
    i
    n
    d -
    n
    a
    m
    e "
    *
    .
    p
    y
    c
    " -
    d
    e
    l
    e
    t
    e
    - m
    a
    k
    e
    u
    n
    i
    t
    : p
    y
    .
    t
    e
    s
    t t
    e
    s
    t
    s
    /
    u
    n
    i
    t
    i
    n
    t
    e
    g
    r
    a
    t
    i
    o
    n
    : p
    y
    .
    t
    e
    s
    t t
    e
    s
    t
    s
    /
    i
    n
    t
    e
    g
    r
    a
    t
    i
    o
    n
    p
    u
    b
    l
    i
    s
    h
    : p
    y
    t
    h
    o
    n s
    e
    t
    u
    p
    .
    p
    y b
    d
    i
    s
    t
    _
    w
    h
    e
    e
    l u
    p
    l
    o
    a
    d
    p
    l
    u
    g
    i
    n
    s
    :
    c
    o
    v
    e
    r
    a
    g
    e
    :
    f
    i
    l
    e
    : c
    o
    v
    e
    r
    a
    g
    e
    .
    x
    m
    l
    s
    o
    n
    a
    r
    :

    View Slide

  31. PLUGINS
    Junit/PHPUnit
    Coverage
    Sonar
    Automatic versioning

    View Slide

  32. USE CASE 1: CREATION

    View Slide

  33. c
    l
    a
    s
    s T
    e
    m
    p
    l
    a
    t
    e
    L
    o
    a
    d
    e
    r
    (
    o
    b
    j
    e
    c
    t
    )
    :
    d
    e
    f g
    e
    t
    _
    t
    e
    m
    p
    l
    a
    t
    e
    (
    s
    e
    l
    f
    , n
    a
    m
    e
    )
    :
    a
    s
    s
    e
    r
    t i
    s
    i
    n
    s
    t
    a
    n
    c
    e
    (
    n
    a
    m
    e
    , s
    t
    r
    )
    t
    e
    m
    p
    l
    a
    t
    e
    _
    f
    i
    l
    e = s
    e
    l
    f
    .
    _
    t
    e
    m
    p
    l
    a
    t
    e
    _
    f
    i
    l
    e
    (
    n
    a
    m
    e
    )
    l
    o
    g
    g
    e
    r
    .
    d
    e
    b
    u
    g
    (
    "
    L
    o
    a
    d
    i
    n
    g t
    e
    m
    p
    l
    a
    t
    e %
    s
    "
    , t
    e
    m
    p
    l
    a
    t
    e
    _
    f
    i
    l
    e
    )
    w
    i
    t
    h o
    p
    e
    n
    (
    t
    e
    m
    p
    l
    a
    t
    e
    _
    f
    i
    l
    e
    ) a
    s f
    d
    :
    r
    e
    t
    u
    r
    n T
    e
    m
    p
    l
    a
    t
    e
    (
    f
    d
    .
    r
    e
    a
    d
    (
    )
    )
    d
    e
    f _
    t
    e
    m
    p
    l
    a
    t
    e
    _
    f
    i
    l
    e
    (
    s
    e
    l
    f
    , n
    a
    m
    e
    )
    :
    c
    u
    r
    r
    e
    n
    t = o
    s
    .
    p
    a
    t
    h
    .
    d
    i
    r
    n
    a
    m
    e
    (
    _
    _
    f
    i
    l
    e
    _
    _
    )
    f
    i
    l
    e
    n
    a
    m
    e = n
    a
    m
    e + '
    .
    j
    i
    n
    j
    a
    '
    r
    e
    t
    u
    r
    n o
    s
    .
    p
    a
    t
    h
    .
    j
    o
    i
    n
    (
    c
    u
    r
    r
    e
    n
    t
    , '
    t
    e
    m
    p
    l
    a
    t
    e
    s
    '
    , f
    i
    l
    e
    n
    a
    m
    e
    )
    c
    l
    a
    s
    s G
    e
    n
    e
    r
    a
    t
    o
    r
    (
    o
    b
    j
    e
    c
    t
    )
    :
    T
    E
    M
    P
    L
    A
    T
    E = N
    o
    n
    e
    d
    e
    f _
    _
    i
    n
    i
    t
    _
    _
    (
    s
    e
    l
    f
    , p
    r
    o
    j
    e
    c
    t
    , l
    o
    a
    d
    e
    r
    =
    N
    o
    n
    e
    )
    :
    s
    e
    l
    f
    .
    p
    r
    o
    j
    e
    c
    t = p
    r
    o
    j
    e
    c
    t
    s
    e
    l
    f
    .
    _
    l
    o
    a
    d
    e
    r = l
    o
    a
    d
    e
    r o
    r T
    e
    m
    p
    l
    a
    t
    e
    L
    o
    a
    d
    e
    r
    (
    )
    d
    e
    f g
    e
    n
    e
    r
    a
    t
    e
    (
    s
    e
    l
    f
    , r
    e
    l
    a
    t
    i
    v
    e
    _
    p
    r
    o
    j
    e
    c
    t
    _
    f
    i
    l
    e
    _
    p
    a
    t
    h
    , w
    o
    r
    k
    f
    l
    o
    w
    )
    :
    t
    e
    m
    p
    l
    a
    t
    e = s
    e
    l
    f
    .
    _
    l
    o
    a
    d
    e
    r
    .
    g
    e
    t
    _
    t
    e
    m
    p
    l
    a
    t
    e
    (
    s
    e
    l
    f
    .
    T
    E
    M
    P
    L
    A
    T
    E
    )
    r
    e
    t
    u
    r
    n t
    e
    m
    p
    l
    a
    t
    e
    .
    r
    e
    n
    d
    e
    r
    (
    p
    r
    o
    j
    e
    c
    t
    =
    s
    e
    l
    f
    .
    p
    r
    o
    j
    e
    c
    t
    ,
    c
    r
    e
    a
    t
    i
    o
    n
    _
    t
    i
    m
    e
    =
    t
    i
    m
    e
    .
    t
    i
    m
    e
    (
    )
    ,
    p
    r
    o
    j
    e
    c
    t
    _
    f
    i
    l
    e
    _
    p
    a
    t
    h
    =
    r
    e
    l
    a
    t
    i
    v
    e
    _
    p
    r
    o
    j
    e
    c
    t
    _
    f
    i
    l
    e
    _
    p
    a
    t
    h
    ,
    w
    o
    r
    k
    f
    l
    o
    w
    =
    w
    o
    r
    k
    f
    l
    o
    w
    ,
    c
    i
    g
    e
    n
    =
    C
    I
    G
    E
    N
    ,
    )

    View Slide

  34. c
    l
    a
    s
    s J
    e
    n
    k
    i
    n
    s
    G
    e
    n
    e
    r
    a
    t
    o
    r
    (
    G
    e
    n
    e
    r
    a
    t
    o
    r
    )
    :
    T
    E
    M
    P
    L
    A
    T
    E
    N
    A
    M
    E = '
    j
    e
    n
    k
    i
    n
    s
    '

    View Slide

  35. p
    r
    o
    j
    e
    c
    t = c
    i
    g
    e
    n
    .
    l
    o
    a
    d
    e
    r
    .
    p
    r
    o
    j
    e
    c
    t
    _
    f
    r
    o
    m
    _
    f
    i
    l
    e
    (
    a
    r
    g
    s
    .
    p
    r
    o
    j
    e
    c
    t
    _
    f
    i
    l
    e
    )
    p
    r
    o
    j
    e
    c
    t
    .
    r
    e
    p
    o
    s
    i
    t
    o
    r
    y
    .
    u
    r
    l = '
    h
    t
    t
    p
    :
    /
    /
    e
    x
    a
    m
    p
    l
    e
    .
    o
    r
    g
    /
    f
    o
    o
    /
    b
    a
    r
    '
    p
    r
    o
    j
    e
    c
    t
    .
    r
    e
    p
    o
    s
    i
    t
    o
    r
    y
    .
    k
    i
    n
    d = c
    i
    g
    e
    n
    .
    p
    r
    o
    j
    e
    c
    t
    .
    R
    e
    p
    o
    s
    i
    t
    o
    r
    y
    .
    G
    I
    T
    p
    r
    o
    j
    e
    c
    t
    .
    r
    e
    p
    o
    s
    i
    t
    o
    r
    y
    .
    b
    r
    a
    n
    c
    h = a
    r
    g
    s
    .
    r
    e
    p
    o
    _
    b
    r
    a
    n
    c
    h
    g
    e
    n
    e
    r
    a
    t
    o
    r = c
    i
    g
    e
    n
    .
    g
    e
    n
    e
    r
    a
    t
    o
    r
    .
    J
    e
    n
    k
    i
    n
    s
    G
    e
    n
    e
    r
    a
    t
    o
    r
    (
    p
    r
    o
    j
    e
    c
    t
    )
    p
    r
    i
    n
    t g
    e
    n
    e
    r
    a
    t
    o
    r
    .
    g
    e
    n
    e
    r
    a
    t
    e
    (
    a
    r
    g
    s
    .
    p
    r
    o
    j
    e
    c
    t
    _
    f
    i
    l
    e
    , a
    r
    g
    s
    .
    w
    o
    r
    k
    f
    l
    o
    w
    )

    View Slide

  36. USE CASE 3: CREATION

    View Slide

  37. d
    e
    f r
    u
    n
    (
    s
    e
    l
    f
    )
    :
    s
    e
    l
    f
    .
    _
    c
    o
    n
    f
    i
    g
    u
    r
    e
    _
    p
    l
    u
    g
    i
    n
    s
    (
    )
    s
    e
    l
    f
    .
    _
    u
    p
    d
    a
    t
    e
    _
    e
    n
    v
    i
    r
    o
    n
    m
    e
    n
    t
    (
    )
    i
    f '
    s
    e
    t
    u
    p
    ' i
    n s
    e
    l
    f
    .
    _
    c
    o
    n
    f
    i
    g
    :
    s
    e
    l
    f
    .
    _
    r
    u
    n
    _
    s
    t
    a
    g
    e
    (
    '
    D
    e
    f
    a
    u
    l
    t S
    e
    t
    u
    p
    '
    , s
    e
    l
    f
    .
    _
    c
    o
    n
    f
    i
    g
    [
    '
    s
    e
    t
    u
    p
    '
    ]
    )
    t
    r
    y
    :
    f
    o
    r s
    t
    a
    g
    e i
    n s
    e
    l
    f
    .
    _
    p
    r
    o
    j
    e
    c
    t
    .
    w
    o
    r
    k
    f
    l
    o
    w
    s
    [
    s
    e
    l
    f
    .
    _
    w
    o
    r
    k
    f
    l
    o
    w
    ]
    :
    s
    t
    e
    p
    s = s
    e
    l
    f
    .
    _
    p
    r
    o
    j
    e
    c
    t
    .
    s
    t
    a
    g
    e
    s
    .
    g
    e
    t
    (
    s
    t
    a
    g
    e
    , N
    o
    n
    e
    )
    i
    f s
    t
    e
    p
    s i
    s N
    o
    n
    e
    :
    l
    o
    g
    g
    e
    r
    .
    w
    a
    r
    n
    i
    n
    g
    (
    '
    E
    r
    r
    o
    r o
    n S
    t
    a
    g
    e "
    %
    s
    "
    '
    , s
    t
    a
    g
    e
    )
    c
    o
    n
    t
    i
    n
    u
    e
    s
    e
    l
    f
    .
    _
    r
    u
    n
    _
    s
    t
    a
    g
    e
    (
    s
    t
    a
    g
    e
    , s
    t
    e
    p
    s
    )
    s
    e
    l
    f
    .
    _
    r
    u
    n
    _
    s
    t
    a
    g
    e
    (
    '
    D
    e
    f
    a
    u
    l
    t S
    u
    c
    c
    e
    s
    s
    '
    ,
    s
    e
    l
    f
    .
    _
    c
    o
    n
    f
    i
    g
    .
    g
    e
    t
    (
    '
    o
    n
    _
    s
    u
    c
    c
    e
    s
    s
    '
    , [
    ]
    )
    ,
    p
    l
    u
    g
    i
    n
    =
    '
    o
    n
    _
    s
    '
    )
    l
    o
    g
    g
    e
    r
    .
    i
    n
    f
    o
    (
    '
    S
    U
    C
    C
    E
    S
    S
    '
    )
    e
    x
    c
    e
    p
    t E
    x
    c
    e
    p
    t
    i
    o
    n
    :
    i
    f '
    o
    n
    _
    f
    a
    i
    l
    u
    r
    e
    ' i
    n s
    e
    l
    f
    .
    _
    c
    o
    n
    f
    i
    g
    :
    s
    e
    l
    f
    .
    _
    r
    u
    n
    _
    s
    t
    a
    g
    e
    (
    '
    D
    e
    f
    . F
    a
    i
    l
    u
    r
    e
    '
    ,
    s
    e
    l
    f
    .
    _
    c
    o
    n
    f
    i
    g
    [
    '
    o
    n
    _
    f
    '
    ]
    )
    r
    a
    i
    s
    e
    f
    i
    n
    a
    l
    l
    y
    :
    i
    f '
    t
    e
    a
    r
    d
    o
    w
    n
    ' i
    n s
    e
    l
    f
    .
    _
    c
    o
    n
    f
    i
    g
    :
    s
    e
    l
    f
    .
    _
    r
    u
    n
    _
    s
    t
    a
    g
    e
    (
    '
    D
    e
    f
    . T
    e
    a
    r
    d
    o
    w
    n
    '
    , s
    e
    l
    f
    .
    _
    c
    o
    n
    f
    i
    g
    [
    '
    t
    .
    '
    ]
    )
    s
    e
    l
    f
    .
    _
    p
    r
    i
    n
    t
    _
    r
    e
    p
    o
    r
    t
    (
    )

    View Slide

  38. THE FUTURE

    View Slide

  39. View Slide

  40. IMPROVEMENTS
    More configurable by developers:
    Plots
    Multi-platform (server+client)
    Artifacts to store
    Notifications

    View Slide

  41. NEW REQUIREMENTS
    Cigen should communicate to Jenkins
    Check if a plugin is installed
    Check plugin version and compatibility
    Modularize the current (HUGE) template
    Move logic to code instead of template
    Improve integration:
    Go-CI, Travis, ...
    Jira, Taiga, ...

    View Slide

  42. DEPLOYMENTS
    Just one idea:
    To manage deployments in the same way

    View Slide

  43. GOAL

    View Slide

  44. GOAL

    View Slide

  45. GOAL

    View Slide

  46. GOAL

    View Slide

  47. GOAL

    View Slide

  48. GOAL

    View Slide

  49. GOAL

    View Slide

  50. GOAL

    View Slide

  51. REQUIREMENT
    Software knows WHAT to be deployed
    But not WHERE, WHEN or HOW

    View Slide

  52. VAGRANT

    View Slide

  53. DOCKER

    View Slide

  54. PANAMAX

    View Slide

  55. MESOS

    View Slide

  56. GOAL

    View Slide

  57. View Slide

  58. CONTACT
    Miguel Ángel García
    @magmax9
    http://magmax.org
    [email protected]

    View Slide