Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On our CI and Builds

On our CI and Builds

Presentation to internal Engineering team on exactly how our builds and CI work.

Nathan Kleyn

March 03, 2015
Tweet

More Decks by Nathan Kleyn

Other Decks in Programming

Transcript

  1. On our
    CI and BUILDS
    HOW THEY WORK

    View full-size slide

  2. Or: How shit gets built and
    deployed.

    View full-size slide

  3. Builds involve:

    View full-size slide

  4. GitHub ∙ CircleCI
    Ruby ∙ Thor
    Docker

    View full-size slide

  5. And copious amounts of:

    View full-size slide

  6. Swearing ∙ Fear
    Punching ∙ Hate
    Quiet Weeping

    View full-size slide

  7. On the
    PROCESS
    OF BUILDING A COMMIT

    View full-size slide

  8. The steps to build:

    View full-size slide

  9. 1. Make a commit.

    View full-size slide

  10. 2. Push to GitHub.

    View full-size slide

  11. 3. CircleCI is notified.

    View full-size slide

  12. 4. The build begins.

    View full-size slide

  13. 5. The Docker build begins.

    View full-size slide

  14. 6. The sanity build begins.
    (if master)

    View full-size slide

  15. 7. Deployment can begin.

    View full-size slide

  16. On the
    PUSHING
    OF A COMMIT

    View full-size slide

  17. GitHub receives your commit
    and dispatches a notification
    to CircleCI.

    View full-size slide

  18. Cheers bro.
    Welcome blud.
    all of teh infos

    View full-size slide

  19. CircleCI receives a payload of
    information telling it what
    commit this was.

    View full-size slide

  20. CircleCI starts a container up
    for the build.

    View full-size slide

  21. Boom bitches.

    View full-size slide

  22. It then looks for our
    circle.yml file.

    View full-size slide

  23. Each step of this file is
    executed.

    View full-size slide

  24. These steps execute a bunch
    of shell scripts in /circle.

    View full-size slide

  25. These steps...
    ∙ Configure versions of software to use (Ruby,
    Java, Kafka, HBase, etc).
    ∙ Restore the cache.
    ∙ Checkout the repository.
    ∙ Build all the projects.
    ∙ Test all the projects.
    ∙ Save all the artifacts to CircleCI.

    View full-size slide

  26. We have three different
    kinds of builds:

    View full-size slide

  27. Standard ∙ Docker
    Sanity

    View full-size slide

  28. We tell it what scripts to use
    by passing a MODE env var.

    View full-size slide

  29. Demo: The CircleCI build
    scripts.

    View full-size slide

  30. On the
    DOCKER BUILDS
    OUR CODE IN CONTAINERS

    View full-size slide

  31. Quick Docker refresher:

    View full-size slide

  32. Docker is built on the idea of
    containers.

    View full-size slide

  33. Many container!

    View full-size slide

  34. You build images.
    You deploy containers.

    View full-size slide

  35. How it happens:

    View full-size slide

  36. A second parameterised build
    is started for Docker.

    View full-size slide

  37. These builds have whatever
    info they need passed as
    parameters.

    View full-size slide

  38. It builds, tags and pushes the
    Docker images.

    View full-size slide

  39. The building happens on a
    separate machine.

    View full-size slide

  40. The pushed images are
    stored in the Docker registry.

    View full-size slide

  41. When we deploy, we pull
    down these images and run
    them (making containers!)

    View full-size slide

  42. On
    BUILD SCRIPTS
    USING AND ABUSING THOR

    View full-size slide

  43. Thor is a task framework.

    View full-size slide

  44. I’m actually really
    sensitive and kind.

    View full-size slide

  45. All of the things we do on
    CircleCI are Thor tasks.

    View full-size slide

  46. The Thor task sources are in
    /build-scripts/thor.

    View full-size slide

  47. Each task is namespaced.
    thor foo:bar

    View full-size slide

  48. You can get help for a task by
    running thor help task.

    View full-size slide

  49. All Thor tasks are just Ruby
    files. Thor is just a nice DSL.

    View full-size slide

  50. Demo: The Thor build
    scripts.

    View full-size slide

  51. On
    COMMON PROBLEMS
    SOLVING COMMON ISSUES

    View full-size slide

  52. Q: Gradle tasks hanging
    locally.

    View full-size slide

  53. A: Start the Nexus server.

    View full-size slide

  54. Q: Deployments fail as unable
    to pull from repository.

    View full-size slide

  55. A: Start the Docker registry.

    View full-size slide

  56. Q: Docker build fails with “no
    space left on device”.

    View full-size slide

  57. A: Restart the Docker server,
    if it still fails call for help.

    View full-size slide

  58. Q: “Expected ID but got nil”.

    View full-size slide

  59. A: (Sadly) Just retry the
    build. This is a
    Docker/library bug that
    happens randomly.

    View full-size slide

  60. Q: The tests are failing.

    View full-size slide

  61. A: Your tests are shit. Fix
    your tests.

    View full-size slide

  62. On
    BEING A BUILD EXPERT
    BECOME A ZOMG HACKER

    View full-size slide

  63. Demo: How to find the build
    to deploy.

    View full-size slide

  64. Demo: How to see what SHA
    is being built for Docker.

    View full-size slide

  65. Fin.
    I’m gonna eat your hairy legs.

    View full-size slide