Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On our CI and Builds

On our CI and Builds

Presentation to internal Engineering team on exactly how our builds and CI work.

Nathan Kleyn

March 03, 2015
Tweet

More Decks by Nathan Kleyn

Other Decks in Programming

Transcript

  1. On our
    CI and BUILDS
    HOW THEY WORK

    View Slide

  2. Or: How shit gets built and
    deployed.

    View Slide

  3. Builds involve:

    View Slide

  4. GitHub ∙ CircleCI
    Ruby ∙ Thor
    Docker

    View Slide

  5. And copious amounts of:

    View Slide

  6. Swearing ∙ Fear
    Punching ∙ Hate
    Quiet Weeping

    View Slide

  7. View Slide

  8. On the
    PROCESS
    OF BUILDING A COMMIT

    View Slide

  9. The steps to build:

    View Slide

  10. 1. Make a commit.

    View Slide

  11. 2. Push to GitHub.

    View Slide

  12. 3. CircleCI is notified.

    View Slide

  13. 4. The build begins.

    View Slide

  14. 5. The Docker build begins.

    View Slide

  15. 6. The sanity build begins.
    (if master)

    View Slide

  16. 7. Deployment can begin.

    View Slide

  17. On the
    PUSHING
    OF A COMMIT

    View Slide

  18. GitHub receives your commit
    and dispatches a notification
    to CircleCI.

    View Slide

  19. Cheers bro.
    Welcome blud.
    all of teh infos

    View Slide

  20. CircleCI receives a payload of
    information telling it what
    commit this was.

    View Slide

  21. CircleCI starts a container up
    for the build.

    View Slide

  22. Boom bitches.

    View Slide

  23. It then looks for our
    circle.yml file.

    View Slide

  24. Each step of this file is
    executed.

    View Slide

  25. These steps execute a bunch
    of shell scripts in /circle.

    View Slide

  26. These steps...
    ∙ Configure versions of software to use (Ruby,
    Java, Kafka, HBase, etc).
    ∙ Restore the cache.
    ∙ Checkout the repository.
    ∙ Build all the projects.
    ∙ Test all the projects.
    ∙ Save all the artifacts to CircleCI.

    View Slide

  27. We have three different
    kinds of builds:

    View Slide

  28. Standard ∙ Docker
    Sanity

    View Slide

  29. We tell it what scripts to use
    by passing a MODE env var.

    View Slide

  30. Demo: The CircleCI build
    scripts.

    View Slide

  31. On the
    DOCKER BUILDS
    OUR CODE IN CONTAINERS

    View Slide

  32. Quick Docker refresher:

    View Slide

  33. Docker is built on the idea of
    containers.

    View Slide

  34. Many container!

    View Slide

  35. You build images.
    You deploy containers.

    View Slide

  36. How it happens:

    View Slide

  37. A second parameterised build
    is started for Docker.

    View Slide

  38. These builds have whatever
    info they need passed as
    parameters.

    View Slide

  39. It builds, tags and pushes the
    Docker images.

    View Slide

  40. The building happens on a
    separate machine.

    View Slide

  41. The pushed images are
    stored in the Docker registry.

    View Slide

  42. When we deploy, we pull
    down these images and run
    them (making containers!)

    View Slide

  43. On
    BUILD SCRIPTS
    USING AND ABUSING THOR

    View Slide

  44. Thor is a task framework.

    View Slide

  45. I’m actually really
    sensitive and kind.

    View Slide

  46. All of the things we do on
    CircleCI are Thor tasks.

    View Slide

  47. The Thor task sources are in
    /build-scripts/thor.

    View Slide

  48. Each task is namespaced.
    thor foo:bar

    View Slide

  49. You can get help for a task by
    running thor help task.

    View Slide

  50. All Thor tasks are just Ruby
    files. Thor is just a nice DSL.

    View Slide

  51. Demo: The Thor build
    scripts.

    View Slide

  52. On
    COMMON PROBLEMS
    SOLVING COMMON ISSUES

    View Slide

  53. Q: Gradle tasks hanging
    locally.

    View Slide

  54. A: Start the Nexus server.

    View Slide

  55. Q: Deployments fail as unable
    to pull from repository.

    View Slide

  56. A: Start the Docker registry.

    View Slide

  57. Q: Docker build fails with “no
    space left on device”.

    View Slide

  58. A: Restart the Docker server,
    if it still fails call for help.

    View Slide

  59. Q: “Expected ID but got nil”.

    View Slide

  60. A: (Sadly) Just retry the
    build. This is a
    Docker/library bug that
    happens randomly.

    View Slide

  61. Q: The tests are failing.

    View Slide

  62. A: Your tests are shit. Fix
    your tests.

    View Slide

  63. On
    BEING A BUILD EXPERT
    BECOME A ZOMG HACKER

    View Slide

  64. Demo: How to find the build
    to deploy.

    View Slide

  65. Demo: How to see what SHA
    is being built for Docker.

    View Slide

  66. Fin.
    I’m gonna eat your hairy legs.

    View Slide