On our CI and Builds

Presentation to internal Engineering team on exactly how our builds and CI work.

Transcript

1. On our
   CI and BUILDS
   HOW THEY WORK
   

   View Slide

2. Or: How shit gets built and
   deployed.
   

   View Slide

3. Builds involve:
   

   View Slide

4. GitHub ∙ CircleCI
   Ruby ∙ Thor
   Docker
   

   View Slide

5. And copious amounts of:
   

   View Slide

6. Swearing ∙ Fear
   Punching ∙ Hate
   Quiet Weeping
   

   View Slide

7. View Slide

8. On the
   PROCESS
   OF BUILDING A COMMIT
   

   View Slide

9. The steps to build:
   

   View Slide

10. 1. Make a commit.
    

    View Slide

11. 2. Push to GitHub.
    

    View Slide

12. 3. CircleCI is notified.
    

    View Slide

13. 4. The build begins.
    

    View Slide

14. 5. The Docker build begins.
    

    View Slide

15. 6. The sanity build begins.
    (if master)
    

    View Slide

16. 7. Deployment can begin.
    

    View Slide

17. On the
    PUSHING
    OF A COMMIT
    

    View Slide

18. GitHub receives your commit
    and dispatches a notification
    to CircleCI.
    

    View Slide

19. Cheers bro.
    Welcome blud.
    all of teh infos
    

    View Slide

20. CircleCI receives a payload of
    information telling it what
    commit this was.
    

    View Slide

21. CircleCI starts a container up
    for the build.
    

    View Slide

22. Boom bitches.
    

    View Slide

23. It then looks for our
    circle.yml file.
    

    View Slide

24. Each step of this file is
    executed.
    

    View Slide

25. These steps execute a bunch
    of shell scripts in /circle.
    

    View Slide

26. These steps...
    ∙ Configure versions of software to use (Ruby,
    Java, Kafka, HBase, etc).
    ∙ Restore the cache.
    ∙ Checkout the repository.
    ∙ Build all the projects.
    ∙ Test all the projects.
    ∙ Save all the artifacts to CircleCI.
    

    View Slide

27. We have three different
    kinds of builds:
    

    View Slide

28. Standard ∙ Docker
    Sanity
    

    View Slide

29. We tell it what scripts to use
    by passing a MODE env var.
    

    View Slide

30. Demo: The CircleCI build
    scripts.
    

    View Slide

31. On the
    DOCKER BUILDS
    OUR CODE IN CONTAINERS
    

    View Slide

32. Quick Docker refresher:
    

    View Slide

33. Docker is built on the idea of
    containers.
    

    View Slide

34. Many container!
    

    View Slide

35. You build images.
    You deploy containers.
    

    View Slide

36. How it happens:
    

    View Slide

37. A second parameterised build
    is started for Docker.
    

    View Slide

38. These builds have whatever
    info they need passed as
    parameters.
    

    View Slide

39. It builds, tags and pushes the
    Docker images.
    

    View Slide

40. The building happens on a
    separate machine.
    

    View Slide

41. The pushed images are
    stored in the Docker registry.
    

    View Slide

42. When we deploy, we pull
    down these images and run
    them (making containers!)
    

    View Slide

43. On
    BUILD SCRIPTS
    USING AND ABUSING THOR
    

    View Slide

44. Thor is a task framework.
    

    View Slide

45. I’m actually really
    sensitive and kind.
    

    View Slide

46. All of the things we do on
    CircleCI are Thor tasks.
    

    View Slide

47. The Thor task sources are in
    /build-scripts/thor.
    

    View Slide

48. Each task is namespaced.
    thor foo:bar
    

    View Slide

49. You can get help for a task by
    running thor help task.
    

    View Slide

50. All Thor tasks are just Ruby
    files. Thor is just a nice DSL.
    

    View Slide

51. Demo: The Thor build
    scripts.
    

    View Slide

52. On
    COMMON PROBLEMS
    SOLVING COMMON ISSUES
    

    View Slide

53. Q: Gradle tasks hanging
    locally.
    

    View Slide

54. A: Start the Nexus server.
    

    View Slide

55. Q: Deployments fail as unable
    to pull from repository.
    

    View Slide

56. A: Start the Docker registry.
    

    View Slide

57. Q: Docker build fails with “no
    space left on device”.
    

    View Slide

58. A: Restart the Docker server,
    if it still fails call for help.
    

    View Slide

59. Q: “Expected ID but got nil”.
    

    View Slide

60. A: (Sadly) Just retry the
    build. This is a
    Docker/library bug that
    happens randomly.
    

    View Slide

61. Q: The tests are failing.
    

    View Slide

62. A: Your tests are shit. Fix
    your tests.
    

    View Slide

63. On
    BEING A BUILD EXPERT
    BECOME A ZOMG HACKER
    

    View Slide

64. Demo: How to find the build
    to deploy.
    

    View Slide

65. Demo: How to see what SHA
    is being built for Docker.
    

    View Slide

66. Fin.
    I’m gonna eat your hairy legs.
    

    View Slide