Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa 2014

Have you ever wrote code that worked when you wrote it, but failed when it went to QA? Or staging? Or production? Learning how to build a high quality development environment is a serious undertaking and will involve a lot of trial and error — unless you come to this talk. Having spent the better part of the last year prepping, implementing, rolling out, maintaining, stabilizing, and growing a large scale development environment platform, I will show you the ropes. The real secret to building a system everyone can love: it’s all about making eggs, not bacon.

Find out more at: http://jmather.com/talks/2014-02-26/cookbook/

Transcript

1. Much more Eggs than Bacon 

   A Development Environment

   Jacob Mather
   Software Engineer, Mashery
   Cookbook

   

   View Slide

2. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
   Who am I?
   What I do:
   • Software Engineer at Mashery
   • Co-organizer of the

   San Francisco PHP User Group
   • Community Evangelist
   Where I can be found:
   • Blog: http://jmather.com
   • Twitter: @thejmather
   

   View Slide

3. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
   Why I am giving this talk
   

   View Slide

4. Let’s talk about

   why we need environments
   

   View Slide

5. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
   What environments could we have?
   • Production
   • Where people paying money interact
   • Staging
   • Test your production deployment plan
   • Quality Assurance
   • Cheaper version of staging to do comprehensive testing
   • Development
   • Designed for easy debugging
   

   View Slide

6. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
   Production
   Server
   Server
   Server
   Load
   Balancer
   Service 1
   Server
   Server
   Server
   Load
   Balancer
   Service 2
   Server
   Server
   Server
   Load
   Balancer
   Web Portal
   DB
   Cache
   

   View Slide

7. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
   Staging
   Server
   Server
   Load
   Balancer
   Service 1
   Server
   Server
   Load
   Balancer
   Service 2
   Server
   Server
   Load
   Balancer
   Web Portal
   DB
   Cache
   

   View Slide

8. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
   QA
   Server
   Service 1
   Server
   Service 2
   Server
   Web Portal
   DB
   Cache
   

   View Slide

9. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
   Development
   Server
   Service 1
   Service 2
   DB
   Cache
   Web Portal
   

   View Slide

10. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Why are these differences important?
    

    View Slide

11. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development
    

    View Slide

12. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA
    

    View Slide

13. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA Staging
    

    View Slide

14. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA Staging Production
    

    View Slide

15. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    What happens when they are too different?
    

    View Slide

16. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development
    

    View Slide

17. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA
    

    View Slide

18. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA
    

    View Slide

19. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA
    

    View Slide

20. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA Staging
    

    View Slide

21. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA Staging
    

    View Slide

22. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA Staging
    

    View Slide

23. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA Staging Production
    

    View Slide

24. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA Staging Production
    

    View Slide

25. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Lifecycle of Code
    Development QA Staging Production
    

    View Slide

26. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Why where you develop matters
    • Environment based bugs are costly to track down
    • If you can't reproduce in dev, you are shooting in the
    dark
    • Devs need to share code without slowing down
    • Makes infrastructure more transparent
    

    View Slide

27. Let’s talk about

    getting ready for change
    

    View Slide

28. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Devise your minimum viable stack
    

    View Slide

29. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Server configuration automation
    • If you have no current automation
    • Chef
    • Puppet
    • Ansible
    • If you do have automation
    • Ops will need to find and remove all production secrets
    • Ops will need to add a way for you to customize
    

    View Slide

30. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Code cleanup
    • Force all configuration to be explicit
    • Not based on path or host name
    • Remove magic strings
    

    View Slide

31. Let’s talk about

    your choices when implementing
    

    View Slide

32. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Choose your platform
    • Traditional VM
    • VirtualBox (version 4.2.16 — trust me.)
    • VMWare
    • Parallels
    • Cloud
    • AWS
    • Digital Ocean
    • Rackspace
    • Container VM (Docker)
    

    View Slide

33. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Verification
    • Smoke tests
    • Manual inspection
    • Functional tests
    

    View Slide

34. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Points to consider
    • CLI interface
    • Learn from my mistake
    • Use un-routed domains for virtual hosts
    • Test with all internet disabled
    

    View Slide

35. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    At this point you should be able to build a
    fully functional environment that you are
    capable of developing software in.
    

    View Slide

36. Let’s talk about

    getting in to the hands of users
    

    View Slide

37. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Make or break time
    • Prepare as much documentation as you can stomach
    • Prepare for lots of confusion and frustration
    

    View Slide

38. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Choose your timing carefully
    You only get one chance at a first impression.
    !
    The first time a user spins up their new environment, it must
    work flawlessly.
    !
    The entire rest of their experience will be influenced by this
    very first interaction.
    

    View Slide

39. Let’s talk about

    what happens after deployment
    

    View Slide

40. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    One thing we all have in common...
    Everyone loves to customize their computer!
    !
    Someone will use CDPATH which makes cd output text.
    !
    A few will use rvm and not appreciate your automated (and
    incorrect) help installing gems.
    !
    Nobody likes you fiddling with their computer.
    

    View Slide

41. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Start building a troubleshooting document
    The less time you have to spend debugging other people’s
    environments, the more time you have to rapidly iterate.
    !
    No only do you need to have them, they need to be
    accessible, and make sense, and have enough context.
    

    View Slide

42. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Document how to tinker with the system
    The sooner you can get others involved with adding features,
    the faster you will be able to keep up with the ever changing
    needs of your new development environment.
    !
    It also lets others try things experimentally, without involving
    you at all!
    

    View Slide

43. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Tips for making improvements
    • Slow and steady wins the race.
    • Do not change your environment’s interface.
    • Except when there’s great value in it.
    • Even then it’s really hard to justify.
    

    View Slide

44. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    48 Minutes
    

    View Slide

45. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    30 Minutes
    

    View Slide

46. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    22 Minutes
    

    View Slide

47. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Replicating production bugs in development
    

    View Slide

48. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Make reporting problems easy .
    easy
    

    View Slide

49. Let’s talk about

    chasing the myth of stability
    

    View Slide

50. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    The march of progress will continue
    • Ops updates will break your dev environments.
    • You will find new third parties your builds depend on.
    • 100 scratch environment builds a week will make 1 in 100
    events happen once a week.
    • You’ll start to have others contribute infrastructure changes
    to meet their project’s needs.
    

    View Slide

51. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Finding your environment’s dependencies
    • Your environment depends on:
    • Your own bootstrapping code
    • Ops automation code
    • Random third party software Ops is installing
    • … anything else?
    

    View Slide

52. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Managing your environment’s dependencies
    • Time to build a CI process
    • You’ll need a desktop to test accurately
    • Watch for changes from all relevant repos
    • Remember it is pairs of commits that are tested
    

    View Slide

53. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Making a quick recovery
    Check to see if your platform of choice supports snapshots.
    !
    When environments go sideways, they can be hard to restore.
    !
    Snapshots eliminate troubleshooting, and take ~20 seconds to
    run a restore.
    

    View Slide

54. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    “I just updated trunk and EVERYTHING IS BROKEN!”
    

    View Slide

55. Let’s talk about

    handing off to a team
    

    View Slide

56. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Eventually you will need to scale, but…
    • There’s a lot of domain specific knowledge in your head.
    • Do you know how to run a team?
    • Do you know how to lead a team?
    • Do you know how to be a product owner?
    

    View Slide

57. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Plus, letting go is hard.
    

    View Slide

58. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    So…
    

    View Slide

59. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    this is cool and all but…
    

    View Slide

60. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    what’s that stuff about eggs and bacon?
    

    View Slide

61. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    The chicken and the pig.
    

    View Slide

62. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Cool story bro, but what are you really trying to say?
    

    View Slide

63. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    !
    We are hiring at
    

    View Slide

64. Much more Eggs than Bacon: A Development Environment Cookbook - Code PaLOUsa - February 26th, 2014
    Thank you!
    More information about this talk can be found at:
    !
    http://bit.ly/clp14-cookbook
    !
    RATE MY TALK AT:
    http://spkr8.com/t/29431
    !
    PLEASE LEAVE FEEDBACK
    It’s the only way I can improve
    

    View Slide