Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hiring SREs may be literally impossible

Hiring SREs may be literally impossible

If we're gonna do this SRE thing, we need to find the right people to do it.

After a few recent discussions, it became clear just how much everyone—at large companies and small—is struggling to find those people.

You can barely get enough applicants in the door, and by the time you've run your interview process you're left making a handful of offers.

Hiring SREs from the outside world is a competitive, expensive game to play. So why focus so much on people outside your company? You've got potential SREs sat all around you!

In this talk, we'll set the scene with a little look at the realities of hiring SREs. We won't stay there for too long though, because that's not what's going to save us!

The bulk of the talk will be spent looking at ways to discover budding SREs in your organisation, how to nurture their interest, and how to coach them in a role that's new to them.

Chris Sinjakli

August 31, 2017
Tweet

More Decks by Chris Sinjakli

Other Decks in Programming

Transcript

  1. Hi

    View Slide

  2. @ChrisSinjo

    View Slide

  3. Hiring SREs
    @ChrisSinjo

    View Slide

  4. Hiring SREs may be
    @ChrisSinjo

    View Slide

  5. Hiring SREs may be
    literally
    @ChrisSinjo

    View Slide

  6. Hiring SREs may be
    literally
    impossible @ChrisSinjo

    View Slide

  7. An SRE

    View Slide

  8. GOCARDLESS

    View Slide

  9. An conference

    View Slide

  10. An friend

    View Slide

  11. An dumb comment

    View Slide

  12. –Some dumbass
    “Hiring SREs may be literally
    impossible”

    View Slide

  13. Except it’s
    literally
    true

    View Slide

  14. People at all kinds
    of companies

    View Slide

  15. People can’t hire
    quickly enough

    View Slide

  16. 1. The problem
    2. A solution

    View Slide

  17. What’s the
    problem?

    View Slide

  18. "Even by Google’s overall standards,
    SRE candidates are difficult to find and
    even harder to interview effectively."
    –SRE Book, Part IV: Management

    View Slide

  19. SRE is weird

    View Slide

  20. Full of breadth

    View Slide

  21. Databases
    Caches
    Queues
    Proxies
    Networking (L4, L7)
    DNS
    Operating systems
    Observability
    Provisioning
    Packaging & deployment

    View Slide

  22. etc

    View Slide

  23. https://jvns.ca/blog/so-you-want-to-be-a-wizard/

    View Slide

  24. Full of depth

    View Slide

  25. Knowledge
    Things

    View Slide

  26. Knowledge
    Things
    “That” incident

    View Slide

  27. Knowledge
    Things
    “That” incident
    Your primary database

    View Slide

  28. https://gocardless.com/blog/in-search-of-performance-how-we-shaved-200ms-off-
    every-post-request/

    View Slide

  29. On-call is
    daunting

    View Slide

  30. Knowledge
    Things
    “That” incident

    View Slide

  31. View Slide

  32. A new take
    on existing roles

    View Slide

  33. Sure, we have
    DevOps

    View Slide

  34. DevOps is
    an approach,
    not a role

    View Slide

  35. SRE is more
    specific

    View Slide

  36. How are we
    doing?

    View Slide

  37. 2017 so far

    View Slide

  38. 2017 so far
    160 applicants

    View Slide

  39. 2017 so far
    160 applicants
    45 VM challenge

    View Slide

  40. 2017 so far
    160 applicants
    45 VM challenge
    15 final round

    View Slide

  41. 2017 so far
    160 applicants
    45 VM challenge
    15 final round
    2 offers

    View Slide

  42. 2017 so far
    160 applicants
    45 VM challenge
    15 final round
    2 offers
    2 hires

    View Slide

  43. The odds are
    against us

    View Slide

  44. 1. The problem
    2. A solution

    View Slide

  45. Where else
    can we find
    SREs?

    View Slide

  46. View Slide

  47. Internal hiring
    is a thing

    View Slide

  48. Some of your devs
    would
    love
    the role

    View Slide

  49. Hi

    View Slide

  50. How do you
    do it?

    View Slide

  51. 3 parts

    View Slide

  52. Find

    View Slide

  53. Find
    Show

    View Slide

  54. Find
    Show
    Mentor

    View Slide

  55. Find
    Show
    Mentor

    View Slide

  56. 3main
    signals

    View Slide

  57. They jump at
    tricky problems

    View Slide

  58. View Slide

  59. They peel
    back
    the stack

    View Slide

  60. They peel
    back
    the stack
    Proxy
    App
    Database
    OS

    View Slide

  61. They show
    empathy when
    things break

    View Slide

  62. GoCardless Slack excerpt

    View Slide

  63. Real talk:
    People learn things

    View Slide

  64. Find
    Show
    Mentor

    View Slide

  65. Find
    Show
    Mentor

    View Slide

  66. Build an open
    operational
    culture

    View Slide

  67. GoCardless post-mortem excerpt

    View Slide

  68. GoCardless Slack excerpt

    View Slide

  69. “The weird
    team in the
    corner”

    View Slide

  70. Projects as
    well as ops

    View Slide

  71. 20 to 50%

    View Slide

  72. GoCardless project scoping excerpt

    View Slide

  73. “The weird
    team in the
    corner”

    View Slide

  74. The public
    stuff helps

    View Slide

  75. https://gocardless.com/blog/tagged/site-reliability-engineering

    View Slide

  76. It’s for your
    coworkers
    too

    View Slide

  77. Take the
    sting out of
    the pager

    View Slide

  78. Don’t support a
    horror show

    View Slide

  79. Post-mortem
    action items

    View Slide

  80. Bump memory limit

    Add connection limiter

    View Slide

  81. Don’t foster a
    hero culture

    View Slide

  82. View Slide

  83. View Slide

  84. Pay
    your damn on-call

    View Slide

  85. Find
    Show
    Mentor

    View Slide

  86. Find
    Show
    Mentor

    View Slide

  87. Part IV:
    Management

    View Slide

  88. Disaster simulation

    View Slide

  89. https://github.com/gocardless/our-postgresql-setup

    View Slide

  90. Disaster simulation
    Reverse engineering

    View Slide

  91. They peel
    back
    the stack

    View Slide

  92. Disaster simulation
    Reverse engineering
    Pager shadowing

    View Slide

  93. View Slide

  94. My favourite:
    guided tours

    View Slide

  95. Databases
    Caches
    Queues
    Proxies
    Networking (L4, L7)
    DNS
    Operating systems
    Observability
    Provisioning
    Packaging & deployment

    View Slide

  96. View Slide

  97. Value
    Tasks
    Task value
    Extremely scientific graph

    View Slide

  98. Tasks
    Task value
    Learning value
    Extremely scientific graph
    Value

    View Slide

  99. What they
    bring to you

    View Slide

  100. View Slide

  101. So how’s it
    going?

    View Slide

  102. 2017 so far
    160 applicants
    45 VM challenge
    15 final round
    2 offers
    2 hires

    View Slide

  103. Our team over time

    View Slide

  104. Our team over time
    8 people (former and current)

    View Slide

  105. Our team over time
    8 people (former and current)
    3 joined GC as devs

    View Slide

  106. Our team over time
    8 people (former and current)
    3 joined GC as devs
    2 dev → infra elsewhere

    View Slide

  107. Hi

    View Slide

  108. IT FREAKIN’
    WORKS

    View Slide

  109. Hiring SREs may be
    literally
    impossible @ChrisSinjo

    View Slide

  110. Hiring enough SREs may be
    literally
    impossible @ChrisSinjo

    View Slide

  111. Coach some
    new ones

    View Slide

  112. Mentoring people
    is awesome
    anyway

    View Slide

  113. Thank you
    $❤
    @ChrisSinjo
    @GoCardlessEng

    View Slide

  114. We’re hiring
    $❤
    @ChrisSinjo
    @GoCardlessEng

    View Slide

  115. Questions?
    $❤
    @ChrisSinjo
    @GoCardlessEng

    View Slide

  116. Image credits
    • Pager - https://www.flickr.com/photos/s_w_ellis/3878337144/
    • Seedling - https://www.flickr.com/photos/kmdoncaster/26122812592/
    • Sparkling heart emoji - https://github.com/mozilla/fxemoji/blob/
    9f68ca9c5bc51521f9ffe284e00ba8b7308e2c41/svgs/people/u1F496-
    sparklingheart.svg
    • Rubik’s cubes - https://www.flickr.com/photos/le-topographe/4840728643/

    View Slide

  117. Image credits
    • SRE Book - https://landing.google.com/sre/book.html
    • Tours start here - https://www.flickr.com/photos/mwichary/2210774000/
    • Trying To Grow - https://www.flickr.com/photos/garryknight/4423598916/

    View Slide