Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling Humans: BigPanda's Fabulous ChatOps Adventure - DevOpsDays Tel Aviv 2016 Talk

Erik Zaadi
December 13, 2016

Scaling Humans: BigPanda's Fabulous ChatOps Adventure - DevOpsDays Tel Aviv 2016 Talk

So you heard of ChatOps right? Ever pondered to take the leap? We'll discuss how BigPanda went from boring deploying manually using ssh to boss level smooth Slack EPICNESS deploying process, How it improved our operations workflows, and finally how it became easy and fun for our developers to improve our operations

Erik Zaadi

December 13, 2016
Tweet

More Decks by Erik Zaadi

Other Decks in Technology

Transcript

  1. • Continuous Delivery • Deploy by ssh to deploy server

    (AWS) • Ran Ansible on our playbooks repository A TALE OF DEPLOYMENT EVOLUTION FLASHBACK TO EARLY 2014 8 deployers 5 Microservices 15 Servers
  2. • Synchronizing - Lack of visibility • Not reliable deploys

    • Disconnections • Debugging • Ansible was not common knowledge WE SIMPLY WANTED TO DEPLOY PROBLEMS:
  3. Bash wrapper for Ansible NEXT STEP: LET'S MAKE IT EASY

    ENTER BASH - DEPLOY.SH DEPLOY.SH IS COMING
  4. • Wrapped SSH login with TMUX • Used local users

    with symlinked playbooks repository • Enabled mosh NEXT STEP: LET'S STAY CONNECTED TMUX TO THE RESCUE TMUX IS COMING
  5. Solved: • Not reliable deploys • Disconnections (session) • Debugging

    Problems: • Synchronization TMUX AND LOCAL USERS
  6. • HipChat bot using Hubot • Written in CoffeeScript •

    Called Beanbot • Limited to staging servers • Used Jenkins to run Ansible • Could query past deploys NEXT STEP: CHATOPS TO THE RESCUE! HELLO THERE HUBOT!
  7. Problems: • Synchronization • Lack of HipChat usage • Almost

    no visibility • Didn’t solve the production pain • Only 3 contributors • People really hated Coffeescript BEANBOT
  8. Slack channel: #deployment-locks We'd manually notify locking. Escalated quickly to

    be unusable. Stupid Humans. NEXT STEP: LET'S SOLVE IT IN CHAT SO, CHAT-NOT-OPS?
  9. • Hackathon rewrite from scratch • Robin (Hubot) • Written

    in Javascript • Using micro services standards • Tests • Deploys NEXT STEP: CHATOPS TO THE RESCUE AGAIN! FIX THE STUPID HUMANS
  10. • Managed deployment locks • Deployed to production (!) •

    Prevented Human Errors • Similar syntax to bash script ZOMG TIS’S DA THINGZ THE HYPE TRAIN TOOK OFF
  11. • Managed using GitHub issues • 10 contributors (~1/2 RND)

    • Javascript vs Coffeescript • Slack vs HipChat • Hubot Scripts • Use anything with an API INSTANT POPULARITY GAINED WHY ROBIN WORKED
  12. • Integrated part of our dev cycle 30 Deployers 30

    Microservices 160 Servers ~100 daily deploys and locks TODAY WE CAN’T LIVE WITHOUT ROBIN
  13. • Ensure deploy can work without ChatOps as well •

    Reuse existing deploy process and syntax • Run the bot both locally and on a staging env • Make contributing to the bot fun and easy • Company bot skeletons (allowing quick POCs) THINGS WE LEARNED THE HARD WAY YOUR BOT WILL BE DOWN
  14. • SuperPanda - Groceries wish list • Gordon - Sales

    and Support • Bellboy - Opens our office door • Splinter - Code linting rewarder DIVIDED THEY CONQUER MOAARR BOTS!!!1
  15. EOF