.co.il www. Continuous Integration, Continuous Deployment & Delivery 

.co.il www. backlog of finished features not in production daily build ⇒ monthly release 

.co.il www. Deploy code vs. Release feature dev deadline == deploy date == release date it doesn’t have to be this way … 

Continuous Delivery A software development discipline where you build software in such a way that the software can be released to production at any time. -- Martin Fowler Code Build Test Deploy Release Monitor Idea Customer 

Continuous Delivery Reduce the cost, time and risk of delivering incremental changes to users. -- Jez Humble 

Continuous Integration Tools Jenkins CI / Bamboo / TeamCity CI-aaS - Travis CI / Circle CI / … magnum-ci.com / semaphoreapp.com / codeship.io / drone.io / solanolabs.com / shiningpanda-ci.com hosted-ci.com / fazend.com / appharbor.com / cloudbees.com /clinkerhq.com 

Continuous Integration / CD ○ 1997 Extreme Programming extremeprogramming.org/rules/integrateoften.html ○ 2000 Martin Fowler martinfowler.com/articles/continuousIntegration.html ○ 2007 Paul Duvall (CI book) ○ 2010 Jez Humble (CD book) 

.co.il www. “Between 12:00 AM and 11:59 PM on April 25, 2013, Quora released new versions of the site 46 times. This was a normal day for us.” - Quora / April 2013 “Deployment every 11.6 seconds, 1,079 max in one hour. 10,000 mean number of hosts per deployment, with 30,000 maximum” - Amazon.com / November 2012 “On the Google Consumer Surveys team, 8 minutes after you commit code it's live in production.” - Google / June 2014 “10+ deploys per day.” - John Allspaw / June 2009 

.co.il www. How long would it take your organization to deploy a change that involved just one single line of code? Do you do this on a repeatable, reliable basis? Continuous Delivery slideshare.net/jezhumble/adopting-continuous-delivery continuousdelivery.com 

.co.il www. Esty.com is an online marketplace for handmade and vintage items. Etsy.com employs 500+ people and has grown to a community of more than 25 million members and 1,000,000+ online shops who generated 1.35$ billion of sales in 2013. extfiles.etsy.com/progress-report/2013-Etsy-Progress-Report.pdf Deploying etsy.com 40+ times every day by a team of 175+ committers supporting 60+ million unique monthly visitors. youtube.com/watch?v=4pVOXLvoUyE infoq.com/news/2014/03/etsy-deploy-50-times-a-day slideshare.net/beamrider9/continuous-deployment-at-etsy-a-tale-of-two-approaches etsy.com 

.co.il www. Business 250+ million active customer accounts. 1+ million active retail partners worldwide. Between 100-150 services are accessed to build a page. fool.com/investing/general/2014/05/24/how-many-customers-does-amazon-have. aspx Deployment 1,079 max deploys per hour affecting 30,000 hosts simultaneous receiving a deployment. youtube.com/watch?v=dxk8b9rSKOo amazon.com 

.co.il www. Netflix.com 36 milliion members, more than one billion hours of video per month. http://www.infoq.com/presentations/netflix-architecture-resilience netflix.com 

.co.il www. SAP.com FlowCon flowcon.org youtube.com/channel/UCMk1sRo1hnTLMA3kpn6BVKg SAP - youtube.com/watch?v=NJJdPlcCYK4 Before: monthly releases, 2 week qa cycle, late night incidents, 12 month lead time → After: bi-weekly release, <1 day qa, single-button rollback, 1 week lead time 

.co.il www. Business HP software sells multiple categories of software, serving 94 of Fortune 100 companies. Third largest company in software revenue in the world. en.wikipedia.org/wiki/HP_Software_Division Deployment 1000+ builds per day, 825,000+ change requests in a year, 100,000+ tests run daily. h30614.www3.hp.com/collateral/Barcelona2014/presentations/B5756.pdf hp.com 

.co.il www. Devops Enterprise Summit devopsenterprise.io Huawei CD - youtube.com/watch?v=G_coTcbj1FU Before: 30 days per feature delivery, 300min prod build, 1440min full test → After: 7 days per feature delivery, 10min prod build, 360min full test Ticketmaster - youtube.com/watch?v=01gfNOwhnro Before: Bi-monthyl/Quarterly releases → After: 109 deployments in September Target.com - youtube.com/watch?v=exrjV9V9vhY 30+ APIs, 80 deployments per week, <10 incidents per month 

.co.il www. lead time for changes release frequency time to restore service change fail rate How long is the delay between a request for a change, and a production system operating with that change implemented? How long does it take for an abnormal behavior in the system to be restored to the normal standard agreed way of operation? How many changes and features are being released to production in a fixed period of time? How often the system fails or service is disrupted? 

.co.il www. continuous integration Continuous integration usually consists of ● compiling ● running unit tests ● deployable asset ● pushing asset to repository repeated in a loop trigged by each commit 

.co.il www. Visualise atlassian.com/wallboards/information-radiators.jsp 

Stoplight 

No content

No content

.co.il www. 

.co.il www. 

.co.il www. 

.co.il www. Change Management 

.co.il www. 

.co.il www. 1950 1998 

.co.il www. 1. identify 2. store 3. change 4. report 

.co.il www. servers (vms) / storage operating system application servers application code + dependencies application configuration network monitoring 

.co.il www. 

.co.il www. 

.co.il www. belongs to app code or to cmdb? ○ packages & versions ○ mail servers ○ data centers ○ database users & passwords ○ connection strings ○ integration urls ○ deployments ○ batch jobs 

.co.il www. avoid regressions 

.co.il www. avoid duplicate effort 

.co.il www. avoid disaster, have recovery 

.co.il www. Configuration Management 

.co.il www. 

.co.il www. Automation infoq.com/presentations/Infrastructure-as-Code #!/bin/sh NODES="webserver database" for n in $NODES do ssh $n complex-script.sh done 

No content

zeroturnaround.com/labs/rebel-labs-release-it-ops-devops-productivity-report-2013 

○ open source ○ manages server config. ○ cookbooks & recipes ○ ruby dsl OpsCode Chef 

redis cookbook cookbooks/redis/recipes/default.rb package "redis-server" do action :upgrade end service "redis-server" do action :nothing supports status: true, restart: true end template "/etc/redis/redis.conf" do source "redis.conf.erb" owner "root" mode "0644" variables({ bind: node[:redis][:bind] }) notifies :restart, "service[redis-server]" end 1. cookbook/redis/attributes/default.rb 2. solo.json ... chef 

cookbooks/redis/attributes/default.rb default[:redis][:bind] = "127.0.0.1" cookbooks/redis/templates/default/redis.conf.erb # If you want you can bind a single interface, if the bind option is not # specified all the interfaces will listen for incoming connections. bind <%= @bind %> ... template "/etc/redis/redis.conf" do ... variables({ bind: node[:redis][:bind] }) end redis cookbook ... chef 

running chef-solo $ chef-solo -c solo.rb -j solo.json ... solo.rb cookbook_path "cookbooks" solo.json { "run_list": "recipe[redis]" } ... chef 

$ sudo chef-solo -c solo.rb -j solo.json Starting Chef Client, version 11.4.0 Compiling Cookbooks... Converging 3 resources Recipe: redis::default * package[redis-server] action upgrade - upgrade package redis-server from uninstalled to 2:2.2.12-1build1 * service[redis-server] action nothing (up to date) * template[/etc/redis/redis.conf] action create (up to date) Chef Client finished, 1 resources updated First run installs redis, creates configuration file. ... chef 

Second time, everything is already up to date. Idempotency! ƒ[ƒ(x)] ≡ ƒ(x) $ sudo chef-solo -c solo.rb -j solo.json Starting Chef Client, version 11.4.0 Compiling Cookbooks... Converging 3 resources Recipe: redis::default * package[redis-server] action upgrade (up to date) * service[redis-server] action nothing (up to date) * template[/etc/redis/redis.conf] action create (up to date) Chef Client finished, 0 resources updated ... chef 

making a small change to JSON parameters passed to Chef run. running chef-solo + modification solo.json { "redis": { "bind": "0.0.0.0" }, "run_list": "recipe[redis]" } ... chef 

$ sudo chef-solo -c solo.rb -j solo.json Starting Chef Client, version 11.4.0 Compiling Cookbooks... Converging 3 resources Recipe: redis::default * package[redis-server] action upgrade (up to date) * service[redis-server] action nothing (up to date) * template[/etc/redis/redis.conf] action create - update template[/etc/redis/redis.conf] from 81b4f1 to 8a6cec --- /etc/redis/redis.conf 2011-07-27 17:26:50.000000000 +0000 +++ /tmp/chef-rendered-template20130406-2537-f8vlv6 2013-04-06 ... @@ -27,7 +27,7 @@ # If you want you can bind a single interface, if the bind option is not # specified all the interfaces will listen for incoming connections. # -bind 127.0.0.1 +bind 0.0.0.0 # Specify the path for the unix socket that will be used to listen for # incoming connections. There is no default, so Redis will not listen * service[redis-server] action restart - restart service service[redis-server] Chef Client finished, 2 resources updated install package run service configuration + restart ... chef 

.co.il www. 

.co.il www. We invite you to join Operations Israel Facebook group on on.fb.me/Ops-IL we are hiring at [email protected] Thank you! www.devops.co.il 

.co.il www. Reference: - http://sysadvent.blogspot.co.il/2011/12/day-19-why-use-configuration-management.html - https://www.youtube.com/watch?v=2a2nWELIk-Y Yegor Bugayenko | Kyiv DevOps, September, 2015 - https://twitter.com/thoughtworks/status/658636712709480448 - 20th Century BIG IDEAS in Operations Management 1920s: Ford & Taylor - Moving production line and standardized work 1930s: Shewhart (& Deming) - Statistical Control of Quality 1960s: Ohno - Lean Production System 1980s: Goldratt & Kaplan - Measurement & Theory of Constraints 1990s: Hammer & Dell - Reengineering / Process Focus & “Direct Model”