Node.js, the Universe and Everything

Greenfield projects are great, but how do you integrate Node.js with an existing stack?

This month Mark does a deep dive on what he and his team @ Hightail are doing to build, release, and integrate Node.js with a diverse infrastructure and a polyglot application stack.

Transcript

1. NODE.JS, THE UNIVERSE AND EVERYTHING
2. DISCLAIMERS
3. YOU ARE CRAZY AND NONE OF THIS WILL WORK FOR MY COMPANY
5. • CLOUD STORAGE, SHARING, AND COLLABORATION PROVIDER • FOUNDED IN 2004 • 45M CUSTOMERS • CUSTOMER BASE ACROSS 190+ COUNTRIES AND 98% OF FORTUNE 500 COMPANIES
6. HIGHTAIL STACK • JAVA • NODE.JS • GOLANG • PHP • NATIVE ANDROID/IOS/OSX/WINDOWS CLIENTS • OUTLOOK PLUGIN • CHROME EXTENSIONS
7. HIGHTAIL COULDN’T RELEASE SOFTWARE
8. HIGHTAIL IN Q2 2014 • 1-2 WEEKS OF DEV + ~1 MONTH CODE FREEZE • INFREQUENT RELEASES TO PROD • PROD RELEASES ALMOST ALWAYS ROLLED BACK • 30+ DAYS TO GET A DEVELOPMENT ENVIRONMENT
9. HIGHTAIL IN Q4 2014 • FREQUENT PROD DEPLOYMENTS (100+ NEW FEATURES IN PROD IN Q4) • ZERO FRICTION DEPLOYS VIA HIPCHAT • ON DEMAND DEVELOPMENT ENVIRONMENTS
10. HIGHTAIL 2015 ROADMAP • CONTINUOUS DELIVERY • COMPLETE OUR STACK MIGRATION • AUTOMATED CODE PROMOTION • AUTOMATED QA
11. PROCESS
12. SUCCESS == CUSTOMERS ARE USING THE CODE
13. CONSISTENCY
14. EMPOWERMENT VERIFICATION AVOIDANCE OF RITUAL
15. EMPOWERMENT
16. If you make it easy for someone to a good job, they’re going to do a good job.
17. DEVELOPERS
18. OPERATIONS
19. THE BUSINESS
20. WHERE DO WE START?
21. PACKAGE RELEASE ARTIFACTS
22. RELEASE ARTIFACTS • VERSIONED • IMMUTABLE • ACCESSIBLE • AUTHORITATIVE
23. NODE.JS ARTIFACTS • BUILD OUTPUT • NODE_MODULES/ • PACKAGE.JSON
24. GENERATING NODE.JS ARTIFACTS { "name": "my-app", "private": true, "version": "1.0.4", "description": "my app", "main": "server.js", "directories": { "test": "test" }, "scripts": { "start": "node server.js", "test": "npm run unit", "build" "gulp dist" "pkg": "npm run build \ && npm run test \ && tar zcvf my-app-$(node -e \"p = require('./package.json'); console.log(p.version)\").tar.gz package.json dist/ node_modules/" },
25. ARTIFACT REPOSITORY
27. GIT FLOW
28. CONTINUOUS INTEGRATION
30. NPM MODULES CHECKED INTO SRC, AND OTHER OPINIONS
31. CONFIGURATION MANAGEMENT
32. CHEF
33. MY-APP COOKBOOK • BERKSFILE • METADATA.RB • ATTRIBUTES • TEMPLATES • RECIPES • .KITCHEN.YML • TESTS
34. METADATA.RB pkginfo = JSON.parse(File.read(File.expand_path("../../package.json", __FILE__))) name 'my_app' maintainer 'Mark Wilkerson' maintainer_email 'mark.wilkerson@hightail.com' license 'All rights reserved' description 'Installs/Configures my-app' long_description 'Installs/Configures my-app' version pkginfo["version"] depends 'nodejs', '~> 2.1.0' depends 'github', '~> 0.3.0' depends 'libarchive', '~> 0.4' depends 'redisio', '~> 2.2.4' depends 'build-essential', '~> 2.1.2' depends 'newrelic', '~> 2.3.0'
35. BERKSFILE source "https://supermarket.getchef.com" metadata cookbook 'nodejs', '2.1.0' cookbook 'github', '0.3.0' cookbook 'libarchive', '0.4' cookbook 'redisio', '2.2.4' cookbook 'build-essential', '2.1.2' cookbook 'newrelic', '2.3.0'
36. ATTRIBUTES default["my_app"]["version"] = run_context.cookbook_collection["my_app"].metadata.version default["my_app"]["app_name"] = "my-app" default["my_app"]["user"] = "doncheadle" default["my_app"]["deploy_root"] = "/opt/#{node["my_app"]["app_name"]}" default["my_app"]["node_env"] = "development" default["my_app"]["repo"] = "hightail/my-app" default["my_app"]["artifact"] = “my-app-#{node["my_app"]["version"]}.tar.gz" default["my_app"]["logfile"] = "/var/log/#{node["my_app"]["app_name"]}.log" default["my_app"]["mq"] = "127.0.0.1" default["apt"]["compile_time_update"] = true default["build-essential"]["compile_time"] = true default["logging"]["sourceEnv"] = node["fqdn"] default["logging"]["key"] = "logstash:#{node["logging"]["sourceEnv"]}"
37. TEMPLATES # <%= @app_name %>.conf # This file is generated by Chef for <%= node[:fqdn] %> # Do not edit by hand description "<%= @app_name %>" expect fork start on started networking stop on runlevel [016] setuid <%= @user %> setgid <%= @user %> respawn respawn limit 5 30 limit nofile 1000000 1000000 console log # Send SIGINT instead of SIGTERM kill signal SIGINT # The default is 5 seconds, we may need a little longer kill timeout 20 pre-start script echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (upstart) Starting" end script pre-stop script echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (upstart) Stopping" end script script chdir <%= @path %> export DOMAIN=<%= @domain %> export LOGSTASH_REDIS=<%= @logstash_transport %> export LOGSTASH_KEY=<%= @logstash_key %> export NODE_PATH=<%= @path %> export NODE_ENV=<%= @node_env %> export REDIS_SERVER=<%= @mq %> export NEWRELIC_KEY=<%= @newrelic_key %> exec npm start end script
38. RECIPES
39. SETUP # Install node.js include_recipe "nodejs" # Create our user deploy_root = node["my_app"]["deploy_root"] username = node["my_app"]["user"] user username do shell "/bin/false" home deploy_root system true action :create end group username do action :modify members username append true end # newrelicify include_recipe 'newrelic::server_monitor_agent'
40. CONFIGURE unless Chef::Config[:solo] mq = search(:node, "chef_environment:" + node.chef_environment + " AND role:mq").first if mq node.normal["my_app"]["mq"] = mq[:ipaddress] end end # Stop service if running service app_name do provider Chef::Provider::Service::Upstart action [ :stop ] end # Generate new config from templates template "/etc/init/#{app_name}.conf" do source "nodejs.upstart.erb" owner "root" group "root" mode "0600" variables({ :path => deploy_link, :app_name => app_name, :node_env => node["my_app"]["node_env"], :m => node["my_app"]["mq"], :user => username, :domain => node["domain_name"], :template_path => template_link, :logstash_transport => node["logging"]["host"], :logstash_key => node["logging"]["key"], :newrelic_key => node["newrelic"]["license"] }) end # Start service with new config service app_name do provider Chef::Provider::Service::Upstart supports :status => true, :restart => true, :reload => true action [ :enable, :start ] subscribes :reload, "template[/etc/init/#{app_name}.conf]", :immediately end
41. DEPLOY include_recipe "libarchive::default" # Stop service if it is running service app_name do provider Chef::Provider::Service::Upstart action [ :stop ] end # Download artifact from GitHub asset = github_asset artifact do repo repo release version github_token node["github_token"] action :download end # Deploy libarchive_file artifact do path asset.asset_path extract_to deploy_path owner username group username action :extract only_if { ::File.exist?(asset.asset_path) } end # Update symlinks if ::File.exists?(deploy_link) link prvs_link do to ::File.readlink(deploy_link) end end link deploy_link do to deploy_path end # Configure include_recipe "my_app::configure"
42. .KITCHEN.YML --- driver: name: vagrant provisioner: name: chef_solo platforms: - name: ubuntu-14.04 suites: - name: default run_list: - recipe[apt] - recipe[build-essential] - recipe[redisio] - recipe[redisio::enable] - recipe[my_app::default] attributes: apt: compile_time_update: true build-essential: compile_time: true domain_name: "localhost" logging: host: "127.0.0.1" sourceEnv: "localhost"
43. TESTS require 'serverspec' set :backend, :exec describe "My App" do describe service("my-app") do it { should be_enabled }= it { should be_running } end describe file("/etc/init/my-app.conf") do it { should be_file } it { should be_owned_by 'root' } it { should be_mode 600 } end describe file("/opt/my-app") do it { should be_directory } end describe user('doncheadle') do it { should exist } it { should belong_to_group 'doncheadle' } it { should have_home_directory '/opt/my-app' } it { should have_login_shell '/bin/false' } end describe process("node") do its(:args) { should match /server/ } its(:user) { should match /doncheadle/ } end end
44. COOKBOOK ARTIFACTS
45. CODE PROMOTION
46. POLYGLOT ARCHITECTURE
47. NODE.JS PAAS OFFERINGS ARE TERRIBLE
48. DOCKER