DevOps @ Basho

Slides from my talk at the NYC DevOps meetup on Aug 23, discussing Riak, and the nature of the DevOps team at Basho.

1. DevOps @ Basho NYC DevOps Meetup - August 23, 2012
2. $
3. whoami $
4. whoami $ Name: Tom Santero $ Title: Technical Evangelist Company: Basho Technologies Twitter: @tsantero
5. whoami $ Name: Tom Santero $ ./presentation Title: Technical Evangelist Company: Basho Technologies Twitter: @tsantero
6. for the next 45 minutes we’ll be discussing...
8. just kidding.
9. • Overview of Riak • Nature of DevOps @ Basho • What has Basho learned from having a DevOps team Tonight’s Agenda
11. need to know: • Written in Erlang/OTP • distributed, key/value store + extras • advanced query features • pre/post commit hooks • pluggable backend storage engines • open source (Apache v2.0)
13. key
14. key value
15. key value bucket
16. key value bucket key value key value key value
17. bucket/key
18. node node node node node
20. request quorums replicas fallbacks consistent hashing
21. Consistent Hashing
22. Consistent Hashing • 160-bit integer keyspace 0 2160/2 2160/4
23. Consistent Hashing • 160-bit integer keyspace • divided into !xed number of evenly-sized partitions 32 partitions 0 2160/2 2160/4
24. Consistent Hashing • 160-bit integer keyspace • divided into !xed number of evenly-sized partitions • partitions are claimed by nodes in the cluster 32 partitions node 0 node 1 node 2 node 3 0 2160/2 2160/4
25. Consistent Hashing • 160-bit integer keyspace • divided into !xed number of evenly-sized partitions • partitions are claimed by nodes in the cluster • replicas go to the N partitions following the key node 0 node 1 node 2 node 3
26. Consistent Hashing • 160-bit integer keyspace • divided into !xed number of evenly-sized partitions • partitions are claimed by nodes in the cluster • replicas go to the N partitions following the key node 0 node 1 node 2 node 3 hash(“meetups/nycdevops”) N=3
27. Disaster Scenario
28. Disaster Scenario • node fails X X X X X X X X
29. Disaster Scenario • node fails • requests go to fallback X X X X X X X X hash(“meetups/nycdevops”)
30. Disaster Scenario • node fails • requests go to fallback • node comes back hash(“meetups/nycdevops”)
31. Disaster Scenario • node fails • requests go to fallback • node comes back • “Hando"” - data returns to recovered node hash(“meetups/nycdevops”)
32. Disaster Scenario • node fails • requests go to fallback • node comes back • “Hando"” - data returns to recovered node • normal operations resume hash(“meetups/nycdevops”)
33. easy to scale
34. Erlang/OTP
35. pre/post commit hooks
36. Bitcask LevelDB Memory
37. full-text search secondary indexes map/reduce
38. DevOps @ Basho
39. DevOps Team • Brief History: • uno#cially created in Oct 2011 • o#cially recognized in June 2012 • pilot program (to be discussed)
40. Internal Customers Product
41. Internal
42. DevOps - Internal • mix of programming and utilities • automate all the things! • instrumental in improving Riak Test Suite • 1,000s of separate tests • various con!gs / platforms
43. guttersnipe
44. guttersnipe • middleware in Vagrant • allows us to: • deploy + run entire test suite • no need to rebuild entire clusters • 500% increase in e#ciency
45. Chef Cookbooks
46. Chef Cookbooks • open source: • $ git clone git@github.com:basho/riak-chef-cookbook.git • over 30 commits in 2012 • deploy riak • future: • simplify use • rolling upgrades, autocon!g, iptables
47. Erlang Template Helper • also open source: • $ git clone git@github.com:basho/erlang_template_helper.git • written by Dan Reverri • specify Erlang con!g and args !les in JSON
48. ETH Example $ irb >> require ‘erlang_template_helper’ => true >> args = Eth::Args.new({“-name” => “riak@127.0.0.1”, “-env” => {“ERL_MAX_PORTS” => 4096}}) => -name riak@127.0.0.1 -env ERL_MAX_PORTS 4096 >> puts args.pp -name riak@127.0.0.1 -env ERL_MAX_PORTS 4096 => nil
49. ETH Example $ .bin/config_to_json multi_backend.config -p { "riak_kv": { "storage_backend": "riak_kv_multi_backend", "multi_backend_default": "first_backend", "multi_backend": [ ["__tuple", "first_backend", "riak_kv_bitcask_backend", { "data_root": "__string_/var/lib/riak/bitcask"}], ["__tuple", "second_backend", "riak_kv_leveldb_backend", { "data_root": "__string_/var/lib/riak/leveldb"}] ] } }
50. Investing in Tech
53. DataCenter Specs • 300 cpu cores • 500 TB storage • ~1 TB RAM • gigabit internet connection
54. Product Engineering
55. DevOps - Product • Basho Engineers all have test boxes • Now DevOps delegates DC resources • Examples of Product Enhancements:
56. Example #1 pushing the limits with socket performance 4gbps over 8 erl sockets
57. Example #2 General Performance Testing and performing full-sync DC replication at scale
58. Testing repl at Scale • full-sync repl • bottleneck in how keys were being read • lexicographical vs disk-ordered • used the Boston datacenter to prototype !x • currently in review to be merged into Riak EE
59. Customer Service
60. DevOps - CliServ • replicate customer issues /w datacenter • secondary datacenter: • potential customers ship us hardware • spec out + provide best implementations
61. Example #3 replicating customer issues with replication
62. Realtime repl Throuput • provisioned 5 machines • constrained bandwidth and increase latency between clusters • but not among nodes in the same cluster (‘management’ network as WAN approx) • discovered + eliminated several bottlenecks • customer achieved 10x jump in performance
63. “In both cases I was unable to replicate these issues on the hardware available to me locally, both through limited machines (2) and major performance di!erences between nodes ("rst gen mbp vs i7 dell tower)” --Andrew Thompson, Basho Engineer
65. RiakCS • released March 27, 2012 • shipped RiakCS v1.1 on Tuesday • S3-compatible cloud storage • built on Riak • multi-tenancy, multi billing, etc...
66. Riak CS Large Object Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak Node Riak Node Riak Node Riak Node Riak Node
67. Riak CS Large Object Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak Node Riak Node Riak Node Riak Node Riak Node
68. Riak CS Large Object Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak Node Riak Node Riak Node Riak Node Riak Node 1mb 1mb 1mb 1mb
69. Riak CS Large Object Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak Node Riak Node Riak Node Riak Node Riak Node 1mb 1mb 1mb 1mb
70. Riak CS Large Object Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak CS Reporting API S3 API Riak Node Riak Node Riak Node Riak Node Riak Node 1mb 1mb 1mb 1mb
71. complete test harness for RiakCS
72. shameless plugs: • Basho is Hiring DevOps Engineers: • work remotely • hack on cool shit • help make Basho and Riak better • send CV to Sean Carey scarey@basho.com
73. http://ricon2012.com When and where? Wednesday, October 10 through Thursday, October 11 at the W Hotel in downtown San Francisco.
74. Questions?