Tweet
Tweet

Slide 1

Slide 1 text

Tom Santero @tsantero October 18, 2012 - ATL DevOps Meetup Tools, Testing and Disasters Basho at Friday, October 19, 12

Slide 2

Slide 2 text

$ whoami Tom Santero @tsantero on Twitter [email protected] I <3 open source software, beer and kitty cats github.com/tsantero Friday, October 19, 12

Slide 3

Slide 3 text

http://ricon2012.com videos will be posted soon! Friday, October 19, 12

Slide 4

Slide 4 text

Riak Design Goals high-availability low-latency horizontal scalability fault tolerance ops friendliness predictability Friday, October 19, 12

Slide 5

Slide 5 text

Riak Design Goals high-availability low-latency horizontal scalability fault tolerance ops friendliness predictability Friday, October 19, 12

Slide 6

Slide 6 text

Riak Design Goals high-availability low-latency horizontal scalability fault tolerance ops friendliness predictability Friday, October 19, 12

Slide 7

Slide 7 text

Riak Design Goals high-availability low-latency horizontal scalability fault tolerance ops friendliness predictability Friday, October 19, 12

Slide 8

Slide 8 text

Riak Design Goals high-availability low-latency horizontal scalability fault tolerance ops friendliness predictability Friday, October 19, 12

Slide 9

Slide 9 text

Riak Design Goals high-availability low-latency horizontal scalability fault tolerance ops friendliness predictability Friday, October 19, 12

Slide 10

Slide 10 text

Riak Design Goals high-availability low-latency horizontal scalability fault tolerance ops friendliness predictability Friday, October 19, 12

Slide 11

Slide 11 text

Riak Design Goals high-availability low-latency horizontal scalability fault tolerance ops friendliness predictability Friday, October 19, 12

Slide 12

Slide 12 text

KEY CONCEPTS Friday, October 19, 12

Slide 13

Slide 13 text

key/value store key/value pairs mapped to buckets values are opaque objects stored as binaries on disk key value key value key value key value bucket Friday, October 19, 12

Slide 14

Slide 14 text

masterless a deployed as a cluster of nodes e b d c any node can serve any request Friday, October 19, 12

Slide 15

Slide 15 text

masterless a deployed as a cluster of nodes request e b d c any node can serve any request Friday, October 19, 12

Slide 16

Slide 16 text

masterless a deployed as a cluster of nodes e b d c any node can serve any request Friday, October 19, 12

Slide 17

Slide 17 text

masterless a deployed as a cluster of nodes e b d c any node can serve any request payload Friday, October 19, 12

Slide 18

Slide 18 text

THE HOOD UNDER Friday, October 19, 12

Slide 19

Slide 19 text

consistent hashing replicas virtual nodes (vnodes) handoff gossip protocols anti-entropy REPLICATION and ADMINISTRATION Friday, October 19, 12

Slide 20

Slide 20 text

THE RING DIVIDED EVENLY INTO 160-BIT INT KEYSPACE PARTITIONS Friday, October 19, 12

Slide 21

Slide 21 text

VNODES PARTITIONS node 0 node 1 node 2 CLAIM Friday, October 19, 12

Slide 22

Slide 22 text

VNODES PARTITIONS node 0 node 1 node 2 Friday, October 19, 12

Slide 23

Slide 23 text

REBALANCE VNODES PARTITIONS node 0 node 1 node 2 node 3 + Friday, October 19, 12

Slide 24

Slide 24 text

DEMO (can someone sacri!ce a few bits to the demo gods?) Friday, October 19, 12

Slide 25

Slide 25 text

Quorum requests N R W PR/PW DR/DW Friday, October 19, 12

Slide 26

Slide 26 text

consistent hashing node 0 node 1 node 2 node 3 hash(“meetups/DevOpsATL”) N = 3 Friday, October 19, 12

Slide 27

Slide 27 text

consistent hashing node 0 node 1 node 2 node 3 hash(“meetups/DevOpsATL”) N = 3 Friday, October 19, 12

Slide 28

Slide 28 text

disaster scenario node 0 node 1 node 3 node 2 Friday, October 19, 12

Slide 29

Slide 29 text

disaster scenario node 0 node 1 node 3 node 2 Friday, October 19, 12

Slide 30

Slide 30 text

disaster scenario node 0 node 1 node 3 node 2 requests go to fallback Friday, October 19, 12

Slide 31

Slide 31 text

disaster scenario node 0 node 1 node 3 node 2 Friday, October 19, 12

Slide 32

Slide 32 text

disaster scenario node 0 node 1 node 3 node 2 node comes back online Friday, October 19, 12

Slide 33

Slide 33 text

disaster scenario node 0 node 1 node 3 node 2 Friday, October 19, 12

Slide 34

Slide 34 text

disaster scenario node 0 node 1 node 3 node 2 normal operations resume Friday, October 19, 12

Slide 35

Slide 35 text

vector clocks establish temporality Friday, October 19, 12

Slide 36

Slide 36 text

vector clocks establish temporality Friday, October 19, 12

Slide 37

Slide 37 text

access your programmatically DATA with Erlang Ruby Python Java Python Haskell Perl Go Clojure OCaml Node.JS Dart ...your language of choice Friday, October 19, 12

Slide 38

Slide 38 text

GET PUT DELETE CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD Friday, October 19, 12

Slide 39

Slide 39 text

map/reduce Friday, October 19, 12

Slide 40

Slide 40 text

store and index documents Riak Search JSON / XML plain text english prose query against VALUES Friday, October 19, 12

Slide 41

Slide 41 text

2i Secondary Indexes Riak Object Riak Object Riak Object X-Riak-Index-email_bin X-Riak-Index-email_bin X-Riak-Index-email_bin “[email protected]” “[email protected]” “[email protected]” “[email protected]” Friday, October 19, 12

Slide 42

Slide 42 text

2i Secondary Indexes Riak Object Riak Object Riak Object tag objects X-Riak-Index-email_bin X-Riak-Index-email_bin X-Riak-Index-email_bin “[email protected]” “[email protected]” “[email protected]” “[email protected]” Friday, October 19, 12

Slide 43

Slide 43 text

pluggable backends Friday, October 19, 12

Slide 44

Slide 44 text

Bitcask write ahead log in-memory LUT append-only !les Friday, October 19, 12

Slide 45

Slide 45 text

LevelDB persistent ordered map generational sstables append-only !les snappy unbounded Friday, October 19, 12

Slide 46

Slide 46 text

Memory Friday, October 19, 12

Slide 47

Slide 47 text

Multi-backend use all the backends! Friday, October 19, 12

Slide 48

Slide 48 text

Ask 10 engineers what they think DevOps is and you’ll get 12.79234 di"erent answers Friday, October 19, 12

Slide 49

Slide 49 text

Internal Customers Product basho DevOps at Basho Friday, October 19, 12

Slide 50

Slide 50 text

Internal programming and utilities automation improve test suite by 500% Friday, October 19, 12

Slide 51

Slide 51 text

guttersnipe middleware in Vagrant deploy and run test suite no need to rebuild clusters 500% increase in e"ciency Friday, October 19, 12

Slide 52

Slide 52 text

https://github.com/basho/riak-chef-cookbook CHEF COOKBOOKS use to deploy riak open source over 40 commits in 2012 rolling upgrades, iptables, etc... future development Friday, October 19, 12

Slide 53

Slide 53 text

Erlang Template Helper https://github.com/basho/erlang_template_helper con!g_to_json json_to_con!g specify Erlang con!g !les very useful with Chef Friday, October 19, 12

Slide 54

Slide 54 text

PUPPET MODULES https://github.com/basho/puppet-riak use to deploy riak just open sourced I don’t know puppet, so I’m not going to bullshit you Friday, October 19, 12

Slide 55

Slide 55 text

PUPPET MODULES https://github.com/basho/puppet-riak use to deploy riak just open sourced I don’t know puppet, so I’m not going to bullshit you Friday, October 19, 12

Slide 56

Slide 56 text

RIAK TEST https://github.com/basho/riak_test utility for testing builds run tests in a sandbox reset to clean state after tests easy to bootstrap envs also open source! Friday, October 19, 12

Slide 57

Slide 57 text

GiddyUp https://github.com/basho/giddyup visual scorecard for Riak Test seed Riak Test with list of tests to be run on each platform receive test results and logs via REST interface Friday, October 19, 12

Slide 58

Slide 58 text

Friday, October 19, 12

Slide 59

Slide 59 text

Investing in Technology Friday, October 19, 12

Slide 60

Slide 60 text

DataCenter 300 cpu cores 1 TB RAM 500 TB storage gigabit internet Friday, October 19, 12

Slide 61

Slide 61 text

DataCenter 300 cpu cores 1 TB RAM 500 TB storage gigabit internet Friday, October 19, 12

Slide 62

Slide 62 text

Product reduce the overall cost of engineering enhance dat dev cycle Friday, October 19, 12

Slide 63

Slide 63 text

Example #1 pushing the limits with socket performance 4gbps over 8 erl sockets Friday, October 19, 12

Slide 64

Slide 64 text

Example #2 General Performance Testing and performing full-sync DC replication at scale Friday, October 19, 12

Slide 65

Slide 65 text

testing repl at SCALE Friday, October 19, 12

Slide 66

Slide 66 text

Customers replicate issues support spec + test hardware Friday, October 19, 12

Slide 67

Slide 67 text

Example #3 replicating customer issues with replication Friday, October 19, 12

Slide 68

Slide 68 text

“In both cases I was unable to replicate these issues on the hardware available to me locally, both through limited machines (2) and major performance di!erences between nodes ("rst gen mbp vs i7 dell tower)” --Andrew Thompson, Basho Sr. Engineer Friday, October 19, 12

Slide 69

Slide 69 text

test harness Friday, October 19, 12

Slide 70

Slide 70 text

!n hugs! Any and all questions can be sent to /dev/null Friday, October 19, 12