Slide 1

Slide 1 text

Building a global startup David Gardner, Chief Architect at Hailo #CassandraSummit

Slide 2

Slide 2 text

CASSANDRASUMMIT2014

Slide 3

Slide 3 text

CASSANDRASUMMIT2014

Slide 4

Slide 4 text

CASSANDRASUMMIT2014

Slide 5

Slide 5 text

CASSANDRASUMMIT2014

Slide 6

Slide 6 text

CASSANDRASUMMIT2014 Factors driving C* adoption Resilience Geographical replication Cover future growth

Slide 7

Slide 7 text

CASSANDRASUMMIT2014 London Launch North America Launch Atlanta Launch Introduction of C* November 2011 September 2012 March 2014 Ireland Migration August C* only cities launched MySQL primary storage C* adoption timeline

Slide 8

Slide 8 text

CASSANDRASUMMIT2014 Starting a startup

Slide 9

Slide 9 text

CASSANDRASUMMIT2014

Slide 10

Slide 10 text

CASSANDRASUMMIT2014 Introduction of C* C* only cities launched MySQL primary storage From MySQL to Cassandra

Slide 11

Slide 11 text

PHP Cust API eu-west-1 PHP Driver API Java Hailo Engine MySQL PHP Driver API PHP Driver API PHP Cust API PHP Cust API Load Balancer Load Balancer MySQL Redis H1

Slide 12

Slide 12 text

PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 H1.5

Slide 13

Slide 13 text

us-east-1 C* C* C* eu-west-1 ELB Go “Thin” API RabbitMQ Message Bus (federated clusters per AZ) Go Service Go Service Java Service C* C* C* ELB Go “Thin” API RabbitMQ Message Bus (federated clusters per AZ) Go Service Go Service Java Service H2

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

CASSANDRASUMMIT2014 •  15,000 jobs/hour •  3,000 req/sec •  20,000 drivers on shift Testing at Hailo

Slide 16

Slide 16 text

CASSANDRASUMMIT2014 Economy Premium us-east-1 eu-west-1 us-east-1 eu-west-1 C* 2.0.8 i2.xlarge 8GB heap 800GB storage (1 SSD ephemeral) C* 2.0.8 m1.4xlarge 8GB heap 1.7TB storage (4x400GB ephemeral)

Slide 17

Slide 17 text

CASSANDRASUMMIT2014 Premium •  Read heavy (10:1) •  Avoid issues caused by compactions Economy •  Write heavy (10:1) •  Secondary use cases

Slide 18

Slide 18 text

CASSANDRASUMMIT2014 CASSANDRA-4430

Slide 19

Slide 19 text

CASSANDRASUMMIT2014 Entity storage Basic CRUD for passengers, drivers, jobs Time series Logs of actions during jobs, logins Search For management portals Analytics Counting events

Slide 20

Slide 20 text

CASSANDRASUMMIT2014 1/ Entity storage

Slide 21

Slide 21 text

CASSANDRASUMMIT2014

Slide 22

Slide 22 text

CASSANDRASUMMIT2014 H1 $sql = "INSERT INTO `customers` (`email`, `password`, `created`, `status`) VALUES ('$e', '$p', '$t', '$s')"; #php #mysql * parameter sanitisation snipped

Slide 23

Slide 23 text

CASSANDRASUMMIT2014 Functionality •  IDs generated by MySQL auto-increment •  Index constraints (phone, email) checked by unique MySQL indexes automatically •  ACID consistency, familiar H1 #php #mysql

Slide 24

Slide 24 text

77669839702851584 41 bits Timestamp millisecond precision, bespoke epoch 10 bits Configured machine ID 12 bits Sequence number CASSANDRASUMMIT2014 H2 ID Generation using Snowflake derivative #golang #cassandra

Slide 25

Slide 25 text

lock, err := lockUser(user) defer lock.Unlock() if err != nil { return errors.ISE(..) } CASSANDRASUMMIT2014 H2 Index checking using flawed ZK/C* combo #golang #cassandra

Slide 26

Slide 26 text

CASSANDRASUMMIT2014 Ideal solution: C* CAS with CQL INSERT INTO users (email, name) VALUES ('[email protected]', 'Jane Doe’) IF NOT EXISTS

Slide 27

Slide 27 text

CASSANDRASUMMIT2014 H1 $sql = "UPDATE `customers` SET `email`='$e' WHERE id = '1234'"; #php #mysql * parameter sanitisation snipped

Slide 28

Slide 28 text

CASSANDRASUMMIT2014 CQL or Thrift the same rule applies •  Update individual columns that equate to user actions •  Example: set auto tip should mutate one column H2 #cassandra

Slide 29

Slide 29 text

CASSANDRASUMMIT2014 2/ Time series

Slide 30

Slide 30 text

CASSANDRASUMMIT2014

Slide 31

Slide 31 text

CASSANDRASUMMIT2014 H1 $sql = "SELECT * FROM `quotes` WHERE `customer`='$c’ ORDER BY `timestamp` DESC LIMIT $o,$n"; #php #mysql * parameter sanitisation snipped

Slide 32

Slide 32 text

CASSANDRASUMMIT2014 H2 iter := ts.ReversedIterator( start, end, lastId, customerId ) #cassandra

Slide 33

Slide 33 text

CASSANDRASUMMIT2014 H2 for iter.Next() { job := &Job{} if err := iter.Item().Unmarshal(job); err != nil { return nil, "”, err } jobs = append(jobs, job) if len(jobs) >= count { break } } return jobs, iter.Last(), nil #cassandra

Slide 34

Slide 34 text

CASSANDRASUMMIT2014 Time series library •  Buckets by configurable row time range •  Supports an index for sparse datasets to track which time ranges exist (can iterate all) H2 #cassandra

Slide 35

Slide 35 text

CASSANDRASUMMIT2014 Ideal solution: CQL CREATE TABLE temperature ( weatherstation_id text, event_time timestamp, temperature text, PRIMARY KEY (weatherstation_id,event_time) );

Slide 36

Slide 36 text

CASSANDRASUMMIT2014 Ideal solution: CQL •  Better data model (discrete columns) •  Lacks row time partitioning •  Ideally would still wrap in a nice library

Slide 37

Slide 37 text

CASSANDRASUMMIT2014 3/ Search

Slide 38

Slide 38 text

CASSANDRASUMMIT2014

Slide 39

Slide 39 text

CASSANDRASUMMIT2014 Functionality •  Search to power admin portals for company management •  Ideally not on the critical path for getting people Hailos to where they want to be

Slide 40

Slide 40 text

CASSANDRASUMMIT2014 H1 $sql = "SELECT * FROM customers WHERE ( firstname LIKE '$keywords%’ OR lastname LIKE '$keywords%’ OR email LIKE '$keywords%’ OR phone LIKE '%$keywords%')" #php #mysql * parameter sanitisation snipped

Slide 41

Slide 41 text

CASSANDRASUMMIT2014 H2 #elasticsearch Customer Search Service Search Service Event Federation Service us-east-1 Regional Elastic Search Customer Search Service Search Service Event Federation Service eu-west-1 Regional Elastic Search

Slide 42

Slide 42 text

CASSANDRASUMMIT2014 Implementation •  Primary data store is C* •  Microservice listens to changes and asynchronously indexes documents •  Documents federated between regions to independent Elastic Search clusters

Slide 43

Slide 43 text

CASSANDRASUMMIT2014 4/ Analytics

Slide 44

Slide 44 text

CASSANDRASUMMIT2014

Slide 45

Slide 45 text

H1 SELECT COUNT(*) as numJobs, SUM(jobs.fare + jobs.tip) as totalFare, AVG(jobs.cleared - jobs.created) AS avgJobTimeInSeconds, SUM(jobs.cleared - jobs.created) AS timePOBInSeconds, MAX(jobs.distance) AS distanceLongestJobInMetres, (SELECT pickup_sector FROM ( SELECT COUNT(*) as numPickups, pickup_sector FROM jobs WHERE jobs.cleared >= $from AND jobs.cleared < $to AND jobs.driver=$driverId AND pickup_sector IS NOT NULL AND pickup_sector != '' GROUP BY pickup_sector ORDER BY numPickups DESC LIMIT 1 ) AS cpp) as mostFrequentPickupSector FROM jobs WHERE jobs.cleared >= $from AND jobs.cleared < $to AND jobs.driver = $driverId #php #mysql #truestory #omg

Slide 46

Slide 46 text

CASSANDRASUMMIT2014 H2 #nsq #redshift #cassandra Event firehose (NSQ) Service Service Service Redshift loader service S3 archive service HOB activity service Generation Processing

Slide 47

Slide 47 text

{ “eventType”: “point”, “timestamp”: “123456789”, “driver”: “LON1234”, “lat”: “lng”: } for { get event from firehose add to map[time.Time]map[string]*HyperLogLog if batch full || max flush time { flush batch to C* and complete in NSQ } } ROW: metric name + time component COL: time component + random batch UID VAL: binary HLL data Raw events In memory batched rollup C* storage

Slide 48

Slide 48 text

CASSANDRASUMMIT2014 Counting distinct things with C* •  HyperLogLog data structure bounds memory and gives probabilistic answers •  Idempotent flush to C* plus global replication •  Can make lots of columns

Slide 49

Slide 49 text

CASSANDRASUMMIT2014 Ideal solution: split out analytics •  Turn things that happen into “events” •  Put events into NSQ or Kafka •  Use events in different ways to suit different use cases

Slide 50

Slide 50 text

CASSANDRASUMMIT2014

Slide 51

Slide 51 text

CASSANDRASUMMIT2014 Operational and organisational challenges

Slide 52

Slide 52 text

CASSANDRASUMMIT2014 2012 2013 2014 Zero downtime operational changes first cluster phpcassa v1.0.9 eu-west stats cluster astynax v1.1 us-east, eu-west expand into ap-northeast-1 split into premium/ economy clusters v2.0 us-east, eu-west deprecate stats cluster adopt Go and gossie client expand into us-east-1

Slide 53

Slide 53 text

CASSANDRASUMMIT2014 Meet the experts Julien Campan Cassandra Analyst Since the C* Summit 2013 we have added dedicated C* support Chris Hoolihan Infrastructure Architect

Slide 54

Slide 54 text

CASSANDRASUMMIT2014 Welp! My client has been abandoned

Slide 55

Slide 55 text

CASSANDRASUMMIT2014 Huzzah! @mattstump to the rescue!

Slide 56

Slide 56 text

CASSANDRASUMMIT2014 Welp! 16.6GB maximum row size

Slide 57

Slide 57 text

CASSANDRASUMMIT2014

Slide 58

Slide 58 text

Monolithic DB Monolithic app Monolithic team Distributed DB Distributed app Distributed team CASSANDRASUMMIT2014

Slide 59

Slide 59 text

CASSANDRASUMMIT2014 Wrapping up

Slide 60

Slide 60 text

CASSANDRASUMMIT2014

Slide 61

Slide 61 text

CASSANDRASUMMIT2014 Today, I would start with C* for Hailo •  Technology startup which has to be big to work (network effect) •  Experienced and well-funded founders with immediate global ambition •  C* 2.x with CQL and C*-experienced team

Slide 62

Slide 62 text

CASSANDRASUMMIT2014 Micro- services

Slide 63

Slide 63 text

CASSANDRASUMMIT2014 Thanks

Slide 64

Slide 64 text

CASSANDRASUMMIT2014 Image credits h"ps://www.flickr.com/photos/wrwetzel/7302103558/in/photostream/   h"ps://www.flickr.com/photos/w4nd3rl0st/12491208355   h"ps://www.flickr.com/photos/cdevers/5702488800   h"ps://www.flickr.com/photos/rh2ox/9990016123   h"ps://www.flickr.com/photos/davidnunn/11933033385   h"ps://www.flickr.com/photos/bre"morrison/4780958663   h"ps://www.flickr.com/photos/nathaninsandiego/2732195668