The top five questions to ask about NoSQL. JONATHAN ELLIS at Big Data Spain 2012

Session presented at Big Data Spain 2012 Conference
16th Nov 2012
ETSI Telecomunicacion UPM Madrid
www.bigdataspain.org
More info: http://www.bigdataspain.org/es-2012/conference/top-five-questions-about-nosql/jonathan-ellis

Transcript

1. Five questions
   for your NoSQL solution!
   Jonathan Ellis
   CTO, DataStax
   Project Chair, Apache Cassandra
   

   View Slide

2. ©2012 DataStax
   how do I
   model
   my application?
   

   View Slide

3. ©2012 DataStax
   Popular options
   • Key/value
   • Tabular
   • Document
   • Graph?
   

   View Slide

4. ©2012 DataStax
   Schema is your friend
   {
   "id": "e451dd42-ece3-11e1-a0a3-34159e154f4c",
   "name": "jbellis",
   "state": "TX",
   "birthdate": "1/1/1976",
   "email_addresses": ["jbellis@gmail", "jbellis@datastax.com"],
   }
   

   View Slide

5. ©2012 DataStax
   SQL can be your friend too
   CREATE TABLE users (
   id uuid PRIMARY KEY,
   name text,
   state text,
   birth_date date
   );
   CREATE INDEX ON users(state);
   SELECT * FROM users
   WHERE state=‘Texas’ AND birth_date > ‘1950-01-01’;
   

   View Slide

6. ©2012 DataStax
   CREATE TABLE users (
   id uuid PRIMARY KEY,
   name text,
   state text,
   birth_date date
   );
   CREATE TABLE users_addresses (
   user_id uuid REFERENCES users,
   email text
   );
   SELECT *
   FROM users NATURAL JOIN users_addresses;
   Collections
   

   View Slide

7. ©2012 DataStax
   CREATE TABLE users (
   id uuid PRIMARY KEY,
   name text,
   state text,
   birth_date date
   );
   CREATE TABLE users_addresses (
   user_id uuid REFERENCES users,
   email text
   );
   SELECT *
   FROM users NATURAL JOIN users_addresses;
   Collections
   X
   

   View Slide

8. ©2012 DataStax
   CREATE TABLE users (
   id uuid PRIMARY KEY,
   name text,
   state text,
   birth_date date,
   email_addresses set
   );
   UPDATE users
   SET email_addresses = email_addresses + {‘jbellis@gmail.com’,
   ‘jbellis@datastax.com’};
   Collections
   

   View Slide

9. ©2012 DataStax
   Joins don’t scale
   • No joins
   • No subqueries
   • No aggregation functions* or GROUP BY
   • ORDER BY?
   

   View Slide

10. ©2012 DataStax
    SELECT * FROM tweets
    WHERE user_id IN (SELECT follower FROM followers
    WHERE user_id = ’driftx’)
    followers
    ?
    tweets
    

    View Slide

11. ©2012 DataStax
    CREATE TABLE timeline (
    user_id uuid,
    tweet_id timeuuid,
    tweet_author uuid,
    tweet_body text,
    PRIMARY KEY (user_id,
    tweet_id)
    );
    Clustering in Cassandra
    user_id tweet_id _author _body
    jbellis 3290f9da.. rbranson lorem
    jbellis 3895411a.. tjake ipsum
    ... ... ...
    driftx 3290f9da.. rbranson lorem
    driftx 71b46a84.. yzhang dolor
    ... ... ...
    yukim 3290f9da.. rbranson lorem
    yukim e451dd42.. tjake amet
    ... ... ...
    

    View Slide

12. ©2012 DataStax
    CREATE TABLE timeline (
    user_id uuid,
    tweet_id timeuuid,
    tweet_author uuid,
    tweet_body text,
    PRIMARY KEY (user_id,
    tweet_id)
    );
    Clustering in Cassandra
    user_id tweet_id _author _body
    jbellis 3290f9da.. rbranson lorem
    jbellis 3895411a.. tjake ipsum
    ... ... ...
    driftx 3290f9da.. rbranson lorem
    driftx 71b46a84.. yzhang dolor
    ... ... ...
    yukim 3290f9da.. rbranson lorem
    yukim e451dd42.. tjake amet
    ... ... ...
    SELECT * FROM timeline
    WHERE user_id = ’driftx’;
    

    View Slide

13. ©2012 DataStax
    how does it
    perform?
    

    View Slide

14. ©2012 DataStax
    VLDB benchmark
    

    View Slide

15. ©2012 DataStax
    Locking
    

    View Slide

16. ©2012 DataStax
    Efficiency
    

    View Slide

17. ©2012 DataStax
    UPDATE users
    SET email_addresses = email_addresses + {...}
    WHERE user_id = ‘jbellis’;
    

    View Slide

18. ©2012 DataStax
    Durability
    

    View Slide

19. ©2012 DataStax
    Log-structured storage engine
    Memory
    Hard drive
    Memtable
    write( , )
    k1 c1:v1
    Commit log
    

    View Slide

20. ©2012 DataStax
    Memory
    Hard drive
    Memtable
    write( , )
    k1 c1:v1
    Commit log
    k1 c1:v1
    k1 c1:v1
    

    View Slide

21. ©2012 DataStax
    Memory
    Hard drive
    write( , )
    k1 c2:v2
    k1 c1:v1
    k1 c1:v1
    k1 c2:v2
    c2:v2
    

    View Slide

22. ©2012 DataStax
    Memory
    Hard drive
    k1 c1:v1
    k1 c1:v1
    k1 c2:v2
    c2:v2
    write( , )
    k2 c1:v1 c2:v2
    k2 c1:v1 c2:v2
    k2 c1:v1 c2:v2
    

    View Slide

23. ©2012 DataStax
    Memory
    Hard drive
    k1 c1:v1
    k1 c1:v4
    k1 c2:v2
    c2:v2
    write( , )
    k1 c1:v4 c3:v3
    k2 c1:v1 c2:v2
    k2 c1:v1 c2:v2
    k1 c1:v4 c3:v3
    c3:v3
    

    View Slide

24. ©2012 DataStax
    Memory
    Hard drive
    SSTable
    flush
    k1 c1:v4 c2:v2
    k2 c1:v1 c2:v2
    c3:v3
    index / BF
    cleanup
    

    View Slide

25. ©2012 DataStax
    No random writes
    

    View Slide

26. ©2012 DataStax
    The gory details
    

    View Slide

27. ©2012 DataStax
    Larger than memory datasets
    

    View Slide

28. ©2012 DataStax
    how does it handle
    failure?
    

    View Slide

29. ©2012 DataStax
    Classic partitioning with SPOF
    partition 1 partition 2 partition 3 partition 4
    router
    client
    

    View Slide

30. ©2012 DataStax
    Availability
    • “High availability implies that a single fault will
    not bring down your system. Not ‘we’ll recover
    quickly.’”
    -- Ben Coverston: DataStax
    • “The biggest problem with failover is that you're
    almost never using it until it really hurts. It's like
    backups that you never test.”
    -- Rick Branson: Instagram
    

    View Slide

31. ©2012 DataStax
    Fully distributed, no SPOF
    client
    p1
    p1
    p1
    p3
    p6
    

    View Slide

32. ©2012 DataStax
    Multiple datacenters
    

    View Slide

33. ©2012 DataStax
    

    View Slide

34. ©2012 DataStax
    Self-healing
    Client
    request
    Coordinator
    Replica
    internal
    request
    internal
    response
    response
    1
    2
    3
    4
    

    View Slide

35. ©2012 DataStax
    Self-healing
    Client
    request
    Coordinator
    Replica
    internal
    request
    internal
    response
    response
    1
    2
    3
    4
    

    View Slide

36. ©2012 DataStax
    Self-healing
    Client
    request
    Coordinator
    Replica
    internal
    request
    1
    2
    replica fails
    timeout
    response
    4
    

    View Slide

37. ©2012 DataStax
    Self-healing
    Client
    request
    Coordinator
    Replica
    internal
    request
    1
    2
    X
    replica fails
    timeout
    response
    4
    

    View Slide

38. ©2012 DataStax
    Self-healing
    Client
    request
    Coordinator
    Replica
    internal
    request
    1
    2
    4
    replica fails
    timeout
    response
    hint
    3
    

    View Slide

39. ©2012 DataStax
    Self-healing
    Client
    request
    Coordinator
    Replica
    internal
    request
    1
    2
    4
    X
    replica fails
    timeout
    response
    hint
    3
    

    View Slide

40. ©2012 DataStax
    Other healing modes
    • AntiEntropyService
    • Read repair
    

    View Slide

41. ©2012 DataStax
    Dynamic snitch
    (dealing with partial failure)
    Client Coordinator
    40% busy
    90% busy
    30% busy
    

    View Slide

42. ©2012 DataStax
    how does it
    scale?
    

    View Slide

43. ©2012 DataStax
    VLDB benchmark
    

    View Slide

44. ©2012 DataStax
    Scaling antipatterns
    • Metadata servers
    • Router bottlenecks
    • Overloading existing nodes when adding
    capacity
    

    View Slide

45. ©2012 DataStax
    how
    flexible
    is it?
    

    View Slide

46. ©2012 DataStax
    

    View Slide

47. ©2012 DataStax
    Data model: Realtime
    Portfolios
    StockHist
    stock last
    GOOG $95.52
    AAPL $186.10
    AMZN $112.98
    LiveStocks
    stock date price
    GOOG 2011-01-01 $8.23
    GOOG 2011-01-02 $6.14
    GOOG 2011-001-03 $7.78
    user stock shares
    jbellis GOOG 80
    jbellis LNKD 20
    yukim AMZN 100
    

    View Slide

48. ©2012 DataStax
    Data model: Analytics
    worst_date loss
    2011-07-23 -$34.81
    2011-03-11 -$11432.24
    2011-05-21 -$1476.93
    Portfolio1
    HistLoss
    Portfolio2
    Portfolio3
    

    View Slide

49. ©2012 DataStax
    Data model: Analytics
    stock rdate return
    GOOG 2011-07-25 $8.23
    GOOG 2011-07-24 $6.14
    GOOG 2011-07-23 $7.78
    AAPL 2011-07-25 $15.32
    AAPL 2011-07-24 $12.68
    10dayreturns
    INSERT OVERWRITE TABLE 10dayreturns
    SELECT a.stock,
    b.date as rdate,
    b.price - a.price
    FROM StockHist a
    JOIN StockHist b
    ON (a.stock = b.stock
    AND date_add(a.date, 10) = b.date);
    

    View Slide

50. ©2012 DataStax
    Data model: Analytics
    portfolio rdate preturn
    Portfolio1 2011-07-25 $118.21
    Portfolio1 2011-07-24 $60.78
    Portfolio1 2011-07-23 -$34.81
    Portfolio2 2011-07-25 $2143.92
    Portfolio3 2011-07-24 -$10.19
    portfolio_returns
    INSERT OVERWRITE TABLE portfolio_returns
    SELECT portfolio,
    rdate,
    SUM(b.return)
    FROM portfolios a JOIN 10dayreturns b
    ON (a.stock = b.stock)
    GROUP BY portfolio, rdate;
    

    View Slide

51. ©2012 DataStax
    Data model: Analytics
    INSERT OVERWRITE TABLE HistLoss
    SELECT a.portfolio, rdate, minp
    FROM (
    SELECT portfolio, min(preturn) as minp
    FROM portfolio_returns
    GROUP BY portfolio
    ) a
    JOIN portfolio_returns b
    ON (a.portfolio = b.portfolio and a.minp = b.preturn);
    worst_date loss
    2011-07-23 -$34.81
    2011-03-11 -$11432.24
    2011-05-21 -$1476.93
    Portfolio1
    HistLoss
    Portfolio2
    Portfolio3
    

    View Slide

52. ©2012 DataStax
    

    View Slide

53. ©2012 DataStax
    Some Cassandra users
    

    View Slide

54. Questions?
    • http://www.flickr.com/photos/26817893@N05/2573006312/
    • http://www.flickr.com/photos/rowanbank/7686239548
    • http://www.flickr.com/photos/mervtheswerve/6081933265
    • http://www.flickr.com/photos/dg_pics/2526208830
    • http://www.flickr.com/photos/wainwright/351684037
    • http://www.flickr.com/photos/mikeneilson/1606662529
    • http://www.flickr.com/photos/sbisson/3852905534
    • http://www.flickr.com/photos/breadnbadger/2674928517
    Image credits
    

    View Slide