Building a SaaS app on CouchDB

Transcript

1. BUILDING A SAAS APP ON A . CouchConf 2011

2. we’re in a movie theatre, i’m sorry you don’t get

   to watch a movie. let’s watch a trailer at least.

3. ALEXANDER LANG @langalex, @upstream_agile, @cobot_me

4. COUCH POTATO CouchDB Ruby Library used by cobot

5. SaaS

6. AGENDA • Cobot overview • What works well • Problems,

   Solutions • Q/A

7. COBOT OVERVIEW one main web app

8. COWORKING SPACE MANAGEMENT

9. Did you know that there is a global community of

   people dedicated to the values of Collaboration, Openness,Community, Accessibility, and Sustainability in their workplaces? It's called Coworking. And people seem to think it's swell. coworking.com our space co.up (co-up.de)

10. COWORKING SPACE? shared office, rent desks

11. MEMBERS PLANS

12. PAYMENTS INVOICING

13. BOOKING ATTENDANCE radius

14. ANALYTICS pretty charts

15. {“API”: true} api, some small apps on heroku, non-core features,

    customizations

16. how many people know rails?

17. COUCH POTATO Intro what does it do?

18. class User include CouchPotato::Persistence property :email property :password before_save :encrypt_password

    end

19. user = User.new email: ‘[email protected]’ user.valid? db = CouchPotato.database db.save

    user # true|false db separated from models, easier to test, separation of concerns

20. class User include CouchPotato::Persistence view :by_email, key: :email view :by_email_and_encrypted_password,

    map: <<-JS function(doc) { if(doc.ruby_class == ‘User’) { emit([doc.email, encrypt(doc.password)]) } } JS end 1st: auto generated, also: erlang 2nd: custom

21. db.first User.by_email(‘[email protected]’) db.view(User.by_email(reduce: true)

22. NUMBERS

23. NUMBERS almost 2 years in

24. 1.5 Programmers NUMBERS * fully bootstrapped. 2 founders, me +

    thilo, who brings in most of the money * I also du customer support, UI, ops

25. 16K LOC NUMBERS 9k tests, 5k production

26. 41 Models NUMBERS

27. 1 CouchDB 1.1.0 instance, 153 Views, ~60K documents NUMBERS

28. 1 High-CPU Medium EC2 Instance NUMBERS upgraded from small. 5x

    cpu for 2x price

29. 1. WHAT WORKS WELL

30. I. COMPLEX DATA

31. Membership Plans Payment Method Address Time Passes Discounts Extras Credits

32. {...} One big document no foreign keys, only one save

    operation

33. EASY QUERYING by any attribute, no matter how deeply nested.

34. LOADING LOTS OF DOCS is slow downside

35. EMIT(..., DOC.NAME) only load parts of it

36. SUMMARY Complex Data CouchDB is a great fit

37. II. REPLICATION

38. CONTINUOUS BACKUPS Replication

39. Replication Local Storage Cloudant BigCouch Cluster other backups: copy db

    file to s3 every hour, retain for 4 weeks append only - can just copy file

40. DEPLOYMENTS Replication

41. NEW VIEWS, DATA MIGRATIONS Replication

42. DEPLOYMENTS Replication Production Instance http://cobot.me

43. DEPLOYMENTS Replication Production Instance http://cobot.me New Instance automatic provisioning: scalarium/chef

44. DEPLOYMENTS Replication Production Instance http://cobot.me New Instance Continuous Replication

45. DEPLOYMENTS Replication Production Instance http://cobot.me New Instance Continuous Replication Migrate

    Data... or build views. on error: repeat.

46. DEPLOYMENTS Replication Production Instance http://cobot.me New Instance Continuous Replication

47. DEPLOYMENTS Replication http://cobot.me New Instance

48. SUMMARY Replication is awesome Backups + Deployments

49. III. STATISTICS/ ANALYTICS

50. REVENUE by month and plan

51. { “type”: “Membership”, “confirmed_at”: “2011/01/01” “plans”: [ {“price”: 100, “canceled_to”:

    “2011/03/01”}, {“price”: 120, “canceled_to”: null} ] }

52. Key Value Comment 2011/01 100 Confirmed at 2011/03 -100 Canceled

    plan 2011/03 120 New plan started MAP

53. Key Value 2011/01 100 2011/03 20 REDUCE group by key/month

    => relative change per month

54. RESULT 0 30 60 90 120 2011/01 2011/02 2011/03 Revenue

55. MAP FUNCTION if(doc.ruby_class == 'Membership' && doc.confirmed_at) { emit([doc.space_id, to_month(doc.confirmed_at),

    doc.plans[0].name], emit_value(doc.plans[0])); if(doc.plans.length > 1) { var last_canceled_to = null; for(var i in doc.plans) { var plan = doc.plans[i]; if(plan.canceled_to) { last_canceled_to = plan.canceled_to; emit([doc.space_id, to_month(plan.canceled_to), plan.name], emit_value(plan) * -1); }; if(i > 0) { emit([doc.space_id, to_month(last_canceled_to), plan.name], emit_value(plan)); }; }; }; if(doc.canceled_to) { emit([doc.space_id, to_month(doc.canceled_to), doc.plans[0].name], emit_value(doc.plans[0]) * -1); }; function emit_value(plan) { return plan.total_price_per_cycle_in_cents; }; function to_month(date) { return date.substring(0,4) + '-' + date.substring(5, 7) }; };

56. MAP FUNCTION emit(doc.confirmed_at, doc.plans[0].price); for(var i in doc.plans) { var

    plan = doc.plans[i]; if(plan.canceled_to) { emit(plan.canceled_to, plan.price * -1); }; if(i > 0) { emit(plan.starts_at, plan.price); }; }; };

57. REDUCE-FUNCTION _sum

58. SUMMARY Use clever map/reduce to generate fancy charts.

59. 2. PROBLEMS and what we did to solve them

60. I. TRANSACTIONS

61. NO MULTI-DOC TRANSACTIONS

62. INVOICING

63. {“type”: “Charge”, “price”: 15} {“type”: “Booking”, “price”: 50} items to

    invoice

64. LOAD ITEMS function(doc) { if(doc.type == ‘Booking’) { emit(null, doc.price)

    } }

65. {“type”: “Charge”, “price”: 15} {“type”: “Booking”, “price”: 50} {“type”: “Invoice”,

    “price”: 65, “number”: 1} create invoice with sum of prices

66. {“type”: “Charge”, “price”: 15} {“type”: “Booking”, “price”: 50} {“type”: “Invoice”,

    “price”: 0, “number”: 2} don’t invoice items twice

67. ATOMIC save must be atomic, otherwise invoice twice or not

    at all

68. {“type”: “Charge”, “price”: 15, “invoiced”: true} {“type”: “Booking”, “price”: 50,

    “invoiced”: true} {“type”: “Invoice”, “price”: 65, “number”: 1} “traditional approach”, spans multiple docs

69. SINGLE DOC OPS ARE ATOMIC

70. {“type”: “Charge”, “price”: 15, “invoiced”: true, “_id”: “1”} {“type”: “Booking”,

    “price”: 50, “invoiced”: true, “_id”: “2”} {“type”: “Invoice”, “price”: 65, “number”: 1, “item_ids”: [“1”, “2”] } store everything in invoice

71. MAP FUNCTION function(doc) { if(doc.type == ‘Booking’) { emit(doc._id, doc.price)

    } ...

72. MAP FUNCTION ... if(doc.type == 'Invoice') { doc.invoiced_ids.forEach( function(id) {

    emit(null, id) }); }

73. Key Value Comment null “1” Invoice 1 null “2” Invoice

    1 “1” 15 Booking 1 “2” 50 Booking 2 result of map

74. var invoiced_ids = []; if(row['key'] == null) { invoiced_ids.push(row['value']); }

    else { if(!invoiced_ids.include(row['_id'])) { send_row(row); }; }; LIST FUNCTION emits json, filters invoiced items

75. Key Value Comment null “1” Invoice 1 null “2” Invoice

    1 “1” 15 Booking 1 “2” 50 Booking 2 all invoiced items removed

76. SUMMARY Try to keep documents self-contained. Transactions can be simulated.

77. II. VIEW BUILDING

78. A. SLOW VIEWS

79. VIEWS BUILT ONLY ONCE

80. VIEWS BUILT ON READ

81. EXTERNAL JS PROCESS not very fast

82. CASE 1: NEW VIEWS

83. PROCESS ALL DOCS can take a long time

84. BUILD TIME > CLIENT TIMEOUT

85. None

86. CASE 2: MANY WRITES, READ TIMEOUT e.g. weekly cron job

87. SOLUTION?

88. WORKAROUNDS depending on use case

89. 1. ERLANG VIEWS

90. AUTO-GENERATED BY COUCH POTATO

91. CUSTOM VIEWS STILL JAVASCRIPT one design doc per view

92. ~25% FASTER others reported up to 10x

93. 2. VIEW HEATER

94. CRON JOB find “slowest” views, query with stale=ok biggest lag,

    update sequence

95. QUERY RANDOM VIEW Alternative lots of overhead finding slow views

96. BIGCOUCH HAZ IT Coming to CouchDB (?)

97. B. STALE VIEWS part solution, new problems

98. ?STALE=OK

99. ?STALE=UPDATE_AFTER

100. INCONSISTENT DATA

101. ACTIVITY FEED ANALYTICS display non-critical data - ok

102. LIST AFTER CREATE add member, not there

103. CRITICAL DATA invoices, link from email

104. CALCULATIONS invoice items

105. DELETION

106. VIEWS RETURN ROWS FOR DELETED DOCS

107. LIST AFTER DELETE

108. BUT NO DOCS ?include_docs=true errors

109. SUMMARY No silver bullet. Choose your lead bullets wisely. stale

     views, optimize performance (erlang), view heater

110. III. MIGRATIONS

111. SCHEMA-FREE no schema migrations

112. DATA MIGRATIONS

113. RENAME TYPES MOVE ATTRIBUTES NEW ATTRIBUTES

114. day_tickets = DB.view('day_ticket/by_space_id', :reduce => false, :include_docs => true) day_tickets.each

     do |doc| doc['ruby_class'] = 'DayPass' DB.save_doc doc end RENAME DAY TICKETS TO DAY PASSES

115. CHANGING 10K DOCS TIMES 150 VIEWS = 1.5M UPDATES 3k

     updates/s = 8 minutes, high load, website non-responsive, timeouts

116. OFFLINE MIGRATIONS New Instance

117. ALTERNATIVE IDEA: MIGRATE ON READ

118. SUMMARY Avoid migrations Ask me in a year. Migrate offline.

     ask me about migrate on read

119. 3. CONCLUSION

120. <3 SCHEMALESS DESIGN <3 document design

121. REPLICATION <3 <3 <3 Backups, Migrations, Deployments

122. VIEW BUILDING Stale views, Erlang, view heater

123. MIGRATIONS On read?

124. TOOL SUPPORT Rails Plugins crud, pagination

125. WOULD WE USE IT AGAIN?

126. just kidding

127. YES love flexibility, problems can be solved

128. THANK YOU. QUESTIONS? @langalex, [email protected] http://github.com/langalex/couch_potato