Databases and the Modern Web

A high-level walk through of data storage options for modern web applications and an introduction to using MongoDB. This talk was given at DevCon5 2013 in Los Angeles, CA.

Transcript

1. brandonmblack.com @brandonmblack Software Engineer, MongoDB, Inc. Brandon Black DATABASES AND THE MODERN WEB
2. brandonmblack.com @brandonmblack +
3. brandonmblack.com @brandonmblack AGENDA • What is Big Data? • What is a Modern Web App? • A Brief History of the Web • Picking the Right Database • MongoDB’s Approach • Examples
4. What is Big Data?
5. brandonmblack.com @brandonmblack What is big data? Do we all mean the same thing? Velocity: the speed at which you’re dealing with data. Volume: the quantity of data, in sheer size. Variety: diversity of the data you’re dealing with. Fun facts about data growth: ! Photos: • 4 billion in the last year alone, 4x last decade • Half found their way onto the Internet ! Information: • 1.8 zettabytes annually (Source: IDC 2011) • Increase 50x by 2020
6. What is a Modern Web App?
7. brandonmblack.com @brandonmblack
8. A Brief History of the Web
9. brandonmblack.com @brandonmblack Late 90s • The web was just beginning to move beyond simple markup. • Most everything was happening in languages like Perl and C. • Relational databases were everywhere. • The future was bright!
10. brandonmblack.com @brandonmblack EARLY ‘00s • Crazy growth. • Dynamic page content starting to gain traction. • Dynamic languages start to gain traction. • LAMP stack dominates. • Relational databases.
11. brandonmblack.com @brandonmblack MID 00s • Social and multimedia redefines scale. • Memcache is king, but applications become more complex. • Problem specific frameworks start to emerge.
12. brandonmblack.com @brandonmblack PROBLEMS* • Data is growing dramatically in size with no end in sight. • Data complexity and application complexity is increasing. • Push for real-time and dynamic data limits using your cache layer as a crutch for scalability. ! * Side effects are not limited to, but may include: • Slower development cycles and loss of agility. • Complex architectures. • Over abstraction everywhere.
13. brandonmblack.com @brandonmblack TODAY’S TRENDS • Horizontal Scalability • Variety of Tools • Specialization no longer There is a one size fits all solution.
14. your data. The right database for
15. your development. The right database for
16. scale and performance. The right database for
17. brandonmblack.com @brandonmblack Relational • Two-dimensional storage! • In-place updates! • Query on any field! • Each contains a single value! • Very structured data! • Normalization leads to many tables and poor locality.! ! Examples:! Oracle, PostgreSQL, MySQL, MSSQL
18. brandonmblack.com @brandonmblack Key-value Stores • One-dimensional storage! • Single value blob! • Query on key only! • Some secondary indexes! • No schema! • No updates, replace value! ! Examples:! Cassandra, Redis, Memcache
19. brandonmblack.com @brandonmblack Document Database • N-dimensional storage! • Each field is 0, 1 or many or embedded values! • Query on any field at any level! • Flexible schema! • In-line updates! • Embedded data has great locality, better performance! ! Examples:! MongoDB, CouchDB, RethinkDB
20. MongoDB’s Approach
21. brandonmblack.com @brandonmblack MongoDB: Built to Scale •Document •Open-Source •High Performance •Horizontally Scalable •Full Featured
22. brandonmblack.com @brandonmblack Document Database • Not for PDF or DOC files • A document is essentially an associative array. • Document == JSON Object • Document == Ruby Hash • Document == PHP Array • Document == Python Dict
23. brandonmblack.com @brandonmblack Open-Source • MongoDB is Open-Source • Available on Github • Licensed under AGPL • Started and sponsored by MongoDB, Inc. • Contributions are welcome.
24. brandonmblack.com @brandonmblack High Performance • Written in C++ • Uses extensive memory-mapped files (ex. read-through and write-through memory caching). • Runs nearly everywhere. • Data is serialized as BSON (for fast parsing) • Full support for primary and secondary indexes. • Document model means less work, more agile.
25. brandonmblack.com @brandonmblack Shard N Shard 3 Shard 2 Shard 1 Horizontally Scalable
26. brandonmblack.com @brandonmblack Data Landscape Depth of Functionality Scalability & Performance Memcached MongoDB RDBMS
27. brandonmblack.com @brandonmblack Full Featured • Ad-hoc queries. • Real-time aggregation. • Auto-sharding. • Rich query capabilities. • Strongly consistent. • Geospatial features. • Parallel Processing and Targeted Queries. • Intelligent Replication. • Flexible schema.
28. brandonmblack.com @brandonmblack Native Support
29. brandonmblack.com @brandonmblack Native Support
30. Examples
31. brandonmblack.com @brandonmblack Installation 1 Download From website: http://www.mongodb.org/downloads sudo apt-get install mongodb-10gen brew install mongodb Package manager: 2 Startup MongoDB 3 Connect with the mongo Shell mongod —dbpath /path/to/data
32. brandonmblack.com @brandonmblack Using the Ruby Driver Secondary Secondary Primary Client Application Driver Write Read Read
33. Using the Ruby Driver # install gem and native extensions (optional)! gem install mongo gem install bson_ext require 'mongo'! include Mongo! ! # connecting to the database! client = MongoClient.new # defaults to localhost:27017! client = MongoClient.new('host1.example.com')! client = MongoClient.new('host1.example.com', 27017)! # using configuration options! opts = { :pool_size => 5, :pool_timeout => 5 }! client = MongoClient.new('host1.example.com', 27017, opts)!
34. Using the Ruby Driver seeds = ['h1.example.com:27017',! 'h2.example.com:27017',! 'h3.example.com:27017']! ! # connecting to a replica set! client = MongoReplicaSetClient.new(seeds)! client = MongoReplicaSetClient.new(seeds, :read => :secondary)! # connecting to a sharded cluster! client = MongoShardedClient.new(seeds)! client = MongoShardedClient.new(seeds, :read => :secondary)! # using a connection string! ENV['MONGODB_URI'] = 'mongodb://host1:27017?ssl=true'! client = MongoClient.new!
35. Using the Ruby Driver # using a database! db = client.db('blog')! db = client['blog']! ! client.drop_database('blog')! client.database_names! ! # using a collection! coll = db['posts']! ! coll.drop! db.collection_names!
36. brandonmblack.com @brandonmblack Terminology RDBMS MongoDB Table, View
    Collection Row
    Document Index
    Index Join
    Embedded Document Foreign Key
    Reference Partition
    Shard
37. brandonmblack.com @brandonmblack Blog Example • Models / Entities: • Users • Articles • Comments • Tags or Categories
38. brandonmblack.com @brandonmblack we would start by defining our schema. In a relational application
39. brandonmblack.com @brandonmblack Relational Diagram User ·Name ·Email address Category ·Name ·URL Comment ·Comment ·Date ·Author Article ·Name ·Slug ·Publish date ·Text Tag ·Name ·URL
40. brandonmblack.com @brandonmblack and we let the schema evolve as we go. In MongoDB we start building
41. brandonmblack.com @brandonmblack MongoDB Diagram User ·Name ·Email address Article ·Name ·Slug ·Publish date ·Text ·Author Comment[] ·Comment ·Date ·Author Tag[] ·Value Category[] ·Value
42. Inserting a Document # example document! author = {! :username => 'brandonblack',! :first => 'brandon',! :last => 'black'! }! ! # inserting into my blog database! client['blog']['users'].insert(author) No database or collection creation required!
43. Inserting a Document # example document! article = {! :title => 'Hello World'! :username => 'brandonblack',! :tags => ['ruby', 'getting started'],! :comments => [ # embedded docs ]! }! ! ! # inserting into my blog database! client['blog']['articles'].insert(article) No database or collection creation required!
44. Finding Documents coll = client['blog']['users']! ! # finding a single document! coll.find_one! coll.find_one({ :username => 'brandonblack' })! coll.find_one({ :username => 'brandonblack' }, { :first => 1})! ! coll = client['blog']['articles']! ! # finding multiple documents (using cursors)! cursor = coll.find({ :username => 'brandonblack' }, :limit => 10)! cursor.each do |article|! puts article['title']! end!
45. Updates and Deletes # updating a document! article = { :title => 'Hello Ruby' }! coll.update({ '_id' => article_id }, article)! coll.update({ '_id' => article_id },! {'$set' => {:title => 'Hello Ruby'}})! ! # deleting a single document! coll.remove({ '_id' => article_id })! ! # delete multiple documents! coll.remove({ 'username' => 'brandonblack' })! ! # delete all documents in a collection! coll.remove!
46. Indexes # running explain on a query! coll.find({ :username => 'brandonblack' }).explain! ! # showing collection indexes! coll.index_information! ! # adding an index! coll.ensure_index({ :username => 1 })! coll.ensure_index({ :username => Mongo::ASCENDING })! ! coll.ensure_index({ :username => -1 })! coll.ensure_index({ :username => Mongo::DESCENDING })! ! # adding a special index types! coll.ensure_index({ :loc => Mongo::GEO2D })! coll.ensure_index({ :title => Mongo::TEXT })! ! # droping an index! coll.drop_index('username_1')! ! # dropping all indexes for a collection! coll.drop_indexes!
47. ODM Example # rails setup! rails g mongoid:config! ! # non-rails setup! Mongoid.load!("path/to/your/mongoid.yml", :production)! ! # document examples! class Article! include Mongoid::Document! field :title, type: String! embeds_many :comments! end! ! class Comment! include Mongoid::Document! field :comment_text, type: String! embedded_in :article! end!
48. What’s Next?
49. very quickly. We’ve covered a lot
50. brandonmblack.com @brandonmblack Learn More Free Online Courses! http://education.mongodb.com! ! Events & Webinars! http://www.mongodb.com/events! ! Presentations! http://www.mongodb.com/presentations
51. brandonmblack.com @brandonmblack Learn More
52. brandonmblack.com @brandonmblack Talk with Us • Github • Stack Overflow • Google Group
53. brandonmblack.com @brandonmblack Software Engineer, MongoDB, Inc. Brandon Black Thank You Slides available at: ! http://github.com/brandonblack http://speakerdeck.com/brandonblack