MongoDB Sao Paulo 2012: Open Library on MongoDB

Luciano Ramalho, Principal, Oficinas Turing
O Open Library é um projeto do Internet Archive com o objetivo de compilar mais o completo catálogo de livros do mundo. Tem atualmente mais de 117 milhões de registros bibliográficos, incluindo um histórico de alterações que chega a 18 versões em alguns casos. Esta apresentação mostra como o conjunto de dados OL foi convertido para MongoDB e como Map/Reduce e o novo framework de agregação foram usados para analisar os registros e subsidiar uma refatoração profunda do esquema, tornando-o mais adequado às características de desempenho e funcionalidade do MongoDB.

Transcript

1. Leveraging Map/Reduce and the Aggregation Framework for Data Analysis and Schema Design Luciano Ramalho luciano@ramalho.org @ramalhoorg Open Library on MongoDB Tuesday, July 10, 12
2. @ramalhoorg Topics • About the Open Library project • Converting and importing the OLC dataset • Data analysis with the Aggregation Framework • Data analysis with Map! /Reduce • Refactoring the schema design for MongoDB • Wrapping up Tuesday, July 10, 12
3. @ramalhoorg About the Open Library Project Tuesday, July 10, 12
4. @ramalhoorg About the Open Library • Mission: • A project by the Internet Archive • 117,439,126 million bibliographic records as of June, 2012 • More than 1,000,000 free e-books for download “One web page for every book” Tuesday, July 10, 12
5. @ramalhoorg Open Library Technology • Infobase: a semi-structured database abstraction built on top of normalized PostgreSQL tables • a.k.a ThingDB • Record versioning support • Many joins to retrieve one conceptual entity • Heavily dependent on SOLR/Lucene for production Tuesday, July 10, 12
6. @ramalhoorg Semi-structured data model • Theoretical basis • Literature • Notation • Normalizing without the First Normal Form (N1NF) Tuesday, July 10, 12
7. @ramalhoorg Converting and Importing the OLC Dataset Tuesday, July 10, 12
8. @ramalhoorg Handling the dataset • 118.598.056 lines (as of June 1st, 2012) • 91 GB uncompressed • 32 different types of records • 1.158.930 (~1%) are not bibliographic records Tuesday, July 10, 12
9. @ramalhoorg Converting to load • The simplest conversion possible • Minimal changes to original schema • Choosing a primary key for the _id field Tuesday, July 10, 12
10. @ramalhoorg Using mongoimport • Scripts used • Monitoring import performance • Monitoring database size increase Tuesday, July 10, 12
11. @ramalhoorg Data Analysis with the Aggregation Framework Tuesday, July 10, 12
12. @ramalhoorg Aggregation Framework Basics Tuesday, July 10, 12
13. @ramalhoorg Indexing for data analysis Tuesday, July 10, 12
14. @ramalhoorg Example 1 Tuesday, July 10, 12
15. @ramalhoorg Example 2 Tuesday, July 10, 12
16. @ramalhoorg Limitations of the Aggregation Framework Tuesday, July 10, 12
17. @ramalhoorg Data Analysis with Map/Reduce Tuesday, July 10, 12
18. @ramalhoorg Map/Reduce Basics Tuesday, July 10, 12
19. @ramalhoorg Deep data analysis • Stonebraker’s “schema after” • Schema analysis • Statistics about record structure • Statistics about embedded data types Tuesday, July 10, 12
20. @ramalhoorg Verifying the Referential Integrity of the Dataset • Schema additions • Indexes • Scripts Tuesday, July 10, 12
21. @ramalhoorg Example 3 Tuesday, July 10, 12
22. @ramalhoorg Example 4 Tuesday, July 10, 12
23. @ramalhoorg Refactoring the schema design for MongoDB Tuesday, July 10, 12
24. @ramalhoorg Refactoring the schema • Primary key for the _id field • Choosing related documents to embed Tuesday, July 10, 12
25. @ramalhoorg Representing versions • Record versioning • Statistics from the dataset • Alternative representations • Chosen representation Tuesday, July 10, 12
26. @ramalhoorg Referential Integrity • Maintaining the referential integrity going forward • Support tools: • Schema additions • Indexes • Application Framework Support • Asynchronous tasks Tuesday, July 10, 12
27. @ramalhoorg Sharding • Sharding basics • Criteria for sharding • Setting up sharding Tuesday, July 10, 12
28. @ramalhoorg Incremental conversion • Record migration on demand Tuesday, July 10, 12
29. @ramalhoorg Wrapping Up Tuesday, July 10, 12
30. @ramalhoorg Conclusion Tuesday, July 10, 12
31. @ramalhoorg Next Steps Tuesday, July 10, 12
32. @ramalhoorg Q & A • Open Library mailing lists • ... • My e-mail: luciano@ramalho.org Tuesday, July 10, 12