Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Shutterfly Data Layer and Schema Definitions - Luciano Resende, Shutterfly

mongodb
December 12, 2011

The Shutterfly Data Layer and Schema Definitions - Luciano Resende, Shutterfly

MongoSV 2011

This talk goes over various patterns, techniques and best practices around developing webscale applications using MongoDB. We will cover various techniques including a DSL like schema definition tool, benefits of using data abstraction API, and other best practices we have learned.

mongodb

December 12, 2011
Tweet

More Decks by mongodb

Other Decks in Technology

Transcript

  1. About Luciano Resende •  Staff Software Engineer – Shutterfly, Inc.

    •  Member – Apache Software Foundation •  Committer: Apache Tuscany, Wink, Nuvem, PhotoArk
  2. MongoDB initial POCs December 12, 2011 Business Confidential 4 MongoDB

      Java   Component       .NET   Component       Java   Component       first_name   firstname   firstName   Floor  plan  image  from  :  h3p://benchmarkoffices.com/floorplans.htm  
  3. The Platform •  Applications will interact with MongoDB persistence via

    services •  Enforce a consistence schema across multiple applications and cross-platform •  Data Access Layer •  Data Access Layer that in conjunction with UDDL abstracts the persistence layer allowing for more flexible and clean code •  A cross platform data schema definition (text DSL) •  Simple human (& machine) readable language to describe our domain model entities December 12, 2011 Business Confidential 5 Services   Data  Access  Layer   UDDL  
  4. Schema Definition – Why using Text DSL •  We adopted

    Tuscany SCA, which separates componentization, composition, communication and QoS concerns from the business logic. It becomes obviously natural to abstract the data modeling now (inspired by DDD). •  (Annotated) POJO (JAXB, JPA, Morphia) has too much noises and is abuse-prone. We want to enforce the patterns. •  XSD is too complicated and it doesn’t fit all for web 2.0 •  We need a simple human (& machine) readable language to describe our domain model to provide/promote: •  Service-friendly DTO, Persistence (RDB and NoSQL), Governance, Reuse, Validation, MVC, Best Practice, Documentation •  We call the DSL as Universal Data Definition Language (UDDL). •  Unify Data Definition across different platform application domains
  5. Our decision: Textual DSL •  Derived from an Apache licensed

    open source project Sculptor •  http://fornax.itemis.de/confluence/display/fornax/Sculptor+%28CSC%29 •  Easy to learn, intuitive syntax of the textual DSL, based on the concepts from DDD. •  Textual DSL has a lot of productivity benefits over graphical tools •  Quick development round trip, short feedback loop •  Generation of complete application from a single model, not only fragments that are hard to fit in to the overall design •  Supports JPA (oracle/mysql/postgresql)/MongoDB/JAXB/Spring/SpringMVC •  Great extensibility and customization •  Based on Eclipse Xtext/Xtend/Xpand code generation framework •  Can be used with text editor or any IDE, but DSL editor with error highlight, code completion, and outline is provided for Eclipse users •  Documentation w/ diagrams for domain model •  No runtime magic is built into the tools (generated code can be used without the tooling) 8
  6. Schema Definition - The idea December 12, 2011 9 Domain

      Model   RDB   MongoDB   EnFty   EnFty   EnFty   DTO   REST   Root   Resource   REST   Sub-­‐ Resource   DTO   DTO   REST   Sub-­‐ Resource   Locate   CRUD   CRUD   CRUD   POJO   XSD   JSON   Schema   Textual  DSL   (such  as   Sculptor)   Protocol   Buffer/ThriT   DDL   1 2 1.  Persistence   2.  DTO   3.  Mapping   4.  IntrospecFon   5.  ValidaFon   6.  MVC/UI   3 metadata   4 uddl   Java  code   ConfiguraFon  files   Documents   (UML  diagrams,  HTML   docs)   Generated  arFfacts   Code   generaFon   templates  
  7. Schema Definition - Sample Application AddressBook { basePackage=com.shutterfly.model Module addressbook

    { hint="store=mongodb” Entity Contact { dto mappedName="contact" String id key; String userId ! nullable indexed; String prefix; String firstName; String middleName; String lastName; String company; - @ContactType type; - @Gender gender; List<String> groups; //group names - List<@ContactEmail> emails; - List<@ContactAddress> addresses; @'javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class)' Timestamp created mappedName="updtime"; @'javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class)' Timestamp lastUpdated mappedName="instime"; } BasicType ContactEmail { String label; Boolean isPrimary; String emailAddress ! nullable indexed; } } Persistence  in   use…   EnFty   Required   Indexes   Embedded  and   References   Embedded  and   References  
  8. Schema Definition – Generated Java Code @Generated("uddl:AddressBook/addressbook/Contact") @XmlRootElement @XmlAccessorType(javax.xml.bind.annotation.XmlAccessType.FIELD) @Entity(value

    = "contact", noClassnameStored = true) public class Contact implements Serializable, Cloneable { @Id private String id; @XmlElement() private String userId; @XmlElement() private String prefix; @XmlElement() private String firstName; @XmlElement() private String middleName; @XmlElement() private String lastName; @XmlElement() private String company; @XmlElement() private List<String> groups = new ArrayList<String>(); @XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) @XmlElement(name = "updtime") @Property("updtime") private Date created; @XmlJavaTypeAdapter(com.shutterfly.model.common.jaxb.adapter.UTCDateXmlAdapter.class) @XmlElement(name = "instime") @Property("instime") private Date lastUpdated; @XmlElement() private ContactType type; @XmlElement() private Gender gender; @XmlElementWrapper(name = "emails") @XmlElement(name = "email") @Embedded private List<ContactEmail> emails = new ArrayList<ContactEmail>(); @XmlElementWrapper(name = "addresses") @XmlElement(name = "address") @Embedded private List<ContactAddress> addresses = new ArrayList<ContactAddress>(); … } December 12, 2011 Business Confidential 11
  9. DSL language development   Grammar   Language   (DSL)  

      EMF  Ecore  model   SemanFc   model   Parse  tree   model   Grammar   model   Eclipse   UI   Xtext   EMF  Ecore  model   Xtend/ Check   Load/save   Parse/validate   Extend/ transform/ validate   Java  code   ConfiguraFon   files   Documents   (UML   diagrams,   HTML  docs)   Xpand   Eclipse  Model   Workflow  Engine   (MWE)   Generated  arFfacts     Other  DSLs   (such  as  XSD,   GPB,  ThriT  or   Avro)    
  10. Data Access Layer •  Provides an Abstraction layer to Persistence

    API and storage •  Also facilitating migration from one Persistence storage to another >  Or even Y adaptor pattern to write to different Persistence storages •  Promote rapidly adoption of new technologies •  Provides different concrete APIs •  Query based APIs •  Entity based APIs •  Binary APIs (GridFS) •  Cache •  Query Abstraction •  Ability to plug key generators •  Promotes best practices across all applications
  11. Data Access Layer Services   Data  Access  Layer   UDDL

      ORM   Cache   Persistence  Driver  
  12. Data Access Layer - ORM Discuss about Morphia, and what

    issues we had December 12, 2011 Business Confidential 20