X.Commerce and MongoDB

How X.commerce is building a developer platform on open source--including MongoDB

Transcript

1. How X.commerce is building a platform on open source Vinay Gupta Developer, X.commerce vinay.gupta@x.com
2. X.commerce Vision
3. What is X.commerce? l  eBay, Inc.'s newest business unit l  going to bring everything eBay, Inc. has to merchants, large and small l  a way for commerce developers to reach more merchants with innovative capabilities l  an open platform, built greenfield, to make the last two points happen
4. Commerce Opportunity for Everyone
5. How do we pull this off? l  X.commerce Cloud l  X.commerce Fabric integration platform l  X.commerce Open Commerce Language l  All built on open source!
6. The X.commerce Cloud
7. X.commerce Fabric Point-to-point integrations Reuse model Merchant: easy adoption of new capabilities, low switching costs Developer: no barriers to entry, easy on-boarding of customers
8. Open Commerce Language l  Commerce Esperanto l  Defined with Avro schemas l  Compatibility l  Evolvability l  Developed in the open l  Cannot be an innovation bottleneck l  github.com/xcommerce/X.commerce-Contracts
9. eBay, Inc.'s commitment to open source
10. Why Open Source? l  Our team's experience and willingness to invest l  Our principles l  Transparency l  Consensus l  Distributed responsibility/autonomy l  Collaboration l  Meritocracy l  We're building a community
11. Key Partnerships VMWare l  CloudFoundry l  RabbitMQ l  Spring Rackspace l  OpenStack Cloudera l  Hadoop l  Avro l  ZooKeeper 10gen l  MongoDB
12. MongoDB in X.commerce l  A core component of the X.commerce Cloud l  The standard data store for all in-house capabilities l  The Fabric l  Authorization and configuration data l  Message tracing
13. Why MongoDB? l  Quality attributes l  Elastic l  Reliable l  Manageable l  Deployment l  Uniform l  Easily automated l  Scales down Data model Flexible Constrained Simple optimizations Already integrated with Cloud Foundry (Use service broker to expose the external mongo cluster)
14. Cloud deployment, initial
15. Cloud deployment, redux
16. Cloud deployment, redux
17. Cloud deployment, redux
18. Cloud deployment, redux
19. Deployment in CloudFoundry •  pre-create mongo cluster based on application needs •  Spin a mongos process on each DEA •  Service broker to interact with external service(mongos) •  Can use the same cluster for (2) different apps
20. Schema evolution l  Initial model for inventory data { sku: “123123”, available: 83, reserved: 2, locations: ! [ { location: “austin”, available: 36, reserved: 1 }, ! { location: “san jose”, available: 47, reserver: 1 } ]! } l  Lots of contention at high volume l  Traditional locking mechanisms failed
21. Schema evolution l  More scalable model { item: “123123-1”, sku: “123123”, location: “austin”,! status: “reserved” },! { item: “123123-2”, sku: “123123”, location: “austin”,! status: “available” },! { item: “123123-3”, sku: “123123”, location: “san jose”,! status: “available” } l  document-per-item l  trade-off: space vs. contention
22. Message Tracing l  Fabric-specific logging mechanism { guid: “80B2A”, parentGuid: “80B1E”, ! topic: “/order/created”, size: 112, status: 200 } l  Implement Time-capped collection l  used for real-time access like tail l  Probably built-in in Mongo 2.2 ??? l  M/R for on-the-fly metrics l  Hadoop/Hbase/Hive cluster for historical data
23. Some Lessons Learned l  Open source is an investment l  Get involved with the community l  Leverage partnerships to manage risk l  Plan on implementing (at least) twice
24. Get started! Sandbox integration environment: https://devportal.x.com/ http://api.sandbox.x.com We're hiring! http://www.ebaycareers.com vinay.gupta@x.com
25. Questions?