Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scalable and Flexible File Storage for Financial Websites - Powered by MongoDB GridFS

mongodb
April 05, 2012

Scalable and Flexible File Storage for Financial Websites - Powered by MongoDB GridFS

Marko Gräßlin, Senior Software Architect, Solvians IT-Solutions GmbH

MongoDB Berlin 2012

In this talk we present our migration process to MongoDB as our primary file storage system using GridFS. We will show how GridFS empowered us to build a unified service with a small but powerful interface, addressing the varying requirements of our demanding customers. Beside the migration process, we will alsotalk about performance tuning and solving MongoDB related issues that have occurred.

mongodb

April 05, 2012
Tweet

More Decks by mongodb

Other Decks in Technology

Transcript

  1. 1 ©SolviansIT-Solutions GmbH Solvians IT-Solutions GmbH Scalable and flexible file

    storage for financial websites – powered by MongoDB GridFS
  2. 2 ©SolviansIT-Solutions GmbH Pro le Who we are Marko Gräßlin

    Senior Software Architect at Solvians Backend systems and framework development Frederik Happel System Administrator at Solvians Automation of hardware and software con guration
  3. 3 ©SolviansIT-Solutions GmbH Pro le Main focus of Solvians  

    Concept, development and operation of nance websites   Realtime delivery of price data   Neartime delivery of master data
  4. 4 ©SolviansIT-Solutions GmbH  Legal obligation for providing fact sheet documents

     About 100.000 investment products  300.000 corresponding les  Daily updates Pro le Challenge: Disclosure requirement
  5. 5 ©SolviansIT-Solutions GmbH 5 ©SolviansIT-Solutions GmbH Overview The former FileDB

      Infrastructure Overview   MySQL based le storage   The moment when this wasn't enough The new FileService MongoDB as storage layer Conclusion
  6. 7 ©SolviansIT-Solutions GmbH  Supports versioning  Links between entities and les

     Additional metadata elds which were not searchable The former FileDB MySQL based le storage
  7. 8 ©SolviansIT-Solutions GmbH  Not scalable  Backup and restore almost impossible

     Schema changes take too long  Not exible The former FileDB The moment when this wasn't enough
  8. 9 ©SolviansIT-Solutions GmbH 9 ©SolviansIT-Solutions GmbH Overview The former FileDB

    The new FileService   Our Solution   Application design   Architecture of the FileService   MongoDB`s in uence in our design   Migration MongoDB as storage layer Conclusion
  9. 11 ©SolviansIT-Solutions GmbH  Simple and compact interface  Schema based validation

     Free schemas per application The new FileService Application design
  10. 13 ©SolviansIT-Solutions GmbH  Links embedded in the object  Latest version

    ag (no aggregation)  Upserts for access timestamps (unsafe)  Calculation of used quotas as map reduce job The new FileService MongoDB´s in uence in our design
  11. 14 ©SolviansIT-Solutions GmbH  Legacy adapter  Migration of all les  Switch

    services  Monitor accesses to old FileDB for two weeks The new FileService Migration
  12. 15 ©SolviansIT-Solutions GmbH 15 ©SolviansIT-Solutions GmbH Overview The former FileDB

    The new FileService MongoDB as storage layer   Goals when using MongoDB   Availability and data protection with replicasets   Scalability with sharding   Faster backup and restore   Problems we ran into Conclusion
  13. 16 ©SolviansIT-Solutions GmbH  Lower resource consumption per node (CPU, memory,

    disk)  Availability and data protection  Scalability: Growing amount of data  Scalability: Add more nodes to cope with more writes  Faster backup and restore MongoDB as storage layer Goals when using MongoDB
  14. 17 ©SolviansIT-Solutions GmbH  2 data nodes providing dataprotection in case

    of node crash  1 arbiter providing high availibility in case of node crash  Lowest resource consumption for replicaset MongoDB as storage layer Availability and data protection with replicasets
  15. 18 ©SolviansIT-Solutions GmbH  Sharding providing simple way to cope with

    growing data  Increase write performance by adding shards MongoDB as storage layer Scalability with sharding
  16. 19 ©SolviansIT-Solutions GmbH  Reduced backup time to 3 hours  Speed

    up restore process to 20 minutes MongoDB as storage layer Faster backup and restore
  17. 20 ©SolviansIT-Solutions GmbH  Query optimization with elemMatch  Sharding key (preventing

    hotspots)  Monitoring of con g servers difficult  Restore of con gservers not as straight forward as expected MongoDB as storage layer Problems we ran into
  18. 23 ©SolviansIT-Solutions GmbH Contact 2 3 Marko Gräßlin Senior Software

    Architect Phone: +49 (69) 945 188 – 621 E-Mail: [email protected] Speicherstraße 49 – 51 60327 Frankfurt am Main Germany Frederik Happel System Administrator Phone: +49 (69) 945 188 – 572 E-Mail: [email protected] Speicherstraße 49 – 51 60327 Frankfurt am Main Germany