Scalable and Flexible File Storage for Financial Websites - Powered by MongoDB GridFS

Marko Gräßlin, Senior Software Architect, Solvians IT-Solutions GmbH

MongoDB Berlin 2012

In this talk we present our migration process to MongoDB as our primary file storage system using GridFS. We will show how GridFS empowered us to build a unified service with a small but powerful interface, addressing the varying requirements of our demanding customers. Beside the migration process, we will alsotalk about performance tuning and solving MongoDB related issues that have occurred.

Transcript

1. 1 ©SolviansIT-Solutions GmbH Solvians IT-Solutions GmbH Scalable and flexible file storage for financial websites – powered by MongoDB GridFS
2. 2 ©SolviansIT-Solutions GmbH Pro le Who we are Marko Gräßlin Senior Software Architect at Solvians Backend systems and framework development Frederik Happel System Administrator at Solvians Automation of hardware and software con guration
3. 3 ©SolviansIT-Solutions GmbH Pro le Main focus of Solvians   Concept, development and operation of nance websites   Realtime delivery of price data   Neartime delivery of master data
4. 4 ©SolviansIT-Solutions GmbH  Legal obligation for providing fact sheet documents  About 100.000 investment products  300.000 corresponding les  Daily updates Pro le Challenge: Disclosure requirement
5. 5 ©SolviansIT-Solutions GmbH 5 ©SolviansIT-Solutions GmbH Overview The former FileDB   Infrastructure Overview   MySQL based le storage   The moment when this wasn't enough The new FileService MongoDB as storage layer Conclusion
6. 6 ©SolviansIT-Solutions GmbH The former FileDB Infrastructure overview
7. 7 ©SolviansIT-Solutions GmbH  Supports versioning  Links between entities and les  Additional metadata elds which were not searchable The former FileDB MySQL based le storage
8. 8 ©SolviansIT-Solutions GmbH  Not scalable  Backup and restore almost impossible  Schema changes take too long  Not exible The former FileDB The moment when this wasn't enough
9. 9 ©SolviansIT-Solutions GmbH 9 ©SolviansIT-Solutions GmbH Overview The former FileDB The new FileService   Our Solution   Application design   Architecture of the FileService   MongoDB`s in uence in our design   Migration MongoDB as storage layer Conclusion
10. 10 ©SolviansIT-Solutions GmbH  Complete rewrite  Using MongoDB GridFS  Legacy adapter for old interface The new FileService Our solution
11. 11 ©SolviansIT-Solutions GmbH  Simple and compact interface  Schema based validation  Free schemas per application The new FileService Application design
12. 12 ©SolviansIT-Solutions GmbH The new FileService Architecture of the FileService
13. 13 ©SolviansIT-Solutions GmbH  Links embedded in the object  Latest version ag (no aggregation)  Upserts for access timestamps (unsafe)  Calculation of used quotas as map reduce job The new FileService MongoDB´s in uence in our design
14. 14 ©SolviansIT-Solutions GmbH  Legacy adapter  Migration of all les  Switch services  Monitor accesses to old FileDB for two weeks The new FileService Migration
15. 15 ©SolviansIT-Solutions GmbH 15 ©SolviansIT-Solutions GmbH Overview The former FileDB The new FileService MongoDB as storage layer   Goals when using MongoDB   Availability and data protection with replicasets   Scalability with sharding   Faster backup and restore   Problems we ran into Conclusion
16. 16 ©SolviansIT-Solutions GmbH  Lower resource consumption per node (CPU, memory, disk)  Availability and data protection  Scalability: Growing amount of data  Scalability: Add more nodes to cope with more writes  Faster backup and restore MongoDB as storage layer Goals when using MongoDB
17. 17 ©SolviansIT-Solutions GmbH  2 data nodes providing dataprotection in case of node crash  1 arbiter providing high availibility in case of node crash  Lowest resource consumption for replicaset MongoDB as storage layer Availability and data protection with replicasets
18. 18 ©SolviansIT-Solutions GmbH  Sharding providing simple way to cope with growing data  Increase write performance by adding shards MongoDB as storage layer Scalability with sharding
19. 19 ©SolviansIT-Solutions GmbH  Reduced backup time to 3 hours  Speed up restore process to 20 minutes MongoDB as storage layer Faster backup and restore
20. 20 ©SolviansIT-Solutions GmbH  Query optimization with elemMatch  Sharding key (preventing hotspots)  Monitoring of con g servers difficult  Restore of con gservers not as straight forward as expected MongoDB as storage layer Problems we ran into
21. 21 ©SolviansIT-Solutions GmbH 21 ©SolviansIT-Solutions GmbH Overview The former FileDB The new FileService MongoDB as storage layer Conclusion
22. 22 ©SolviansIT-Solutions GmbH  Fexibility through free schemas  Low development effort  Scalability  Data protection Conclusion
23. 23 ©SolviansIT-Solutions GmbH Contact 2 3 Marko Gräßlin Senior Software Architect Phone: +49 (69) 945 188 – 621 E-Mail: Marko.Graesslin@solvians.com Speicherstraße 49 – 51 60327 Frankfurt am Main Germany Frederik Happel System Administrator Phone: +49 (69) 945 188 – 572 E-Mail: Frederik.Happel@solvians.com Speicherstraße 49 – 51 60327 Frankfurt am Main Germany