Salamander: Using Docker with NoSQL

A real-world journey through the evolution of Salamander, a data-intensive application running on AWS and built around MongoDB, Neo4j and Docker.

The presentation explores how Docker was introduced to simplify increasingly complex NoSQL infrastructure, covering data segmentation, isolation, scaling, persistence with Amazon S3, Docker Compose, service discovery and container orchestration. It follows the architecture from an initial proof of concept to a production environment using AWS container services, high availability and failure redundancy, and concludes with the lessons learned and the next architectural challenges.

References:
https://dockerconeu2015.sched.com/list/descriptions/
https://www.slideshare.net/slideshow/using-docker-with-nosql/55437258

Transcript

1. Salamander: Using Docker with NoSQL Manuel E. de Paz Software Architect at BEEVA (a BBVA Company)
3. use #SalamanderAtDockerCon during the track to Q&A, Salamander’s Team it’s awaiting to answer you.
4. Some Ideas • Salamander, an adventure in AWS. • How does Docker helps us? • Our next Challenges
5. What is Salamander?
6. What is Salamander? Frontend, Backend & API Data ETL Batch Import native-drivers Java
7. Data Processing
8. ETL & Data Typologies Relationships 1 day ≅ 250.000 nodes
9. ETL & Data Typologies Data Properties 1 day ≅ 300.000 executions
10. Workflows and pipelines
11. DevOps Flow Deploy & Continuous Integration VPC EC2
12. Data Pipeline Process AWS S3 Java
13. Cloud Architecture
14. Proof of Concept EMR S3 EC2 Elastic Beanstalk
15. Data’s Growth MongoDB DB on Instance ReplicaSet High Availability ReplicaSet Horizontal Scaling Sharding + The data requirements force us to grown...
16. MicroData Neo4j DB on Instance DB on Instance Vertical Scaling ... Concept of MicroData = Isolation + Scaling DATA
17. Production Situation Identity & Access Management VPC ElastiCache EC2 EC2 Container Service S3 Glacier
18. Docker Challenge #1
19. Docker Challenge #1 1 Machine & 3x Container • AWS S3 Persistence, Virtually infinite • Docker Compose • Docker Registry • S3
20. Docker Challenge #1 cluster #2 Amazon S3 bucket cluster #3 EC2 instance contents
21. Docker Challenge #2
22. Docker Challenge #2 Cluster used for support Business Logic • High Availability • Fail Redundancy • AWS EC Services: Containers as a Service • + EC2 Container Service
23. Docker Challenge #2 EC2 Container Service AWS ECS Services EC2 instance DATA web app server EC2 instance contents Elastic Load Balancing DATA EC2 instance contents Cluster #2
24. Docker Challenge #3
25. Docker Challenge #3 Security & Stability Vs Flexibility • Neo4j WebGUI Browser • Stress Isolation • AWS EC2 Container: Container as a Service • EC2 Container Service
26. Docker Challenge #3 EC2 Container Service DATA ???????????? EC2 instance contents Cluster #3 EC2 instance EC2 instance REST server web app server Backend: Reverse Proxy Frontend: Web Proxy user
27. Next Challenges
28. Docker Challenge #4 AWS ECS Container: tasks DB as a Service Containers Auto organized proxy nodes EC2 Container Service Node.js Seaport: Cluster Discover & Management
29. New Challenges Private Registry DATA Amazon S3 bucket DATA EC2 instance EC2 instance EC2 instance REST server REST server web app server DATA ... DATA Orchestrator: Proxy & Mgmt cluster Node.js seaport Backend: REST Server user Frontend: WebApp
30. Next Challenges • Reduce the size of containers • AWS Private Registry Service • Replace RunDeck for AWS Lambda Lambda
31. Conclusions
32. Conclusions Docker help us to avoid a high complex database infrastructure • Data Segmentation • Isolation • Fine Tuning • Performance
33. Conclusions AWS EC2 Container Services API help us from a more complex architecture. • High Availability • Fail Redundancy • Management • Service Discovery • Reducing Costs
34. Requirements Some criteria that help us: • Immutable Data • Segmentable Info • NoSQL with Restful Driver © Albert Uderzo
35. DEMO Link to video
36. Our Team
37. References & Versions Name Technology Version REDIS Database NoSQL N/A / 2:3.0.5-3 Neo4j Database NoSQL 2.1.6 / 2.2 MongoDB Database NoSQL 2.6.11 Docker Registry Service 1.6 NodeJS App Development v4.0.0 / v0.12.2 / v0.10.29 AWS SDK App Development 2.0.31 Express App Development 4.0.0 AngularJS App Development 1.4.7
38. Image Credits & Attributions Image @ slide Link 7 https://www.flickr.com/photos/niznoz/3116766661 10 https://www.flickr.com/photos/fordeu/5709826282 13 https://unsplash.com/photos/ni9mKm62QnA 18 http://mrg.bz/xBONJ8 21 https://www.flickr.com/photos/cote/14692025435/ 24 https://flic.kr/p/imxgqh 27 https://unsplash.com/photos/ZXVk-NMWtgg 31 https://unsplash.com/photos/_r19nfvS3wY All product names, logos, brands, and other trademarks featured or referred in this presentation are the property of their respective trademark holders.
39. WE ARE RECRUITING!!! Feel free to contact us: rrhh@beeva.com hablemos@beeva.com www.beeva.com
40. Thank you! Manuel E. de Paz @manueldepaz manuel.depaz@beeva.com