Heeren Sharma

Data Collection and Processing using AWS together with Docker to orchestrate different services/application for data centric Solutions

Transcript

1. AWS + Docker Heeren Sharma Software Engineer - Machine Learning Cliqz GmbH https://cliqz.com/en
2. Agenda • A little about AWS • Case Study - Data Collection and processing • Docker (better say Fig) jumps in • Best of both worlds
3. AWS
4. Case Study • Data Processing pipeline • Simple Terms - Stream of incoming data, need to collect it (Intelligently), and process it further • Players - Data Providers, SQS, SNS, S3, EC2, EBS, Kinesis (for super advance use cases - optional)
5. How - Step By Step • SNS - Notification service -> Notification get Publish here • SQS - Queuing service -> Subscribed to SNS notifications • S3 - Storage service -> SQS messages point to S3 files • EBS - Hard Disk -> Store data “locally” for processing • EC2 - Computing machine -> Data Processing time • Store processed data on S3 bucket + publish its path in SNS notification
6. How - Data Flow
7. Good Practices • Kinesis vs SQS - Kinesis wins. Why? Message ordering, low latency, Multiple consumers and producers handling • Nomenclature of S3 data files - timestamp based • Based on streaming data, choose the resources.
8. Something Else ?
9. No more talking !!
10. How to realize Docker • Start building containers and in respect Docker images • Can couple various services - But try to create as much independent services in containers as possible. • Push them on S3 bucket or on your own docker registry • Pull docker images from registry, build them, make it up. :-)
11. Mentor Thomas Schroeter thschroeter@gmail.com @thschroeter @thschroeter!
12. Resources • Awsome talk about data processing - https://www.youtube.com/watch? v=yO3SBU6vVKA • Docker - https://www.docker.com/ • Docker Compose - https://docs.docker.com/compose/ • Fig - http://www.fig.sh/ - Legacy • For Mac Users - Boot2Docker - http://boot2docker.io/ • For advance data streaming - Amazon Kinesis - http://docs.aws.amazon.com/ kinesis/latest/dev/kinesis-sample-application.html • AWS - Google bitte!! • Email - heerensharma@yahoo.com
13. Wanna join? • Be part of something awesome. • https://cliqz.com/en/jobs#openings