Hadoop in AWS

How to use Hadoop for old docker deletion? Easy!

Transcript

1. Hadoop in AWS You can already use it without knowing about it
2. SQLSat Kyiv Team Eugene Polonichko Oksana Tkach Oksana Borysenko Denis Reznik Mykola Pobyivovk Yevhen Nedashkivskyi Anton Artomov
3. Sponsor Sessions are at 12:30 and 13:00  Don’t miss them, they might be providing some interesting and valuable information! 12:30 Congress Hall DevArt 12:30 Conference Hall Infopulse 12:30 Room AC Materialize 13:00 Congress Hall DB Best 13:00 Conference Hall Eleks
4. Sponsors
5. Session will begin very soon :)  Please complete the electronic evaluation form for this session and for the event. Your feedback will help us to improve future conferences and speakers will appreciate your feedback!  Enjoy the conference!
6. About the speaker Sergey Dzyuban Intapp LAMP admin 2003 2007 2010 2014 2016 Web Developer Web Developer .NET Developer .NET Developer Technical Account Manager Technical Account Manager Team Lead Team Lead DevOps DevOps
7. Part 1. Newcomer adaptation My first independent task in the company
8. Let’s delete something on prod ! Review artifacts Analyze space usage Find unused files Delete unused file
9. Quick review  Different types of feeds  Artifacts produced by few CIs  Deployment to different target platforms  2.7 Tb total size
10. Artifactory – the real picture
11. Artifactory GUI … provides some basic Storage Summary info
12. Artifactory GUI … provides some basic Storage Summary info
13. Initial analysis GET /api/storage/libs-release-local/…/lib-ver.pom { "uri": "http://localhost:8081/.../lib-ver.pom", "downloadUri": "http://localhost:8081/.../lib-ver.pom", "repo": "libs-release-local", "path": "/org/acme/lib/ver/lib-ver.pom", "remoteUrl": "http://some-remote-repo/.../lib-ver.pom", "created": ISO8601 (yyyy-MM-dd'T'HH:mm:ss.SSSZ), "createdBy": "userY", "lastModified": ISO8601 (yyyy-MM-dd'T'HH:mm:ss.SSSZ), "modifiedBy": "userX", "lastUpdated": ISO8601 (yyyy-MM-dd'T'HH:mm:ss.SSSZ), "size": "1024", //bytes "mimeType": "application/pom+xml", "checksums": { "md5" : string, "sha1" : string, "sha256" : string }, "originalChecksums":{ "md5" : string, "sha1" : string, "sha256" : string } } GET /api/storage/libs-release-local/…/lib-ver.pom { "uri": "http://localhost:8081/.../lib-ver.pom ", "lastDownloaded": Timestamp (ms), "downloadCount": 1337, "lastDownloadedBy": "user1" }
14. Analyzing the data ES kibana
19. Initial issue – long download time ES kibana s3 15 min
20. Initial results  Total count – 360.329  Total size – 2.480 Tb  Docker and time repos – 99% of size  Docker – 90% of downloads  Half of files never were downloaded  Produce speed – 300Gb/Month  No extremely large files found  No extremely large products found
21. Data
22. Data structure image 1.0.0 1.0.1 sha_xxxxxxxxxxxxxxx_1 sha_xxxxxxxxxxxxxxx_2 sha_xxxxxxxxxxxxxxx_3 sha_xxxxxxxxxxxxxxx_1 sha_xxxxxxxxxxxxxxx_2 sha_xxxxxxxxxxxxxxx_4 sha_xxxxxxxxxxxxxxx_5 created: 01-01-2016 updated: 01-02-2017 downloads: 100 size: 100Mb created: 01-01-2016 updated: 01-02-2017 downloaded: 01-2019 downloads: 0 size: 120Mb created: 01-01-2016 updated: 01-02-2017 downloaded: null downloads: 20 size: 30Mb created: 01-01-2016 updated: 01-02-2017 downloaded: 05-2019
23. Aggregations image 1.0.0 1.0.1 sha_xxxxxxxxxxxxxxx_1 sha_xxxxxxxxxxxxxxx_2 sha_xxxxxxxxxxxxxxx_3 sha_xxxxxxxxxxxxxxx_1 sha_xxxxxxxxxxxxxxx_2 sha_xxxxxxxxxxxxxxx_4 downloads: • sum: 120 • max: 100 • min: 0 size: 250Mb created: 01-01-2016 updated: 01-02-2017 downloaded: 05-2019 downloads: 100 size: 100Mb created: 01-01-2016 updated: 01-02-2017 downloaded: 01-2019 downloads: 0 size: 120Mb created: 01-01-2016 updated: 01-02-2017 downloaded: null downloads: 20 size: 30Mb created: 01-01-2016 updated: 01-02-2017 downloaded: 05-2019
24. Data aggregation ES kibana s3 aggregate
25. New index with calculated fields: images
26. Kibana limitations  Slow calculated fields  AWS ES has no export feature  Specific query language  REST API for advanced stuff like data export  Specific query and Group By syntax
27. Part 2. SQL … let’s try to use some regular approach
28. Stack Overflow development
29. AWS Athena  ANSI-standard SQL  based on the Presto distributed SQL engine  Serverless  File formats: JSON, CSV, log files, text with custom delimiters, Apache Parquet, and Apache ORC  $5.00 per TB of data scanned
31. AWS Athena reads from S3 ES s3 Kibana Athena
32. DEMO 1. AWS Athena
33. AWS Athena – how it works ES s3 Kibana Athena
34. Presto
35. Connectors
36. org.apache.hive…
37. Part 3. Let’s Hadoop … or how big is your data
38. Amazon Elastic Map Reduce  Easy to use  Low cost  Flexible  Elastic  Secure
39. AWS EMR – S3 data source ES s3 Kibana Athena EMR Presto
40. AWS EMR – time to deploy
41. EMR Deployment  To save the costs – use Spot Instances  Nodes are simple EC2 machines  Nodes can be accessed by SSH
42. Spot Instances
43. AWS CLI aws emr create-cluster --name "sql-saturday" --profile "emr-dzyuban" --configurations file://presto.config.json --release-label emr-5.20.0 --use-default- roles --ec2-attributes KeyName=Intapp --applications Name=Hadoop Name=Spark Name=Hive Name=PRESTO Name=HUE Name=ZEPPELIN Name=Ganglia Name=Pig Name=Sqoop --instance-fleets=file://fleet.json [ { "Classification": "presto-connector-hive", "Properties": { "hive.metastore": "glue", "hive.metastore.glue.datacatalog.enabled": "true" } }, { "Classification": "spark-hive-site", "Properties": { "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory" } } ] presto.config.json
44. EMR components
45. EMR components Component Description HDFS Hadoop Distributed File System: provides high-throughput access to application data based on commodity hardware YARN Yet Another Resource Negotiator: a framework for cluster resource management including job scheduling MapReduce Software framework for parallel processing of large data sets based on YARN Hive Data warehouse system based on Hadoop Spark Cluster computing framework that utilizes YARN and HDFS. ZooKeeper Distributed name registry, synchronization service and configuration service that is used as a sub-system in Hadoop
46. DEMO 2. Resources
47. EMR URLs YARN ResourceManager http://ec2-3-86-62-50.compute-1.amazonaws.com:8088/ Hadoop HDFS NameNode http://ec2-3-86-62-50.compute-1.amazonaws.com:50070/ Spark HistoryServer http://ec2-3-86-62-50.compute-1.amazonaws.com:18080/ Zeppelin http://ec2-3-86-62-50.compute-1.amazonaws.com:8890/ Hue http://ec2-3-86-62-50.compute-1.amazonaws.com:8888/ Ganglia http://ec2-3-86-62-50.compute-1.amazonaws.com/ganglia/
48. DEMO 3. Zeppelin
49. rm -rf / So, what’s about Artifactory ?
50. Zeppelin to do the work s3 Athena EMR Presto Presto Zeppelin Glue remove
51. References  https://aws.amazon.com/athena/ AWS Athena  http://prestodb.github.io/overview.html Presto Overview  https://www.awsgeek.com/posts/Amazon-Athena/ funny graphics from Jerry Hargrove  https://aws.amazon.com/emr/ Amazon official EMR page  https://www.amazon.com/Hadoop-Dummies-Dirk-deRoos/dp/1118607554 book  https://hadoop.apache.org Apache Hadoop official page  http://zeppelin.apache.org Apache Zeppelin official page  https://flightaware.com/ - thanks for nice photos
52. Thank You !!! Questions…