Floating elephants: Developing data wrangling systems on Docker

If you are building applications for the big data world, you’ll need access to a variety of big data platform configurations during development and continuous integration. These deployments need to scale down to an individual laptop or scale up to a realistic test cluster, and they need to be interchangeable with quick setup and teardown. Chad Metcalf and Seshadri Mahalingam explore several strategies for meeting this challenge by containerizing and deploying Hadoop services in Docker. Chad and Seshadri share lessons learned from solutions that they built, some which have been open sourced, and dig into topics including how to manage the images lifecycle, configuration, persistent data, multihost networking with Docker Engine and Swarm, and creating different deployment environments with Docker Machine.

http://conferences.oreilly.com/strata/hadoop-big-data-eu/public/schedule/detail/49727

Transcript

1. Floating Elephants Seshadri Mahalingam Developing data wrangling systems on Docker
2. Applications on Hadoop What do we need? Docker: Containerizing Environments Floating Elephants: An implementation Outline 1 2 3 4
3. Applications on Hadoop 3 QUESTION ANALYZE INSIGHT
4. Applications on Hadoop, eg. Trifacta 4 QUESTION ANALYZE INSIGHT DISCOVER STRUCTURE CLEANSE ENRICH VALIDATE PUBLISH DATA WRANGLING
5. Applications on Hadoop, eg. Trifacta
6. Applications on Hadoop, eg. Trifacta
7. Applications on Hadoop, eg. Trifacta Sample-to-Scale
8. Applications on Hadoop, eg. Trifacta Sample-to-Scale Execution
9. Applications on Hadoop, eg. Trifacta Sample-to-Scale On-demand Profiling Execution
10. Applications on Hadoop, eg. Trifacta Sample-to-Scale On-demand Profiling Execution Metadata Services
11. What is Hadoop?
12. Same Application…Same Hadoop? …
13. Same Application, Different Hadoop …
14. Same Application, Different Hadoop …
15. Challenges
16. Challenges Security Vast majority of customer issues due to security misconfiguration
17. Challenges Vendor + Version Differences Maintaining compatibility across API changes Security Vast majority of customer issues due to security misconfiguration
18. Challenges Reproducibility Recreating identical Hadoop environments with vendor tools is challenging Vendor + Version Differences Maintaining compatibility across API changes Security Vast majority of customer issues due to security misconfiguration
19. Challenges Reproducibility Recreating identical Hadoop environments with vendor tools is challenging Vendor + Version Differences Maintaining compatibility across API changes Security Vast majority of customer issues due to security misconfiguration Test in Continuous Integration
20. Applications on Hadoop What do we need? Docker: Containerizing Environments Floating Elephants: An implementation Outline 2 1 3 4
21. Dramatis Personae
22. Dramatis Personae Customers
23. Dramatis Personae Customers Sales/Solutions Engineers
24. Dramatis Personae Customers Sales/Solutions Engineers Application Developers
25. Dramatis Personae Customers Sales/Solutions Engineers Application Developers DevOps, Build & Release Engineers
26. Dramatis Personae
27. Dramatis Personae
28. Dramatis Personae Customers Sales/Solutions Engineers
29. Dramatis Personae Customers Sales/Solutions Engineers Application Developers DevOps, Build & Release Engineers
30. Goals
31. Goals Bootstrap Developers Quickly
32. Goals Bootstrap Developers Quickly Test the Matrix
33. Goals Bootstrap Developers Quickly Reproduce Customer Environments Test the Matrix
34. Non-Goals Performance Scale
35. “Bundle JAR” Approach
36. “Bundle JAR” Approach
37. “Bundle JAR” Approach https://github.com/johnrengelman/shadow
38. “Bundle JAR” Approach https://github.com/johnrengelman/shadow
39. Goals
40. Goals Bootstrap Developers Quickly
41. Goals Bootstrap Developers Quickly Test the Matrix
42. Goals Bootstrap Developers Quickly Reproduce Customer Environments Test the Matrix
43. Goals Bootstrap Developers Quickly Test the Matrix Reproduce Customer Environments We need a “real” cluster.
44. Strategies Virtual Machines? Tied to specific infrastructure Shared Cluster? Lack of isolation too chaotic
45. Applications on Hadoop What do we need? Docker: Containerizing Environments Floating Elephants: An implementation Outline 2 1 3 4
47. Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.
48. Docker Containers
49. Docker Containers Clear Definition Dockerfile contains all the commands used to assemble an image
50. Docker Containers Clear Definition Dockerfile contains all the commands used to assemble an image Portable Runs on practically any modern computer, on any infrastructure
51. Docker Containers Clear Definition Dockerfile contains all the commands used to assemble an image Portable Runs on practically any modern computer, on any infrastructure Inheritance Add layers to existing images for more services
52. Docker Containers Clear Definition Dockerfile contains all the commands used to assemble an image Portable Runs on practically any modern computer, on any infrastructure Inheritance Add layers to existing images for more services
59. Hadoop Configuration
60. Hadoop Configuration Data-only Containers Mounted Volumes
61. Security - Kerberos
62. Security - Kerberos Permissions & Directory Structure
63. Security - Kerberos Permissions & Directory Structure User & Group Configuration
64. Security - Kerberos Permissions & Directory Structure User & Group Configuration
65. Security - Kerberos Permissions & Directory Structure User & Group Configuration https://hub.docker.com/r/adelton/freeipa-server/
66. Goals
67. Goals Bootstrap Developers Quickly
68. Goals Bootstrap Developers Quickly Test the Matrix
69. Goals Bootstrap Developers Quickly Reproduce Customer Environments Test the Matrix
70. Complexity - Layers of Services 29
71. Complexity - Layers of Services 29 Base (Java, System deps)
72. Complexity - Layers of Services 29 Base (Java, System deps) HDFS, YARN
73. Complexity - Layers of Services 29 Base (Java, System deps) HDFS, YARN Spark
74. Complexity - Layers of Services 29 Base (Java, System deps) HDFS, YARN Spark Hive
75. Complexity - Layers of Services 29 Base (Java, System deps) HDFS, YARN Spark Hive …
76. Complexity - Layers of Services 29 Base (Java, System deps) HDFS, YARN Spark Hive … Easy to add incremental services
77. Complexity - Layers of Services 29 Base (Java, System deps) HDFS, YARN Spark Hive … Easy to add incremental services Difficult to change lower layers
78. Complexity - Layers of Services 29 Base (Java, System deps) HDFS, YARN Spark Hive … Easy to add incremental services Difficult to change lower layers Code smell: Mirrors underlying implementation without abstraction
79. Complexity - Layers of Services 29 Base (Java, System deps) HDFS, YARN Spark Hive … Easy to add incremental services Difficult to change lower layers Code smell: Mirrors underlying implementation without abstraction Bloated Dockerfile hard to understand
80. Applications on Hadoop What do we need? Docker: Containerizing Environments Floating Elephants: An implementation Outline 2 1 3 4
81. Service-level Containers 31 Globe by Bluetip Design from the Noun Project
82. Service-level Containers 31 Globe by Bluetip Design from the Noun Project
83. Service-level Containers 31 Globe by Bluetip Design from the Noun Project
84. Docker Networking 32 Globe by Bluetip Design from the Noun Project
85. Docker Networking 32 Container = Virtual Node Globe by Bluetip Design from the Noun Project
86. Docker Networking 32 Container = Virtual Node Attach nodes to networks Globe by Bluetip Design from the Noun Project
87. Docker Networking 32 Container = Virtual Node Attach nodes to networks Globe by Bluetip Design from the Noun Project
88. Docker Networking 32 Container = Virtual Node Attach nodes to networks Globe by Bluetip Design from the Noun Project
89. DNS Resolution 33
90. DNS Resolution 33 dnsmasq proxy
91. DNS Resolution 33 dnsmasq proxy
92. DNS Resolution 33 dnsmasq proxy
93. More features with Docker Networking 34 Multi-host Overlay networks with Docker High Availability Redundant NameNode, ResourceManager, ZooKeeper
94. Floating Elephants https://github.com/trifacta/floating-elephants
95. Floating Elephants https://github.com/trifacta/floating-elephants Fork m e on G itH ub
96. Goals Bootstrap Developers Quickly Reproduce Customer Environments Test the Matrix
97. Questions? @seshness
98. 38 See a Demo at Booth #309 Download Trifacta Wrangler for Free trifacta.com/start-wrangling