Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Apache Conference (2015): http://sched.co/3zzs

This presentation will demonstrate how to leverage modern HTML5 technologies with the flexibility of Apache Ambari to build a comprehensive, responsive and attractive management interfaces for your Hadoop applications. In the process we will walk you through the reference implementation of an management interface for SQL-on-Hadoop application and integrate it with Apache Ambari. We will share our experience in using technologies like Google Polymer, Spring Boot and Apache Ambari.

Transcript

1. Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications by Christian Tzolov @christzolov
2. Whoami Christian Tzolov Technical Architect at Pivotal, BigData, Hadoop, SpringXD, Apache Committer, Crunch PMC member ctzolov@pivotal.io blog.tzolov.net @christzolov
4. How To Provision, Manage And Monitor Enterprise Applications At Scale?
5. Contents • Ambari Overview • Stacks and Services (demo) • Blueprints (demo) • Views (demo) • Metrics & Monitoring • On the Cloud
6. Apache Ambari A platform to provision, manage and monitor Hadoop and Enterprise Applications
7. Deployment Topology Ambari Web (REST API) Ambari Server admin node Ambari Agent cluster node 1 Ambari Agent cluster node 2 Ambari Agent cluster node N
8. Special Powers Services & Stacks Blueprints Views Monitoring
9. Services & Stacks
10. Service Concepts • Stack - Set of (managed) Services and software packages for them • Service - Set of (distributed) Components • Component - Service building block of type Master, Slave, Client
11. Service Architecture
12. Service Layout |_ services |_ <service_name> metainfo.xml metrics.json |_ configuration {configuration files} |_ package |_ scripts {command scripts} • metainfo.xml - defines service, components and the command management scripts • configuration - service config files used in and deployed through the command scripts • package/scripts - commandScripts and templates.
13. Metainfo.xml <metainfo> <schemaVersion/> <services> <service> <name/> <displayName/> <comment/> <version/> <components/> <osSpecifics/> <commandScript/> <requiredServices/> <configuration-dependencies/> </service> </services> </metainfo> <components> <component> <name>CUST_MASTER</name> <displayName>Master</displayName> <category>MASTER</category> <cardinality>1</cardinality> <commandScript> <script>scripts/master.py</script> <scriptType>PYTHON</scriptType> <timeout>600</timeout> </commandScript> </component> ……… </components>
14. Service / Command Script import sys from resource_management import * class Master(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Master().execute() import sys from resource_management import * class Slave(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Slave().execute()
15. Service / Demo
16. Ambari Blueprint
17. Why? • Installation Automation • Declarative definition of a cluster • Specify a Stack, the Component layout and the Configurations to materialize a cluster • REST API
18. Blueprint Flow Ambari Server node 1 Cluster 1 node 2 node N Cluster 2 . . . . Cluster Z . . . . Blueprint (json) Host Mapping (json) REST API
19. Blueprint Deploy { "Blueprints" : { "stack_name" : "PHD", "stack_version" : "3.0" }, "configurations" : [ { "hawq-site" : { "properties" : { "hawq.master.port" : "5532" } } } ], "host_groups" : [ { "name" : "management_masters", "components" : [ { "name" : "NAMENODE" }, { "name" : "HAWQMASTER" } ], "cardinality" : "1" }, …… ]} POST /api/v1/blueprints/my-blueprint { "blueprint" : "hdfs-hawq-blueprint", "default_password" : "secret-password", "host_groups" : [ { "name" : "management_masters", "hosts": [ { "fqdn":"ambari.localdomain" } ] }, { "name" : "masters_workers", "hosts": [ { "fqdn":"phd1.localdomain" }, { "fqdn":"phd2.localdomain" }, { "fqdn":"phd3.localdomain" } ] } ] } POST /api/v1/clusters/my-cluster
20. Blueprint REST API /api/v1/blueprint Cluster Logical Structure and configuration /api/v1/cluster Physical cluster mapping. Materializes a blueprint with real infrastructure /api/v1/clusters/<cluster name>? format=blueprint Exports blueprint for an existing cluster
21. Blueprint Demo
22. Ambari Views
23. View Components Ambari Server Ambari Web Server Side (java) Client Side (js,html..) REST API VIEW PACKAGE (JAR) • Client Side - any client-side technologies • Server Side - Java application, Servlet, a REST service or ResourceProvider. • View Package - JAR bundles of the view definition and resources • View Context - Instance attributes and configuration context
24. View Package view.xml <server packages + classes> index.html <supporting UI assets> WEB-INF (lib/*.jar)
25. View Context • On the Server Side • View-instance Config • View-instance Attributes • Execution context, authenticated user principal ViewContext getUsername() : String getViewName() : String getInstanceName() : String getProperties() : Map<String,String> putInstanceData(…) getInstanceData(…) : String
26. View Dashboard
27. View REST API GET /api/v1/views List deployed views GET /api/v1/views/<View Name> List all View instances GET /api/v1/views/<View Name>/versions/<View Version> Show specific View instance POST /api/v1/views/<View Name>/versions/<View Version>/instances/<Instance Name> [{“ViewInstanceInfo":{ "properties" : { "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}] Create new View Instance
28. Google Polymer https://www.polymer-project.org/1.0/
29. View Demo https://github.com/tzolov/ambari-webpage- embedder-view https://youtu.be/qaR_6OBeKsQ
30. Ambari Monitoring
31. Ambari Metrics System Collect, Aggregate and Serve System Metrics • Metric Collector - Collects and Aggregates • Metric Monitor - System-level metrics • Metric Hadoop Sinks - Service-level metrics
32. AMS Architecture
33. Ambari & Cloud
34. Virtualization & Cloud • Vagrant - PivotalHD (https://github.com/tzolov/ vagrant-pivotalhd) • Cloudbreak (http://sequenceiq.com/ cloudbreak/)
35. Thank You!