Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Apache Conference (2015): http://sched.co/3zzs

This presentation will demonstrate how to leverage modern HTML5 technologies with the flexibility of Apache Ambari to build a comprehensive, responsive and attractive management interfaces for your Hadoop applications. In the process we will walk you through the reference implementation of an management interface for SQL-on-Hadoop application and integrate it with Apache Ambari. We will share our experience in using technologies like Google Polymer, Spring Boot and Apache Ambari.

B53016292e87bca26da88fc940070c4f?s=128

Christian Tzolov

September 28, 2015
Tweet

Transcript

  1. Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop

    Applications by Christian Tzolov @christzolov
  2. Whoami Christian Tzolov Technical Architect at Pivotal, BigData, Hadoop, SpringXD,

    Apache Committer, Crunch PMC member ctzolov@pivotal.io blog.tzolov.net @christzolov
  3. None
  4. How To Provision, Manage And Monitor Enterprise Applications At Scale?

  5. Contents • Ambari Overview • Stacks and Services (demo) •

    Blueprints (demo) • Views (demo) • Metrics & Monitoring • On the Cloud
  6. Apache Ambari A platform to provision, manage and monitor Hadoop

    and Enterprise Applications
  7. Deployment Topology Ambari Web (REST API) Ambari Server admin node

    Ambari Agent cluster node 1 Ambari Agent cluster node 2 Ambari Agent cluster node N
  8. Special Powers Services & Stacks Blueprints Views Monitoring

  9. Services & Stacks

  10. Service Concepts • Stack - Set of (managed) Services and

    software packages for them • Service - Set of (distributed) Components • Component - Service building block of type Master, Slave, Client
  11. Service Architecture

  12. Service Layout |_ services |_ <service_name> metainfo.xml metrics.json |_ configuration

    {configuration files} |_ package |_ scripts {command scripts} • metainfo.xml - defines service, components and the command management scripts • configuration - service config files used in and deployed through the command scripts • package/scripts - commandScripts and templates.
  13. Metainfo.xml <metainfo> <schemaVersion/> <services> <service> <name/> <displayName/> <comment/> <version/> <components/>

    <osSpecifics/> <commandScript/> <requiredServices/> <configuration-dependencies/> </service> </services> </metainfo> <components> <component> <name>CUST_MASTER</name> <displayName>Master</displayName> <category>MASTER</category> <cardinality>1</cardinality> <commandScript> <script>scripts/master.py</script> <scriptType>PYTHON</scriptType> <timeout>600</timeout> </commandScript> </component> ……… </components>
  14. Service / Command Script import sys from resource_management import *

    class Master(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Master().execute() import sys from resource_management import * class Slave(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Slave().execute()
  15. Service / Demo

  16. Ambari Blueprint

  17. Why? • Installation Automation • Declarative definition of a cluster

    • Specify a Stack, the Component layout and the Configurations to materialize a cluster • REST API
  18. Blueprint Flow Ambari Server node 1 Cluster 1 node 2

    node N Cluster 2 . . . . Cluster Z . . . . Blueprint (json) Host Mapping (json) REST API
  19. Blueprint Deploy { "Blueprints" : { "stack_name" : "PHD", "stack_version"

    : "3.0" }, "configurations" : [ { "hawq-site" : { "properties" : { "hawq.master.port" : "5532" } } } ], "host_groups" : [ { "name" : "management_masters", "components" : [ { "name" : "NAMENODE" }, { "name" : "HAWQMASTER" } ], "cardinality" : "1" }, …… ]} POST /api/v1/blueprints/my-blueprint { "blueprint" : "hdfs-hawq-blueprint", "default_password" : "secret-password", "host_groups" : [ { "name" : "management_masters", "hosts": [ { "fqdn":"ambari.localdomain" } ] }, { "name" : "masters_workers", "hosts": [ { "fqdn":"phd1.localdomain" }, { "fqdn":"phd2.localdomain" }, { "fqdn":"phd3.localdomain" } ] } ] } POST /api/v1/clusters/my-cluster
  20. Blueprint REST API /api/v1/blueprint Cluster Logical Structure and configuration /api/v1/cluster

    Physical cluster mapping. Materializes a blueprint with real infrastructure /api/v1/clusters/<cluster name>? format=blueprint Exports blueprint for an existing cluster
  21. Blueprint Demo

  22. Ambari Views

  23. View Components Ambari Server Ambari Web Server Side (java) Client

    Side (js,html..) REST API VIEW PACKAGE (JAR) • Client Side - any client-side technologies • Server Side - Java application, Servlet, a REST service or ResourceProvider. • View Package - JAR bundles of the view definition and resources • View Context - Instance attributes and configuration context
  24. View Package view.xml <server packages + classes> index.html <supporting UI

    assets> WEB-INF (lib/*.jar)
  25. View Context • On the Server Side • View-instance Config

    • View-instance Attributes • Execution context, authenticated user principal ViewContext getUsername() : String getViewName() : String getInstanceName() : String getProperties() : Map<String,String> putInstanceData(…) getInstanceData(…) : String
  26. View Dashboard

  27. View REST API GET /api/v1/views List deployed views GET /api/v1/views/<View

    Name> List all View instances GET /api/v1/views/<View Name>/versions/<View Version> Show specific View instance POST /api/v1/views/<View Name>/versions/<View Version>/instances/<Instance Name> [{“ViewInstanceInfo":{ "properties" : { "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}] Create new View Instance
  28. Google Polymer https://www.polymer-project.org/1.0/

  29. View Demo https://github.com/tzolov/ambari-webpage- embedder-view https://youtu.be/qaR_6OBeKsQ

  30. Ambari Monitoring

  31. Ambari Metrics System Collect, Aggregate and Serve System Metrics •

    Metric Collector - Collects and Aggregates • Metric Monitor - System-level metrics • Metric Hadoop Sinks - Service-level metrics
  32. AMS Architecture

  33. Ambari & Cloud

  34. Virtualization & Cloud • Vagrant - PivotalHD (https://github.com/tzolov/ vagrant-pivotalhd) •

    Cloudbreak (http://sequenceiq.com/ cloudbreak/)
  35. Thank You!