Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Apache Conference (2015): http://sched.co/3zzs

This presentation will demonstrate how to leverage modern HTML5 technologies with the flexibility of Apache Ambari to build a comprehensive, responsive and attractive management interfaces for your Hadoop applications. In the process we will walk you through the reference implementation of an management interface for SQL-on-Hadoop application and integrate it with Apache Ambari. We will share our experience in using technologies like Google Polymer, Spring Boot and Apache Ambari.

Transcript

1. Leveraging Ambari to Build
   Comprehensive Management UIs
   For Your Hadoop Applications
   by Christian Tzolov
   @christzolov
   

   View Slide

2. Whoami
   Christian Tzolov
   Technical Architect at Pivotal,
   BigData, Hadoop, SpringXD,
   Apache Committer, Crunch PMC
   member
   ctzolov@pivotal.io
   blog.tzolov.net
   @christzolov
   

   View Slide

3. View Slide

4. How To Provision, Manage
   And Monitor Enterprise
   Applications At Scale?
   

   View Slide

5. Contents
   • Ambari Overview
   • Stacks and Services (demo)
   • Blueprints (demo)
   • Views (demo)
   • Metrics & Monitoring
   • On the Cloud
   

   View Slide

6. Apache Ambari
   A platform to provision, manage and monitor
   Hadoop and Enterprise Applications
   

   View Slide

7. Deployment Topology
   Ambari Web
   (REST API)
   Ambari Server
   admin node
   Ambari Agent
   cluster node 1
   Ambari Agent
   cluster node 2
   Ambari Agent
   cluster node N
   

   View Slide

8. Special Powers
   Services & Stacks
   Blueprints
   Views
   Monitoring
   

   View Slide

9. Services & Stacks
   

   View Slide

10. Service Concepts
    • Stack - Set of (managed) Services and software
    packages for them
    • Service - Set of (distributed) Components
    • Component - Service building block of type
    Master, Slave, Client
    

    View Slide

11. Service Architecture
    

    View Slide

12. Service Layout
    |_ services
    |_
    metainfo.xml
    metrics.json
    |_ configuration
    {configuration files}
    |_ package
    |_ scripts
    {command scripts}
    • metainfo.xml - defines service,
    components and the
    command management
    scripts
    • configuration - service config
    files used in and deployed
    through the command scripts
    • package/scripts -
    commandScripts and
    templates.
    

    View Slide

13. Metainfo.xml
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    CUST_MASTER
    Master
    MASTER
    1
    
    scripts/master.py
    PYTHON
    600
    
    
    ………
    
    

    View Slide

14. Service / Command Script
    import sys
    from resource_management import *
    class Master(Script):
    def install(self, env):
    def stop(self, env):
    def start(self, env):
    def status(self, env):
    def configure(self, env):
    if __name__ == "__main__":
    Master().execute()
    import sys
    from resource_management import *
    class Slave(Script):
    def install(self, env):
    def stop(self, env):
    def start(self, env):
    def status(self, env):
    def configure(self, env):
    if __name__ == "__main__":
    Slave().execute()
    

    View Slide

15. Service / Demo
    

    View Slide

16. Ambari Blueprint
    

    View Slide

17. Why?
    • Installation Automation
    • Declarative definition of a cluster
    • Specify a Stack, the Component layout and the
    Configurations to materialize a cluster
    • REST API
    

    View Slide

18. Blueprint Flow
    Ambari Server
    node 1
    Cluster 1
    node 2 node N
    Cluster 2
    . . . .
    Cluster Z
    . . . .
    Blueprint
    (json)
    Host Mapping
    (json)
    REST API
    

    View Slide

19. Blueprint Deploy
    { "Blueprints" : {
    "stack_name" : "PHD",
    "stack_version" : "3.0"
    },
    "configurations" : [
    { "hawq-site" : {
    "properties" : { "hawq.master.port" : "5532" }
    }
    }
    ],
    "host_groups" : [
    { "name" : "management_masters",
    "components" : [
    { "name" : "NAMENODE" },
    { "name" : "HAWQMASTER" }
    ],
    "cardinality" : "1"
    }, …… ]}
    POST /api/v1/blueprints/my-blueprint
    {
    "blueprint" : "hdfs-hawq-blueprint",
    "default_password" : "secret-password",
    "host_groups" : [
    {
    "name" : "management_masters",
    "hosts": [ { "fqdn":"ambari.localdomain" } ]
    },
    {
    "name" : "masters_workers",
    "hosts": [ { "fqdn":"phd1.localdomain" },
    { "fqdn":"phd2.localdomain" },
    { "fqdn":"phd3.localdomain" } ]
    }
    ]
    }
    POST /api/v1/clusters/my-cluster
    

    View Slide

20. Blueprint REST API
    /api/v1/blueprint
    Cluster Logical Structure and
    configuration
    /api/v1/cluster
    Physical cluster mapping.
    Materializes a blueprint with real
    infrastructure
    /api/v1/clusters/?
    format=blueprint
    Exports blueprint for an existing
    cluster
    

    View Slide

21. Blueprint Demo
    

    View Slide

22. Ambari Views
    

    View Slide

23. View Components
    Ambari Server
    Ambari Web
    Server Side
    (java)
    Client Side
    (js,html..)
    REST API
    VIEW PACKAGE (JAR)
    • Client Side - any client-side
    technologies
    • Server Side - Java application,
    Servlet, a REST service or
    ResourceProvider.
    • View Package - JAR bundles of
    the view definition and resources
    • View Context - Instance
    attributes and configuration
    context
    

    View Slide

24. View Package
    view.xml
    
    index.html
    
    WEB-INF
    (lib/*.jar)
    

    View Slide

25. View Context
    • On the Server Side
    • View-instance Config
    • View-instance Attributes
    • Execution context, authenticated user principal
    ViewContext
    getUsername() : String
    getViewName() : String
    getInstanceName() : String
    getProperties() : Map
    putInstanceData(…)
    getInstanceData(…) : String
    

    View Slide

26. View Dashboard
    

    View Slide

27. View REST API
    GET /api/v1/views List deployed views
    GET /api/v1/views/
    List all View
    instances
    GET /api/v1/views//versions/Version>
    Show specific View
    instance
    POST /api/v1/views//versions/Version>/instances/
    [{“ViewInstanceInfo":{ "properties" : {
    "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}]
    Create new View
    Instance
    

    View Slide

28. Google Polymer
    https://www.polymer-project.org/1.0/
    

    View Slide

29. View Demo
    https://github.com/tzolov/ambari-webpage-
    embedder-view
    https://youtu.be/qaR_6OBeKsQ
    

    View Slide

30. Ambari Monitoring
    

    View Slide

31. Ambari Metrics System
    Collect, Aggregate and Serve System Metrics
    • Metric Collector - Collects and Aggregates
    • Metric Monitor - System-level metrics
    • Metric Hadoop Sinks - Service-level metrics
    

    View Slide

32. AMS Architecture
    

    View Slide

33. Ambari & Cloud
    

    View Slide

34. Virtualization & Cloud
    • Vagrant - PivotalHD
    (https://github.com/tzolov/
    vagrant-pivotalhd)
    • Cloudbreak
    (http://sequenceiq.com/
    cloudbreak/)
    

    View Slide

35. Thank You!
    

    View Slide