Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Apache Conference (2015): http://sched.co/3zzs

This presentation will demonstrate how to leverage modern HTML5 technologies with the flexibility of Apache Ambari to build a comprehensive, responsive and attractive management interfaces for your Hadoop applications. In the process we will walk you through the reference implementation of an management interface for SQL-on-Hadoop application and integrate it with Apache Ambari. We will share our experience in using technologies like Google Polymer, Spring Boot and Apache Ambari.

Christian Tzolov

September 28, 2015
Tweet

More Decks by Christian Tzolov

Other Decks in Technology

Transcript

  1. Leveraging Ambari to Build
    Comprehensive Management UIs
    For Your Hadoop Applications
    by Christian Tzolov
    @christzolov

    View Slide

  2. Whoami
    Christian Tzolov
    Technical Architect at Pivotal,
    BigData, Hadoop, SpringXD,
    Apache Committer, Crunch PMC
    member
    [email protected]
    blog.tzolov.net
    @christzolov

    View Slide

  3. View Slide

  4. How To Provision, Manage
    And Monitor Enterprise
    Applications At Scale?

    View Slide

  5. Contents
    • Ambari Overview
    • Stacks and Services (demo)
    • Blueprints (demo)
    • Views (demo)
    • Metrics & Monitoring
    • On the Cloud

    View Slide

  6. Apache Ambari
    A platform to provision, manage and monitor
    Hadoop and Enterprise Applications

    View Slide

  7. Deployment Topology
    Ambari Web
    (REST API)
    Ambari Server
    admin node
    Ambari Agent
    cluster node 1
    Ambari Agent
    cluster node 2
    Ambari Agent
    cluster node N

    View Slide

  8. Special Powers
    Services & Stacks
    Blueprints
    Views
    Monitoring

    View Slide

  9. Services & Stacks

    View Slide

  10. Service Concepts
    • Stack - Set of (managed) Services and software
    packages for them
    • Service - Set of (distributed) Components
    • Component - Service building block of type
    Master, Slave, Client

    View Slide

  11. Service Architecture

    View Slide

  12. Service Layout
    |_ services
    |_
    metainfo.xml
    metrics.json
    |_ configuration
    {configuration files}
    |_ package
    |_ scripts
    {command scripts}
    • metainfo.xml - defines service,
    components and the
    command management
    scripts
    • configuration - service config
    files used in and deployed
    through the command scripts
    • package/scripts -
    commandScripts and
    templates.

    View Slide

  13. Metainfo.xml


















    CUST_MASTER
    Master
    MASTER
    1

    scripts/master.py
    PYTHON
    600


    ………

    View Slide

  14. Service / Command Script
    import sys
    from resource_management import *
    class Master(Script):
    def install(self, env):
    def stop(self, env):
    def start(self, env):
    def status(self, env):
    def configure(self, env):
    if __name__ == "__main__":
    Master().execute()
    import sys
    from resource_management import *
    class Slave(Script):
    def install(self, env):
    def stop(self, env):
    def start(self, env):
    def status(self, env):
    def configure(self, env):
    if __name__ == "__main__":
    Slave().execute()

    View Slide

  15. Service / Demo

    View Slide

  16. Ambari Blueprint

    View Slide

  17. Why?
    • Installation Automation
    • Declarative definition of a cluster
    • Specify a Stack, the Component layout and the
    Configurations to materialize a cluster
    • REST API

    View Slide

  18. Blueprint Flow
    Ambari Server
    node 1
    Cluster 1
    node 2 node N
    Cluster 2
    . . . .
    Cluster Z
    . . . .
    Blueprint
    (json)
    Host Mapping
    (json)
    REST API

    View Slide

  19. Blueprint Deploy
    { "Blueprints" : {
    "stack_name" : "PHD",
    "stack_version" : "3.0"
    },
    "configurations" : [
    { "hawq-site" : {
    "properties" : { "hawq.master.port" : "5532" }
    }
    }
    ],
    "host_groups" : [
    { "name" : "management_masters",
    "components" : [
    { "name" : "NAMENODE" },
    { "name" : "HAWQMASTER" }
    ],
    "cardinality" : "1"
    }, …… ]}
    POST /api/v1/blueprints/my-blueprint
    {
    "blueprint" : "hdfs-hawq-blueprint",
    "default_password" : "secret-password",
    "host_groups" : [
    {
    "name" : "management_masters",
    "hosts": [ { "fqdn":"ambari.localdomain" } ]
    },
    {
    "name" : "masters_workers",
    "hosts": [ { "fqdn":"phd1.localdomain" },
    { "fqdn":"phd2.localdomain" },
    { "fqdn":"phd3.localdomain" } ]
    }
    ]
    }
    POST /api/v1/clusters/my-cluster

    View Slide

  20. Blueprint REST API
    /api/v1/blueprint
    Cluster Logical Structure and
    configuration
    /api/v1/cluster
    Physical cluster mapping.
    Materializes a blueprint with real
    infrastructure
    /api/v1/clusters/?
    format=blueprint
    Exports blueprint for an existing
    cluster

    View Slide

  21. Blueprint Demo

    View Slide

  22. Ambari Views

    View Slide

  23. View Components
    Ambari Server
    Ambari Web
    Server Side
    (java)
    Client Side
    (js,html..)
    REST API
    VIEW PACKAGE (JAR)
    • Client Side - any client-side
    technologies
    • Server Side - Java application,
    Servlet, a REST service or
    ResourceProvider.
    • View Package - JAR bundles of
    the view definition and resources
    • View Context - Instance
    attributes and configuration
    context

    View Slide

  24. View Package
    view.xml

    index.html

    WEB-INF
    (lib/*.jar)

    View Slide

  25. View Context
    • On the Server Side
    • View-instance Config
    • View-instance Attributes
    • Execution context, authenticated user principal
    ViewContext
    getUsername() : String
    getViewName() : String
    getInstanceName() : String
    getProperties() : Map
    putInstanceData(…)
    getInstanceData(…) : String

    View Slide

  26. View Dashboard

    View Slide

  27. View REST API
    GET /api/v1/views List deployed views
    GET /api/v1/views/
    List all View
    instances
    GET /api/v1/views//versions/Version>
    Show specific View
    instance
    POST /api/v1/views//versions/Version>/instances/
    [{“ViewInstanceInfo":{ "properties" : {
    "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}]
    Create new View
    Instance

    View Slide

  28. Google Polymer
    https://www.polymer-project.org/1.0/

    View Slide

  29. View Demo
    https://github.com/tzolov/ambari-webpage-
    embedder-view
    https://youtu.be/qaR_6OBeKsQ

    View Slide

  30. Ambari Monitoring

    View Slide

  31. Ambari Metrics System
    Collect, Aggregate and Serve System Metrics
    • Metric Collector - Collects and Aggregates
    • Metric Monitor - System-level metrics
    • Metric Hadoop Sinks - Service-level metrics

    View Slide

  32. AMS Architecture

    View Slide

  33. Ambari & Cloud

    View Slide

  34. Virtualization & Cloud
    • Vagrant - PivotalHD
    (https://github.com/tzolov/
    vagrant-pivotalhd)
    • Cloudbreak
    (http://sequenceiq.com/
    cloudbreak/)

    View Slide

  35. Thank You!

    View Slide