Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Apache Conference (2015): http://sched.co/3zzs

This presentation will demonstrate how to leverage modern HTML5 technologies with the flexibility of Apache Ambari to build a comprehensive, responsive and attractive management interfaces for your Hadoop applications. In the process we will walk you through the reference implementation of an management interface for SQL-on-Hadoop application and integrate it with Apache Ambari. We will share our experience in using technologies like Google Polymer, Spring Boot and Apache Ambari.

Christian Tzolov

September 28, 2015
Tweet

More Decks by Christian Tzolov

Other Decks in Technology

Transcript

  1. Leveraging Ambari to Build
    Comprehensive Management UIs
    For Your Hadoop Applications
    by Christian Tzolov
    @christzolov

    View full-size slide

  2. Whoami
    Christian Tzolov
    Technical Architect at Pivotal,
    BigData, Hadoop, SpringXD,
    Apache Committer, Crunch PMC
    member
    [email protected]
    blog.tzolov.net
    @christzolov

    View full-size slide

  3. How To Provision, Manage
    And Monitor Enterprise
    Applications At Scale?

    View full-size slide

  4. Contents
    • Ambari Overview
    • Stacks and Services (demo)
    • Blueprints (demo)
    • Views (demo)
    • Metrics & Monitoring
    • On the Cloud

    View full-size slide

  5. Apache Ambari
    A platform to provision, manage and monitor
    Hadoop and Enterprise Applications

    View full-size slide

  6. Deployment Topology
    Ambari Web
    (REST API)
    Ambari Server
    admin node
    Ambari Agent
    cluster node 1
    Ambari Agent
    cluster node 2
    Ambari Agent
    cluster node N

    View full-size slide

  7. Special Powers
    Services & Stacks
    Blueprints
    Views
    Monitoring

    View full-size slide

  8. Services & Stacks

    View full-size slide

  9. Service Concepts
    • Stack - Set of (managed) Services and software
    packages for them
    • Service - Set of (distributed) Components
    • Component - Service building block of type
    Master, Slave, Client

    View full-size slide

  10. Service Architecture

    View full-size slide

  11. Service Layout
    |_ services
    |_
    metainfo.xml
    metrics.json
    |_ configuration
    {configuration files}
    |_ package
    |_ scripts
    {command scripts}
    • metainfo.xml - defines service,
    components and the
    command management
    scripts
    • configuration - service config
    files used in and deployed
    through the command scripts
    • package/scripts -
    commandScripts and
    templates.

    View full-size slide

  12. Metainfo.xml


















    CUST_MASTER
    Master
    MASTER
    1

    scripts/master.py
    PYTHON
    600


    ………

    View full-size slide

  13. Service / Command Script
    import sys
    from resource_management import *
    class Master(Script):
    def install(self, env):
    def stop(self, env):
    def start(self, env):
    def status(self, env):
    def configure(self, env):
    if __name__ == "__main__":
    Master().execute()
    import sys
    from resource_management import *
    class Slave(Script):
    def install(self, env):
    def stop(self, env):
    def start(self, env):
    def status(self, env):
    def configure(self, env):
    if __name__ == "__main__":
    Slave().execute()

    View full-size slide

  14. Service / Demo

    View full-size slide

  15. Ambari Blueprint

    View full-size slide

  16. Why?
    • Installation Automation
    • Declarative definition of a cluster
    • Specify a Stack, the Component layout and the
    Configurations to materialize a cluster
    • REST API

    View full-size slide

  17. Blueprint Flow
    Ambari Server
    node 1
    Cluster 1
    node 2 node N
    Cluster 2
    . . . .
    Cluster Z
    . . . .
    Blueprint
    (json)
    Host Mapping
    (json)
    REST API

    View full-size slide

  18. Blueprint Deploy
    { "Blueprints" : {
    "stack_name" : "PHD",
    "stack_version" : "3.0"
    },
    "configurations" : [
    { "hawq-site" : {
    "properties" : { "hawq.master.port" : "5532" }
    }
    }
    ],
    "host_groups" : [
    { "name" : "management_masters",
    "components" : [
    { "name" : "NAMENODE" },
    { "name" : "HAWQMASTER" }
    ],
    "cardinality" : "1"
    }, …… ]}
    POST /api/v1/blueprints/my-blueprint
    {
    "blueprint" : "hdfs-hawq-blueprint",
    "default_password" : "secret-password",
    "host_groups" : [
    {
    "name" : "management_masters",
    "hosts": [ { "fqdn":"ambari.localdomain" } ]
    },
    {
    "name" : "masters_workers",
    "hosts": [ { "fqdn":"phd1.localdomain" },
    { "fqdn":"phd2.localdomain" },
    { "fqdn":"phd3.localdomain" } ]
    }
    ]
    }
    POST /api/v1/clusters/my-cluster

    View full-size slide

  19. Blueprint REST API
    /api/v1/blueprint
    Cluster Logical Structure and
    configuration
    /api/v1/cluster
    Physical cluster mapping.
    Materializes a blueprint with real
    infrastructure
    /api/v1/clusters/?
    format=blueprint
    Exports blueprint for an existing
    cluster

    View full-size slide

  20. Blueprint Demo

    View full-size slide

  21. Ambari Views

    View full-size slide

  22. View Components
    Ambari Server
    Ambari Web
    Server Side
    (java)
    Client Side
    (js,html..)
    REST API
    VIEW PACKAGE (JAR)
    • Client Side - any client-side
    technologies
    • Server Side - Java application,
    Servlet, a REST service or
    ResourceProvider.
    • View Package - JAR bundles of
    the view definition and resources
    • View Context - Instance
    attributes and configuration
    context

    View full-size slide

  23. View Package
    view.xml

    index.html

    WEB-INF
    (lib/*.jar)

    View full-size slide

  24. View Context
    • On the Server Side
    • View-instance Config
    • View-instance Attributes
    • Execution context, authenticated user principal
    ViewContext
    getUsername() : String
    getViewName() : String
    getInstanceName() : String
    getProperties() : Map
    putInstanceData(…)
    getInstanceData(…) : String

    View full-size slide

  25. View Dashboard

    View full-size slide

  26. View REST API
    GET /api/v1/views List deployed views
    GET /api/v1/views/
    List all View
    instances
    GET /api/v1/views//versions/Version>
    Show specific View
    instance
    POST /api/v1/views//versions/Version>/instances/
    [{“ViewInstanceInfo":{ "properties" : {
    "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}]
    Create new View
    Instance

    View full-size slide

  27. Google Polymer
    https://www.polymer-project.org/1.0/

    View full-size slide

  28. View Demo
    https://github.com/tzolov/ambari-webpage-
    embedder-view
    https://youtu.be/qaR_6OBeKsQ

    View full-size slide

  29. Ambari Monitoring

    View full-size slide

  30. Ambari Metrics System
    Collect, Aggregate and Serve System Metrics
    • Metric Collector - Collects and Aggregates
    • Metric Monitor - System-level metrics
    • Metric Hadoop Sinks - Service-level metrics

    View full-size slide

  31. AMS Architecture

    View full-size slide

  32. Ambari & Cloud

    View full-size slide

  33. Virtualization & Cloud
    • Vagrant - PivotalHD
    (https://github.com/tzolov/
    vagrant-pivotalhd)
    • Cloudbreak
    (http://sequenceiq.com/
    cloudbreak/)

    View full-size slide