Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HBaseCon 2016 West - Containerizing Apache HBase Clusters

HBaseCon 2016 West - Containerizing Apache HBase Clusters

At Facebook, all production HBase clusters run in a containerized environment, with every daemon running inside its own LXC container. Containerization allows us to ensure isolation between services running on the same host and simplify operations, but sometimes abstractions leak and problems can't be addressed inside the container. In this talk, we will discuss how Facebook runs HBase as a stateful service inside containers and we will discuss some of the issues we've found when doing so.

https://hbase.apache.org/hbasecon-archives.html#HBaseCon_West_2016

Javier Maestro

May 24, 2016
Tweet

More Decks by Javier Maestro

Other Decks in Technology

Transcript

  1. Containerizing Apache HBase
    Clusters
    David Pope & Javier Maestro
    Production Engineers - HBase

    View Slide

  2. Container Overview

    View Slide

  3. View Slide

  4. 1964
    IBM 360
    1979
    UNIX
    chroot
    1982
    BSD
    chroot
    1999
    FreeBSD
    jail
    2013
    Docker
    2007-8
    cgroups
    LXC

    View Slide

  5. Container Platforms

    View Slide

  6. View Slide

  7. HBase Containers @ Facebook

    View Slide

  8. LEASE
    BUY

    View Slide

  9. Buy vs. Lease
    • Our scale
    • Timing
    • Control / Full Ownership
    • Financial
    Infrastructure
    (os, network)
    Container Platform
    (Tupperware)
    Application
    Services
    Physical
    (data center, hardware)

    View Slide

  10. Tupperware Overview
    scheduler
    server db
    host3
    host4
    host1
    host2
    config.tw

    View Slide

  11. Tupperware Spec

    View Slide

  12. Tupperware Benefits
    • Configuration Spec
    • Deployments
    • Scheduler
    • Health Monitor
    • Logging
    • Canary
    • Web UI / CLI / API
    • Elasticity (auto-scaling)

    View Slide

  13. HBase Cell
    rack rack rack rack

    View Slide

  14. Types of servers
    controllers
    nodes

    View Slide

  15. High Availability

    View Slide

  16. Server Pools
    controller pool (jobs)
    node pool (jobs)
    regionserver
    datanode
    master
    zk

    View Slide

  17. Stateful Elastic “cloud”

    View Slide

  18. Behind the Container

    View Slide

  19. The “Noisy Neighbor”

    View Slide

  20. • High iops on /dev/sda
    • Synchronous logging
    The “Noisy Neighbor”
    From the Container
    HBase
    Container
    /dev/sda

    View Slide

  21. • Configuration Management
    putting load on /dev/sda
    • Memory pressure forcing
    paging on /dev/sda
    • Large configuration
    subscriptions
    • Bloated packages
    The “Noisy Neighbor”
    From the Host System
    HBase
    Container
    Host
    System
    /dev/sda

    View Slide

  22. Performance & The Bug

    View Slide

  23. • Increased latency and timeouts
    • Cyclical spikes in io-wait across
    all of the Region Servers /
    Datanodes
    Performance & The Bug
    From the Container

    View Slide

  24. • Cyclical spikes across all disks
    hitting 100% utilization
    • No sign of any applications
    accessing the disks
    (iops == 0)
    Performance & The Bug
    From the Host System

    View Slide

  25. View Slide

  26. • Log entries of a “learning
    cycle” every few minutes
    • Correlation to the drives
    locking up
    • Configuration mode to enable
    this “learning cycle”
    Performance & The Bug
    From the Hardware
    RAID
    Controller
    learning
    cycle
    HBase
    Container
    Host
    System

    View Slide

  27. View Slide

  28. The Scheduler Apocalypse

    View Slide

  29. The Scheduler Apocalypse
    scheduler
    host3
    host4
    host1
    host2

    View Slide

  30. Conclusions

    View Slide

  31. Conclusions
    • Containers provide a rich suite of tools and technologies
    to create standard, consistent and repeatable services
    • However, there are critical decisions to be made:
    • Buy vs. Lease
    • What parts of the Container Platform to use
    • You still need to be aware of what is happening behind the
    container
    • The leverage of the container goes both ways

    View Slide

  32. Q&A

    View Slide

  33. Thanks!

    View Slide