HBaseCon 2016 West - Containerizing Apache HBase Clusters

At Facebook, all production HBase clusters run in a containerized environment, with every daemon running inside its own LXC container. Containerization allows us to ensure isolation between services running on the same host and simplify operations, but sometimes abstractions leak and problems can't be addressed inside the container. In this talk, we will discuss how Facebook runs HBase as a stateful service inside containers and we will discuss some of the issues we've found when doing so.

https://hbase.apache.org/hbasecon-archives.html#HBaseCon_West_2016

Transcript

1. Containerizing Apache HBase
   Clusters
   David Pope & Javier Maestro
   Production Engineers - HBase
   

   View Slide

2. Container Overview
   

   View Slide

3. View Slide

4. 1964
   IBM 360
   1979
   UNIX
   chroot
   1982
   BSD
   chroot
   1999
   FreeBSD
   jail
   2013
   Docker
   2007-8
   cgroups
   LXC
   

   View Slide

5. Container Platforms
   

   View Slide

6. View Slide

7. HBase Containers @ Facebook
   

   View Slide

8. LEASE
   BUY
   

   View Slide

9. Buy vs. Lease
   • Our scale
   • Timing
   • Control / Full Ownership
   • Financial
   Infrastructure
   (os, network)
   Container Platform
   (Tupperware)
   Application
   Services
   Physical
   (data center, hardware)
   

   View Slide

10. Tupperware Overview
    scheduler
    server db
    host3
    host4
    host1
    host2
    config.tw
    

    View Slide

11. Tupperware Spec
    

    View Slide

12. Tupperware Benefits
    • Configuration Spec
    • Deployments
    • Scheduler
    • Health Monitor
    • Logging
    • Canary
    • Web UI / CLI / API
    • Elasticity (auto-scaling)
    

    View Slide

13. HBase Cell
    rack rack rack rack
    

    View Slide

14. Types of servers
    controllers
    nodes
    

    View Slide

15. High Availability
    

    View Slide

16. Server Pools
    controller pool (jobs)
    node pool (jobs)
    regionserver
    datanode
    master
    zk
    

    View Slide

17. Stateful Elastic “cloud”
    

    View Slide

18. Behind the Container
    

    View Slide

19. The “Noisy Neighbor”
    

    View Slide

20. • High iops on /dev/sda
    • Synchronous logging
    The “Noisy Neighbor”
    From the Container
    HBase
    Container
    /dev/sda
    

    View Slide

21. • Configuration Management
    putting load on /dev/sda
    • Memory pressure forcing
    paging on /dev/sda
    • Large configuration
    subscriptions
    • Bloated packages
    The “Noisy Neighbor”
    From the Host System
    HBase
    Container
    Host
    System
    /dev/sda
    

    View Slide

22. Performance & The Bug
    

    View Slide

23. • Increased latency and timeouts
    • Cyclical spikes in io-wait across
    all of the Region Servers /
    Datanodes
    Performance & The Bug
    From the Container
    

    View Slide

24. • Cyclical spikes across all disks
    hitting 100% utilization
    • No sign of any applications
    accessing the disks
    (iops == 0)
    Performance & The Bug
    From the Host System
    

    View Slide

25. View Slide

26. • Log entries of a “learning
    cycle” every few minutes
    • Correlation to the drives
    locking up
    • Configuration mode to enable
    this “learning cycle”
    Performance & The Bug
    From the Hardware
    RAID
    Controller
    learning
    cycle
    HBase
    Container
    Host
    System
    

    View Slide

27. View Slide

28. The Scheduler Apocalypse
    

    View Slide

29. The Scheduler Apocalypse
    scheduler
    host3
    host4
    host1
    host2
    

    View Slide

30. Conclusions
    

    View Slide

31. Conclusions
    • Containers provide a rich suite of tools and technologies
    to create standard, consistent and repeatable services
    • However, there are critical decisions to be made:
    • Buy vs. Lease
    • What parts of the Container Platform to use
    • You still need to be aware of what is happening behind the
    container
    • The leverage of the container goes both ways
    

    View Slide

32. Q&A
    

    View Slide

33. Thanks!
    

    View Slide