Using FreeBSD to Build a Resilient Container (Jails) Infrastructure Md. Zubair Sharief Twitter: @mzs114

About Me ● Blog: kgibran.wordpress.com ● Current: Designing, managing infrastructure for a client of Imaginea/Pramati ● When possible: Developing a web app for BSD Jails, read about security, history, employ strategies during Wesnoth ● Twitter: @mzs114 (used occasionally)

My Challenges ● Web services used across Indian edu institutions, powered by OpenVZ, on a mixture of bare metal and AWS instances. ● Software affected by vulnerabilities – MediaWiki, WordPress results in spam, downtime. ● Backups using rsnapshot, rsync are slow to recover. ● Recovery/Rollback to a previous working copy takes time – data loss, frustrates people, loss of business.

FreeBSD, Jails, ZFS - Using the Right Tool ● One among the oldest Unices ● Support for binary packages with pkgng ● Wide choice of packet filters, my choice - pf ● Jails – first among containers, ~15 years old, mature ● ZFS support – enterprise file system for everyone

ZFS Goodness ● ZFS Snapshots reduce downtime – Apache foundation: https://blogs.apache.org/infra/entry/apache_org_downtime_re port ● Multinode copies of data (zfs send), multi disk redundancy using mirrors, all with two simple command line utilities - zpool, zfs ● Rolling back a container (jail) or a zfs dataset(could be data directory for applications) using zfs rollback ● DB backup? Lock DB, flush data to disk and take snapshot! ● Do we need backup software?

Jailing the daemons ● Lean on storage (~4MB each without applications), using nullfs ● Patching a single jail patches all other jails when mounted using nullfs ● Granular control using jail parameters and kernel variables for jails (sysctl -a | grep jail) ● Harden jails and host using securelevels, makes kernel & firewall settings immutable, compromised service/software cannot wreak havoc ● Wrappers exist to make jail management simple – ezjail, CBSD ● CBSD – supports spanning multiple jails nodes, replication and failover simplified

Current Use at Pramati – HA Squid Previous Setup ● Single host running on GNU/Linux Issues ● Internet access for ~700 users ● Wait for on call IT to fix any issues ● Initial plan for HA required an LB and two Squid nodes – raised the cost of implementation

Current Use at Pramati – HA Squid ● HA forward proxy on different nodes ● Used CARP on aliased interfaces with services inside Jails ● The failover is almost instant with few seconds of interruption ● Updating and moving services is simple and insured - create a fallback clone of the jail, rollback if the update fails. ● Remote webservices (gmail, etc) not affected – same public IP address

In the Future ● OpenVPN server ● Storage server ● Asset, Inventory management service

Thank You.