Using FreeBSD to Build a Resilient Container (Jails) Infrastructure

For many web based businesses, service and data availability are paramount. Majority of the businesses don’t require “internet” scale infrastructure and the complexity that comes with that, they however do need one which enables their businesses, which helps them to do more with less and grows with them, while making it easier for the admins in day to day maintenance.

Transcript

1. Using FreeBSD to Build a Resilient Container (Jails) Infrastructure Md. Zubair Sharief Twitter: @mzs114
2. About Me • Blog: kgibran.wordpress.com • Current: Designing, managing infrastructure for a client of Imaginea/Pramati • When possible: Developing a web app for BSD Jails, read about security, history, employ strategies during Wesnoth • Twitter: @mzs114 (used occasionally)
3. My Challenges • Web services used across Indian edu institutions, powered by OpenVZ, on a mixture of bare metal and AWS instances. • Software affected by vulnerabilities – MediaWiki, WordPress results in spam, downtime. • Backups using rsnapshot, rsync are slow to recover. • Recovery/Rollback to a previous working copy takes time – data loss, frustrates people, loss of business.
4. FreeBSD, Jails, ZFS - Using the Right Tool • One among the oldest Unices • Support for binary packages with pkgng • Wide choice of packet filters, my choice - pf • Jails – first among containers, ~15 years old, mature • ZFS support – enterprise file system for everyone
5. ZFS Goodness • ZFS Snapshots reduce downtime – Apache foundation: https://blogs.apache.org/infra/entry/apache_org_downtime_re port • Multinode copies of data (zfs send), multi disk redundancy using mirrors, all with two simple command line utilities - zpool, zfs • Rolling back a container (jail) or a zfs dataset(could be data directory for applications) using zfs rollback • DB backup? Lock DB, flush data to disk and take snapshot! • Do we need backup software?
6. Jailing the daemons • Lean on storage (~4MB each without applications), using nullfs • Patching a single jail patches all other jails when mounted using nullfs • Granular control using jail parameters and kernel variables for jails (sysctl -a | grep jail) • Harden jails and host using securelevels, makes kernel & firewall settings immutable, compromised service/software cannot wreak havoc • Wrappers exist to make jail management simple – ezjail, CBSD • CBSD – supports spanning multiple jails nodes, replication and failover simplified
7. Current Use at Pramati – HA Squid Previous Setup • Single host running on GNU/Linux Issues • Internet access for ~700 users • Wait for on call IT to fix any issues • Initial plan for HA required an LB and two Squid nodes – raised the cost of implementation
8. Current Use at Pramati – HA Squid • HA forward proxy on different nodes • Used CARP on aliased interfaces with services inside Jails • The failover is almost instant with few seconds of interruption • Updating and moving services is simple and insured - create a fallback clone of the jail, rollback if the update fails. • Remote webservices (gmail, etc) not affected – same public IP address
9. In the Future • OpenVPN server • Storage server • Asset, Inventory management service
10. Thank You.