SNOWFLAKE HOSTING GENERATION 201501 

HISTORY a short retrospect about the past of our infrastructure 

EARLY YEARS 2008: started with 3 servers on Debian Etch 4 "DevOp Style" with SSH loops and patch files 2009: updated everything to Debian Lenny 5 2010: introduced Puppet. No patch files anymore :-) 2011: updated everything to Debian Squeeze 6 broken sites due to changed PHP/MySQL versions 

GENERATIONS 2013: introduced server generations based on GIT, every branch gets published as Puppet env no in place upgrades anymore clean transitions to new generation, often within a (TYPO3) upgrade due to different PHP requirements 

GENERATIONS Generation Webserver PHP DB Management 201201 Apache 2.2.16 5.3.3 MySQL 5.1.73 SysCP 201301 Apache 2.2.22 5.4.35 MySQL 5.5.40 SysCP 

INFRASTRUCTURE Racks hired in two different datacenters dual power/cooling, USV/diesel generators, you name it DWDM between those locations and ZH office redundant VPN to other offices RIPE LIR, own network (AS198249) Transit providers: Init 7 (2015/03), Nine, NTS SwissIX peerings: HE.net, GGA Maur, WWZ, Ticinocom, Leunet, BIT, Swisscom (2015/02), others 

PLATFORM by now, we used more or less the same servers, virtualisation and network since the beginning current setup has some limitations like local storage with 201501, we changed the whole surrounding infrastructure: new servers, new network 

SERVERS: NUTANIX The Nutanix Virtual Computing Platform is a web-scale converged infrastructure solution that consolidates the compute (server) tier and the storage tier into a single, integrated appliance. 

SERVERS: NUTANIX Nutanix uses the same web-scale principles and technologies that power the IT environment at innovative web companies and cloud providers such as Google, Facebook, and Amazon. Nutanix makes web-scale accessible to mainstream enterprises and government agencies without requiring an overhaul of their IT environments. 

SERVERS: NUTANIX care free platform for virtual servers Package of software, management, hardware Software defined Storage multi tier caching 

SERVERS: NUTANIX 

SERVERS: NUTANIX 

SERVERS: NUTANIX 

SERVERS: NUTANIX 

NETWORKING: ARISTA two Arista 7150 Switches ultra low latency (350ns) Throughput 480Gbps redundant, active/active Layer 2 & 3 Setup (MLAG/VARP) each server is connected to each switch by 2x10G 

NETWORKING: ARISTA 

NETWORKING: ARISTA 

PLATFORM with all platform aspects covered, we can solely focus on our virtual servers 

GOALS top performance small footprint, e.g. no mail daemon automate/puppetize everything, no local modifications configuration trough API 1. Click (/HTTP Call) setup for everything 

AUTOMATION Monitoring (new: all Live vHosts included) appropriate firewall rules DNS A/AAAA records for servers Backup (TODO) server creation 

PUPPET separate code (manifests) and configuration (Hiera) use Puppet Forge modules (nginx, mysql, php, ...) wrapper modules which include and configure upstream modules no manual interaction at all (DNS, Backup, Monitoring, Firewall) end user can trigger run trough sudo command 

CONTRACT dedicated VM per customer no shared hosting anymore different vHosts possible (Stage, Subsites, ...) smaller units: 1 CPU, 1GB RAM, 20GB diskspace Pricing: server 95.- / CPU core 50.-/ GB RAM 10.- 

WHAT WE DID development started in April 2014 59 Pull requests merged 523 Commits 31'794 lines Puppet manifests 14'998 lines templates 10'069 lines documentation 

FEATURE COMPARISON Generation Webserver PHP DB Management 201201 Apache 2.2.16 5.3.3 MySQL 5.1.73 SysCP 201301 Apache 2.2.22 5.4.35 MySQL 5.5.40 SysCP 201501 nginx 1.6.2 5.6.2 MariaDB 10.0.15 Puppet/Hiera 

OS Debian 8 Jessie based (RC1, release expected 2015/02) removed all packages which where not required access trough SSH/SCP. FTP possible, but not by default all daemons are locally monitored and restarted if required network configuration automated zsh instead of bash (more features like GIT integration) motd shows host, description, generation and additional modules 

WEBSERVER nginx instead of Apache better performance, lower footprint but: No more .htaccess files you can still alter the configuration but only on vHost level naxsi instead of modsecurity: different but not too different SPDY support, HTTP/2 after RFC release end user can reload daemon trough sudo command 

PHP PHP 5.6.2 PHP-FPM instead of FCGI opcache instead of APC (ZendOptimizer+ moved into Core) 

HHVM HHVM with PHP fallback possible testing for PHP, HHVM & HHVM+PHP required complicated setup for vHosts performance gain for TYPO3 ~20% PHP itself gets faster with every version we decided to drop HHVM by default still interesting for particular sites with heavy requirements 

DATABASE switched from MySQL to MariaDB drop in replacement. Even same paths in filesystem API/ABI compatibility with MySQL better performance true Open Source (MySQL AB > Sun > Oracle) no local phpMyAdmin anymore 

DUALSTACK IPV4/IPV6 all Servers are reachable by IPv4 & IPv6 by default vHosts listen on both protocols Monitoring for both protocols if required, e.g. HTTP DevOp: just remember to add both A + AAAA records in DNS $ f a c t e r i p a d d r e s s i p a d d r e s s 6 i p a d d r e s s = > 1 8 5 . 1 7 . 6 8 . 1 4 1 i p a d d r e s s 6 = > 2 a 0 4 : 5 0 3 : 0 : 1 0 0 3 : : 1 4 1 

VHOST MANAGEMENT by Puppet: Data from Hiera type based: TYPO3, magento, wordpress, php, html environment based: DEV, STAGE, LIVE by now: YAML files in GIT repository (access: DevOps) w e b s i t e : : s i t e s : " s t v h o c h d o r f " : " p a s s w o r d " : " 1 2 3 4 " " s e r v e r _ n a m e " : " s t v h o c h d o r f . c h " " e n v " : " L I V E " " h t p a s s w d " : " 4 3 2 1 " " t y p e " : " T Y P O 3 " 

PERFORMANCE we lost quite a bit trough new features in TYPO3 6 Speed was one of the purposes for this generation Gains due to faster platform and software rough performance tests with preset.snowflake.ch Generation Frontend Backend 201301 140ms 1100ms 201501 50ms 300ms improvement between 2-4x 

DEVELOPMENT Vagrant Team Server (external network, access by SSH only) 

VAGRANT 201501 was developed on Vagrant only only required packages (e.g. no monitoring) Linux Container based, no overhead, but Linux/Mac OS only root access, Puppet pull requests 

NEXT STEPS create Team Servers (Mail Catcher Pending) finish documentation DevOp training order your employee hosting migrate todoyu hostings go Live with customer sites 

OUTLOOK create and modify VMs trough Puppet integrate billing replace YAML with REST API 

OUTLOOK 

(DISTANT) OUTLOOK API interface on Angular/Ember/Whatever connect more services to the API (DNS, Logs, Metrics, ...) automate networking VLAN per customer. IPv6 subnet per customer IPv6 only datacenter. IPv4 trough Proxy/NAT64 

THANK YOU FEEDBACK WELCOME