Slide 1

Slide 1 text

SNOWFLAKE HOSTING GENERATION 201501

Slide 2

Slide 2 text

HISTORY a short retrospect about the past of our infrastructure

Slide 3

Slide 3 text

EARLY YEARS 2008: started with 3 servers on Debian Etch 4 "DevOp Style" with SSH loops and patch files 2009: updated everything to Debian Lenny 5 2010: introduced Puppet. No patch files anymore :-) 2011: updated everything to Debian Squeeze 6 broken sites due to changed PHP/MySQL versions

Slide 4

Slide 4 text

GENERATIONS 2013: introduced server generations based on GIT, every branch gets published as Puppet env no in place upgrades anymore clean transitions to new generation, often within a (TYPO3) upgrade due to different PHP requirements

Slide 5

Slide 5 text

GENERATIONS Generation Webserver PHP DB Management 201201 Apache 2.2.16 5.3.3 MySQL 5.1.73 SysCP 201301 Apache 2.2.22 5.4.35 MySQL 5.5.40 SysCP

Slide 6

Slide 6 text

INFRASTRUCTURE Racks hired in two different datacenters dual power/cooling, USV/diesel generators, you name it DWDM between those locations and ZH office redundant VPN to other offices RIPE LIR, own network (AS198249) Transit providers: Init 7 (2015/03), Nine, NTS SwissIX peerings: HE.net, GGA Maur, WWZ, Ticinocom, Leunet, BIT, Swisscom (2015/02), others

Slide 7

Slide 7 text

PLATFORM by now, we used more or less the same servers, virtualisation and network since the beginning current setup has some limitations like local storage with 201501, we changed the whole surrounding infrastructure: new servers, new network

Slide 8

Slide 8 text

SERVERS: NUTANIX The Nutanix Virtual Computing Platform is a web-scale converged infrastructure solution that consolidates the compute (server) tier and the storage tier into a single, integrated appliance.

Slide 9

Slide 9 text

SERVERS: NUTANIX Nutanix uses the same web-scale principles and technologies that power the IT environment at innovative web companies and cloud providers such as Google, Facebook, and Amazon. Nutanix makes web-scale accessible to mainstream enterprises and government agencies without requiring an overhaul of their IT environments.

Slide 10

Slide 10 text

SERVERS: NUTANIX care free platform for virtual servers Package of software, management, hardware Software defined Storage multi tier caching

Slide 11

Slide 11 text

SERVERS: NUTANIX

Slide 12

Slide 12 text

SERVERS: NUTANIX

Slide 13

Slide 13 text

SERVERS: NUTANIX

Slide 14

Slide 14 text

SERVERS: NUTANIX

Slide 15

Slide 15 text

NETWORKING: ARISTA two Arista 7150 Switches ultra low latency (350ns) Throughput 480Gbps redundant, active/active Layer 2 & 3 Setup (MLAG/VARP) each server is connected to each switch by 2x10G

Slide 16

Slide 16 text

NETWORKING: ARISTA

Slide 17

Slide 17 text

NETWORKING: ARISTA

Slide 18

Slide 18 text

PLATFORM with all platform aspects covered, we can solely focus on our virtual servers

Slide 19

Slide 19 text

GOALS top performance small footprint, e.g. no mail daemon automate/puppetize everything, no local modifications configuration trough API 1. Click (/HTTP Call) setup for everything

Slide 20

Slide 20 text

AUTOMATION Monitoring (new: all Live vHosts included) appropriate firewall rules DNS A/AAAA records for servers Backup (TODO) server creation

Slide 21

Slide 21 text

PUPPET separate code (manifests) and configuration (Hiera) use Puppet Forge modules (nginx, mysql, php, ...) wrapper modules which include and configure upstream modules no manual interaction at all (DNS, Backup, Monitoring, Firewall) end user can trigger run trough sudo command

Slide 22

Slide 22 text

CONTRACT dedicated VM per customer no shared hosting anymore different vHosts possible (Stage, Subsites, ...) smaller units: 1 CPU, 1GB RAM, 20GB diskspace Pricing: server 95.- / CPU core 50.-/ GB RAM 10.-

Slide 23

Slide 23 text

WHAT WE DID development started in April 2014 59 Pull requests merged 523 Commits 31'794 lines Puppet manifests 14'998 lines templates 10'069 lines documentation

Slide 24

Slide 24 text

FEATURE COMPARISON Generation Webserver PHP DB Management 201201 Apache 2.2.16 5.3.3 MySQL 5.1.73 SysCP 201301 Apache 2.2.22 5.4.35 MySQL 5.5.40 SysCP 201501 nginx 1.6.2 5.6.2 MariaDB 10.0.15 Puppet/Hiera

Slide 25

Slide 25 text

OS Debian 8 Jessie based (RC1, release expected 2015/02) removed all packages which where not required access trough SSH/SCP. FTP possible, but not by default all daemons are locally monitored and restarted if required network configuration automated zsh instead of bash (more features like GIT integration) motd shows host, description, generation and additional modules

Slide 26

Slide 26 text

WEBSERVER nginx instead of Apache better performance, lower footprint but: No more .htaccess files you can still alter the configuration but only on vHost level naxsi instead of modsecurity: different but not too different SPDY support, HTTP/2 after RFC release end user can reload daemon trough sudo command

Slide 27

Slide 27 text

PHP PHP 5.6.2 PHP-FPM instead of FCGI opcache instead of APC (ZendOptimizer+ moved into Core)

Slide 28

Slide 28 text

HHVM HHVM with PHP fallback possible testing for PHP, HHVM & HHVM+PHP required complicated setup for vHosts performance gain for TYPO3 ~20% PHP itself gets faster with every version we decided to drop HHVM by default still interesting for particular sites with heavy requirements

Slide 29

Slide 29 text

DATABASE switched from MySQL to MariaDB drop in replacement. Even same paths in filesystem API/ABI compatibility with MySQL better performance true Open Source (MySQL AB > Sun > Oracle) no local phpMyAdmin anymore

Slide 30

Slide 30 text

DUALSTACK IPV4/IPV6 all Servers are reachable by IPv4 & IPv6 by default vHosts listen on both protocols Monitoring for both protocols if required, e.g. HTTP DevOp: just remember to add both A + AAAA records in DNS $ f a c t e r i p a d d r e s s i p a d d r e s s 6 i p a d d r e s s = > 1 8 5 . 1 7 . 6 8 . 1 4 1 i p a d d r e s s 6 = > 2 a 0 4 : 5 0 3 : 0 : 1 0 0 3 : : 1 4 1

Slide 31

Slide 31 text

VHOST MANAGEMENT by Puppet: Data from Hiera type based: TYPO3, magento, wordpress, php, html environment based: DEV, STAGE, LIVE by now: YAML files in GIT repository (access: DevOps) w e b s i t e : : s i t e s : " s t v h o c h d o r f " : " p a s s w o r d " : " 1 2 3 4 " " s e r v e r _ n a m e " : " s t v h o c h d o r f . c h " " e n v " : " L I V E " " h t p a s s w d " : " 4 3 2 1 " " t y p e " : " T Y P O 3 "

Slide 32

Slide 32 text

PERFORMANCE we lost quite a bit trough new features in TYPO3 6 Speed was one of the purposes for this generation Gains due to faster platform and software rough performance tests with preset.snowflake.ch Generation Frontend Backend 201301 140ms 1100ms 201501 50ms 300ms improvement between 2-4x

Slide 33

Slide 33 text

DEVELOPMENT Vagrant Team Server (external network, access by SSH only)

Slide 34

Slide 34 text

VAGRANT 201501 was developed on Vagrant only only required packages (e.g. no monitoring) Linux Container based, no overhead, but Linux/Mac OS only root access, Puppet pull requests

Slide 35

Slide 35 text

NEXT STEPS create Team Servers (Mail Catcher Pending) finish documentation DevOp training order your employee hosting migrate todoyu hostings go Live with customer sites

Slide 36

Slide 36 text

OUTLOOK create and modify VMs trough Puppet integrate billing replace YAML with REST API

Slide 37

Slide 37 text

OUTLOOK

Slide 38

Slide 38 text

(DISTANT) OUTLOOK API interface on Angular/Ember/Whatever connect more services to the API (DNS, Logs, Metrics, ...) automate networking VLAN per customer. IPv6 subnet per customer IPv6 only datacenter. IPv4 trough Proxy/NAT64

Slide 39

Slide 39 text

THANK YOU FEEDBACK WELCOME