Upgrade to Pro — share decks privately, control downloads, hide ads and more …

snowflake Hosting Generation 2015

2745c78242fec52bd69ff276ce5ebe17?s=47 Andri Steiner
February 19, 2015

snowflake Hosting Generation 2015

Introduction of our new Server Generation


Andri Steiner

February 19, 2015



  2. HISTORY a short retrospect about the past of our infrastructure

  3. EARLY YEARS 2008: started with 3 servers on Debian Etch

    4 "DevOp Style" with SSH loops and patch files 2009: updated everything to Debian Lenny 5 2010: introduced Puppet. No patch files anymore :-) 2011: updated everything to Debian Squeeze 6 broken sites due to changed PHP/MySQL versions
  4. GENERATIONS 2013: introduced server generations based on GIT, every branch

    gets published as Puppet env no in place upgrades anymore clean transitions to new generation, often within a (TYPO3) upgrade due to different PHP requirements
  5. GENERATIONS Generation Webserver PHP DB Management 201201 Apache 2.2.16 5.3.3

    MySQL 5.1.73 SysCP 201301 Apache 2.2.22 5.4.35 MySQL 5.5.40 SysCP
  6. INFRASTRUCTURE Racks hired in two different datacenters dual power/cooling, USV/diesel

    generators, you name it DWDM between those locations and ZH office redundant VPN to other offices RIPE LIR, own network (AS198249) Transit providers: Init 7 (2015/03), Nine, NTS SwissIX peerings: HE.net, GGA Maur, WWZ, Ticinocom, Leunet, BIT, Swisscom (2015/02), others
  7. PLATFORM by now, we used more or less the same

    servers, virtualisation and network since the beginning current setup has some limitations like local storage with 201501, we changed the whole surrounding infrastructure: new servers, new network
  8. SERVERS: NUTANIX The Nutanix Virtual Computing Platform is a web-scale

    converged infrastructure solution that consolidates the compute (server) tier and the storage tier into a single, integrated appliance.
  9. SERVERS: NUTANIX Nutanix uses the same web-scale principles and technologies

    that power the IT environment at innovative web companies and cloud providers such as Google, Facebook, and Amazon. Nutanix makes web-scale accessible to mainstream enterprises and government agencies without requiring an overhaul of their IT environments.
  10. SERVERS: NUTANIX care free platform for virtual servers Package of

    software, management, hardware Software defined Storage multi tier caching




  15. NETWORKING: ARISTA two Arista 7150 Switches ultra low latency (350ns)

    Throughput 480Gbps redundant, active/active Layer 2 & 3 Setup (MLAG/VARP) each server is connected to each switch by 2x10G


  18. PLATFORM with all platform aspects covered, we can solely focus

    on our virtual servers
  19. GOALS top performance small footprint, e.g. no mail daemon automate/puppetize

    everything, no local modifications configuration trough API 1. Click (/HTTP Call) setup for everything
  20. AUTOMATION Monitoring (new: all Live vHosts included) appropriate firewall rules

    DNS A/AAAA records for servers Backup (TODO) server creation
  21. PUPPET separate code (manifests) and configuration (Hiera) use Puppet Forge

    modules (nginx, mysql, php, ...) wrapper modules which include and configure upstream modules no manual interaction at all (DNS, Backup, Monitoring, Firewall) end user can trigger run trough sudo command
  22. CONTRACT dedicated VM per customer no shared hosting anymore different

    vHosts possible (Stage, Subsites, ...) smaller units: 1 CPU, 1GB RAM, 20GB diskspace Pricing: server 95.- / CPU core 50.-/ GB RAM 10.-
  23. WHAT WE DID development started in April 2014 59 Pull

    requests merged 523 Commits 31'794 lines Puppet manifests 14'998 lines templates 10'069 lines documentation
  24. FEATURE COMPARISON Generation Webserver PHP DB Management 201201 Apache 2.2.16

    5.3.3 MySQL 5.1.73 SysCP 201301 Apache 2.2.22 5.4.35 MySQL 5.5.40 SysCP 201501 nginx 1.6.2 5.6.2 MariaDB 10.0.15 Puppet/Hiera
  25. OS Debian 8 Jessie based (RC1, release expected 2015/02) removed

    all packages which where not required access trough SSH/SCP. FTP possible, but not by default all daemons are locally monitored and restarted if required network configuration automated zsh instead of bash (more features like GIT integration) motd shows host, description, generation and additional modules
  26. WEBSERVER nginx instead of Apache better performance, lower footprint but:

    No more .htaccess files you can still alter the configuration but only on vHost level naxsi instead of modsecurity: different but not too different SPDY support, HTTP/2 after RFC release end user can reload daemon trough sudo command
  27. PHP PHP 5.6.2 PHP-FPM instead of FCGI opcache instead of

    APC (ZendOptimizer+ moved into Core)
  28. HHVM HHVM with PHP fallback possible testing for PHP, HHVM

    & HHVM+PHP required complicated setup for vHosts performance gain for TYPO3 ~20% PHP itself gets faster with every version we decided to drop HHVM by default still interesting for particular sites with heavy requirements
  29. DATABASE switched from MySQL to MariaDB drop in replacement. Even

    same paths in filesystem API/ABI compatibility with MySQL better performance true Open Source (MySQL AB > Sun > Oracle) no local phpMyAdmin anymore
  30. DUALSTACK IPV4/IPV6 all Servers are reachable by IPv4 & IPv6

    by default vHosts listen on both protocols Monitoring for both protocols if required, e.g. HTTP DevOp: just remember to add both A + AAAA records in DNS $ f a c t e r i p a d d r e s s i p a d d r e s s 6 i p a d d r e s s = > 1 8 5 . 1 7 . 6 8 . 1 4 1 i p a d d r e s s 6 = > 2 a 0 4 : 5 0 3 : 0 : 1 0 0 3 : : 1 4 1
  31. VHOST MANAGEMENT by Puppet: Data from Hiera type based: TYPO3,

    magento, wordpress, php, html environment based: DEV, STAGE, LIVE by now: YAML files in GIT repository (access: DevOps) w e b s i t e : : s i t e s : " s t v h o c h d o r f " : " p a s s w o r d " : " 1 2 3 4 " " s e r v e r _ n a m e " : " s t v h o c h d o r f . c h " " e n v " : " L I V E " " h t p a s s w d " : " 4 3 2 1 " " t y p e " : " T Y P O 3 "
  32. PERFORMANCE we lost quite a bit trough new features in

    TYPO3 6 Speed was one of the purposes for this generation Gains due to faster platform and software rough performance tests with preset.snowflake.ch Generation Frontend Backend 201301 140ms 1100ms 201501 50ms 300ms improvement between 2-4x
  33. DEVELOPMENT Vagrant Team Server (external network, access by SSH only)

  34. VAGRANT 201501 was developed on Vagrant only only required packages

    (e.g. no monitoring) Linux Container based, no overhead, but Linux/Mac OS only root access, Puppet pull requests
  35. NEXT STEPS create Team Servers (Mail Catcher Pending) finish documentation

    DevOp training order your employee hosting migrate todoyu hostings go Live with customer sites
  36. OUTLOOK create and modify VMs trough Puppet integrate billing replace

    YAML with REST API

  38. (DISTANT) OUTLOOK API interface on Angular/Ember/Whatever connect more services to

    the API (DNS, Logs, Metrics, ...) automate networking VLAN per customer. IPv6 subnet per customer IPv6 only datacenter. IPv4 trough Proxy/NAT64