Drupal On Demand - PICC10

Slide 1

Slide 1 text

Drupal On-Demand Drupal On-Demand Nick Silkey Yale University PICC'10 - May 2010

Slide 2

Slide 2 text

Drupal On-Demand Who is Nick Silkey, anyway? UNIX infrastructure generalist Higher-education background utexas.edu yale.edu Obsessions high-scalability fault-tolerance and high-availability automation and autonomics

Slide 3

Slide 3 text

Drupal On-Demand Web at Yale before deciding Drupal yale.edu => static Apache LMS => centralized Sakai Pockets of dynamic web stacks Wordpress in academia Confluence here and there Shared PHP from Med School Drupal users outsourcing

Slide 4

Slide 4 text

Drupal On-Demand "Drupal it is!" -Yale Leaders, 2009 Needs to scale ('thousands of sites!') Some people want 'blogs' Others dont; they want 'calendar' Or 'image gallery' ... or 'brochure site' Now infrastructure is tasked with delivering What my team decided was needed Needs to re-use existing MySQL Enterprise Needs campus SSO (CAS 2.x) End users get user1-like 'root'

Slide 5

Slide 5 text

Drupal On-Demand What my team figured was needed ('Good artists borrow. Great artists steal') High-scalability Re-use scalability work from others Optimize caching at framework Optimize caching at reverse-proxy Optimize caching at db-layer

Slide 6

Slide 6 text

Drupal On-Demand What my team figured was needed Version control system This will be highly customized Drupal Multiple ITS teams committing code SFTPing to and fro is error prone Likely need vendor branch goodness

Slide 7

Slide 7 text

Drupal On-Demand What my team figured was needed Deployment tools Reach out and touch New customers quickly Customized 'Yale code' updates Module code bumps Drupal core patches for vulns

Slide 8

Slide 8 text

Drupal On-Demand What my team knew would fail Dedicated pizzaboxes Doesnt scale Datacenters out of space/power Customers are cash-strapped tar xvf and handing keys to users Not enough 'service' People can have that now in places And theyre not happy

Slide 9

Slide 9 text

Drupal On-Demand What my team knew would fail Extensive human involvement s/Extensive/Expensive/ Error prone, even with rote docs Shared databases Table prefixing Hack attack FUD => info disclosure

Slide 10

Slide 10 text

Drupal On-Demand What my team knew would fail Drupal multi-site No account for disparate codebases Yale is paranoid about change control 'Set and forget' lest you need to touch What if Yale Drupal forks? More Yale infrastructure to support? Hack attack FUD => info disclosure We hear its brittle

Slide 11

Slide 11 text

Drupal On-Demand What my team decided upon Shared codebase 'Code up; content down' Customized bits flow upstream Templates, modules, etc dev => tst => prd Merges + deploys Opt not to push/pull This allows us to commit emergency breakfixes to prd and bring back down when the storm clears

Slide 12

Slide 12 text

Drupal On-Demand What my team decided upon 'Code up; content down' Content bits flow downstream Pages/nodes, module configs, etc prd => {dev,tst,sbx} Refresh + optional deploy Great for testing away from production!

Slide 13

Slide 13 text

Drupal On-Demand At the frontend: how Yale did it Apache => Name-based vhosts SSL a no-go Combine with wildcard DNS *.sites.domain.com ; *.web.domain.com Greatly diminishes turnaround PHP5 => open_basedir restriction Stops hack attack directory traversal

Slide 14

Slide 14 text

Drupal On-Demand At the frontend: how Yale did it Pressflow is an awesome Drupal 'fork' Performance DB replication support Memcache plug-and-play Scalability/Availability squid/varnish support MySQL-optimized PHP5-optimized Testing SimpleTest support

Slide 15

Slide 15 text

Drupal On-Demand At the frontend: how Yale did it Alternative PHP cache (APC) Optimizes PHP intermediate code Caches data/compiled bytecode Ramcache module at f5 load-balancers Drupal App servers already 'behind' Dedicated ASICs are --omg-optimized

Slide 16

Slide 16 text

Drupal On-Demand At the frontend: how Yale did it Memcached + memcache PHP extension + memcache Drupal module Opt-in service for high-throughput users Standardized on 9 memcached bins per site (default, block, content, filter, form, menu, page, update, views) Fast (fnv > crc32), yet complex Cfengine puts opt-in code where needed Stands up memcached bins Points Pressflow to the bins Site's bins flushed on deploy Plan/manage port splay for these customers

Slide 17

Slide 17 text

Drupal On-Demand At the backend: how Yale did it Drupal node attachments (binary files) NetApp filer NFS exports to app servers Broken up per environment (dev, prd, etc) open_basedir locks sites to their place Keeps $site1 files from leaking via PHP powering $site2, etc Scales well across horizontal nodes Drupal wont write-collide Reads available at all, instantly Big win; some sites were glorified smb shares!

Slide 18

Slide 18 text

Drupal On-Demand Behind the scenes: how Yale did it Lots of tools let us build en-masse Apache Ant Yale loves J2EE already Potential for Maven later Capistrano Maturing, but who wants all the ruby? :) Aegir Immature, but promising

Slide 19

Slide 19 text

Drupal On-Demand Behind the scenes: how Yale did it Apache Ant won out Dev deploys by webteam on-demand! Prd deploys by release manager on change schedule! Hash algorithm => MySQL bits Allows divination at Ant runtime No one knows, no one cares 'Seed' Ant to allow later authn changes Creds can always leak!

Slide 20

Slide 20 text

Drupal On-Demand Behind the scenes: how Yale did it Bazaar DVCS Handles vendor branches well No svn_load_dirs! Easy to punt to webteam Yale Drupal easily most extensive vendor drop install on-campus Easily lets webdev team GTD Mac => MAMP ; Win => XAMP Easy to use, well supported at both DVCS also a boon for infrastructure Were also users (utilities in bzr)

Slide 21

Slide 21 text

Drupal On-Demand What powers Drupal Architecture f5 LAMP MySQL NetApp Ant Bazaar

Slide 22

Slide 22 text

Drupal On-Demand Behind the scenes: how Yale did it Drush (DRUpal SHell) CLI tool availabile to PHP >= 5.2 users Our Ant work is married to Drush Cache flushes Getting/setting variables stored in MySQL Stages authz for new customers Maintains authz with lifecycle management Stands up Drupal 'Features'

Slide 23

Slide 23 text

Drupal On-Demand Behind the scenes: how Yale did it Drupal Features drupal.org community module Allows bootstrap of vetted mods+cfgs Combined with strongarm, enforces cfg Great for standup; terrible for teardown

Slide 24

Slide 24 text

Drupal On-Demand Drush+Features rules at enabling mods+cfgs!

Slide 25

Slide 25 text

Drupal On-Demand So thats how we got from this ...

Slide 26

Slide 26 text

Drupal On-Demand ... to this. In under a minute.

Slide 27

Slide 27 text

Drupal On-Demand Ant workflow for new namespaces Export Drupal branch from bzr Bootstrap Apache DocRoot on filesystem Push+uncompress branch bits Clean up 'restricted' theme data Bootstrap settings.php from templates + Ant runtime Use MySQL stored procedures to create database + set ACLs Push Pressflow schema into db Bootstrap Apache VirtualHost config from templates + Ant runtime Configure files/ to point to NetApp

Slide 28

Slide 28 text

Drupal On-Demand Ant workflow for new namespaces Do the 'switch' (point current => this revno) Export utilities branch from bzr Drush: Stand up Yale features Drush: Set $site_name Drush: Blast $drupal_private_key Drush: Flush Drupal cache Drush: Push autorole data Bop Apache

Slide 29

Slide 29 text

Drupal On-Demand Ant workflow for existing namespaces Export Drupal branch from bzr Validate bootstrapped Apache DocRoot on filesystem Push+uncompress branch bits Clean up 'restricted' theme data Validate bootstrapped settings.php from templates + Ant runtime Validate bootstrapped Apache VirtualHost config from templates + Ant runtime Validate configured files/ to point to NetApp Take paranoid database snapshot Drush: $site_offline => 1 ('offline')

Slide 30

Slide 30 text

Drupal On-Demand Ant workflow for existing namespaces Do the 'switch' (point current => this revno; point prior => prior revno) Export utilities branch from bzr Drush: Update schema if needed (module bumps, etc.) Drush: Flush Drupal cache Drush: Enforce pushed autorole data Bop Apache Drush: $site_offline => 0 ('online')

Slide 31

Slide 31 text

Drupal On-Demand "But what if youre bricked/WSOD post-deploy?" Do the 'switch' back (point current => prior revno) Restore paranoid database snapshot Viola! Now prd is restored; leave no trace ... "The fire is out. Now what?" Refresh prd db => dev db Deploy prd revno => dev You now have 'prd' ... but in 'dev'! Debug, test, etc.

Slide 32

Slide 32 text

Drupal On-Demand What my team learned Ant can do way more Lifecycle mgmt of deploys on app servers Vuln? Reach out and touch everyone Call MySQL stored procedures Create DB Stand up MySQL authz Recycle authz should we leak Automates deployment

Slide 33

Slide 33 text

Drupal On-Demand What my team learned Ant can do way more Juggle performance variables? Convert MyISAM to InnoDB? Reinstantiate admin role to stripped users? Strip admin role from gone/rogue users?

Slide 34

Slide 34 text

Drupal On-Demand Drupal Ant dependencies, August 2009

Slide 35

Slide 35 text

Drupal On-Demand Drupal Ant dependencies, October 2009

Slide 36

Slide 36 text

Drupal On-Demand Drupal Ant dependencies, December 2009

Slide 37

Slide 37 text

Drupal On-Demand Drupal Ant dependencies, February 2010

Slide 38

Slide 38 text

Drupal On-Demand Looks scary ... but its not :) We wrote it, maintained it, documented it Internals easily taught; new staffer jumped in Because of this, we keep extending it Some of our acrobatics included: Updating every namespace in minutes for vulns Converting every namespace erroneously using MyISAM to InnoDB (bug spotted post-launch) Switching running configs regarding performance vars within Pressflow (caches, etc) Recycling MySQL authn for every namespace when we had leak problems*

Slide 39

Slide 39 text

Drupal On-Demand What my team learned Front Ant Currently called interactively Continuous integration tools enable users Hudson is a clear winner GUI wrapper for cryptic Ant Stage Ant properties in Hudson jobs

Slide 40

Slide 40 text

Drupal On-Demand What my team learned More than ITS needs to commit to VCS Power users on-campus want read-write Maybe svn > bzr regarding authz? bzr shines with svn+ssh gids/facls == mess

Slide 41

Slide 41 text

Drupal On-Demand What my team learned PHP >=5.2 is a necessity for today's LAMP RHEL5.x provides 5.1 :( drupal.org code/mods want 5.2 drush wants 5.2 core is safe with 5.1 ... currently Yale built 5.2 for drush Drupal runs 5.1

Slide 42

Slide 42 text

Drupal On-Demand What my team learned Open-source reverse-proxies == 'easier' Deployment tools can easily reach out to: Varnish Pound Perbal iControl API via SDK for f5, but steep curve

Slide 43

Slide 43 text

Drupal On-Demand Where Yale Drupal is going Subversion is back Refresh lone svn branch nightly from bzr Give power-users branches Weve solved authz in svn Use externals to keep their branches to-date Use externals + authz for r-w where needed Templates => yes Modules => no Core => no Network-aware VCS == continuous integration

Slide 44

Slide 44 text

Drupal On-Demand Where Yale Drupal is going Hudson is coming Let webteam play build/release manager For institutional namespaces/sites Push power-user code to their sites on-commit Power-users hack/slash in their local MAMP Once baked, they commit back to svn Hudson watches their svn branches Hudson auto-deploys to power-user sites Keeps 'sandbox' in Yale infrastructure

Slide 45

Slide 45 text

Drupal On-Demand Where Yale Drupal is going Hudson is coming Field power-user merge requests Let webteam play branch manager Get power-user svn code back to bzr Easy with bzr-svn plugin Webteam plays release manager to push up

Slide 46

Slide 46 text

Drupal On-Demand Where Yale Drupal is going Remember 'Code up, content down'? Provide DB slices down to power-users Enables power-user MAMP/XAMP Enables webteam MAMP/XAMP Still evaluating risk of portablity

Slide 47

Slide 47 text

Drupal On-Demand Kudos to Four Kitchens Austin-based Drupal consultant shop One of their own is the author of Pressflow Helped in design and build of early Yale Drupal I dont speak for Yale, but I highly-recommend them Yale/4K code given back to drupal.org

Slide 48

Slide 48 text

Drupal On-Demand Questions? Ask away. Drupalers at Yale want to: Help others facing similar challenges Be public with our work Perhaps give it away We already did with some Many Edus, One Wheel