Drupal On Demand - PICC10

Transcript

1. Drupal On-Demand Drupal On-Demand Nick Silkey Yale University PICC'10 -

   May 2010

2. Drupal On-Demand Who is Nick Silkey, anyway? UNIX infrastructure generalist

   Higher-education background utexas.edu yale.edu Obsessions high-scalability fault-tolerance and high-availability automation and autonomics

3. Drupal On-Demand Web at Yale before deciding Drupal yale.edu =>

   static Apache LMS => centralized Sakai Pockets of dynamic web stacks Wordpress in academia Confluence here and there Shared PHP from Med School Drupal users outsourcing

4. Drupal On-Demand "Drupal it is!" -Yale Leaders, 2009 Needs to

   scale ('thousands of sites!') Some people want 'blogs' Others dont; they want 'calendar' Or 'image gallery' ... or 'brochure site' Now infrastructure is tasked with delivering What my team decided was needed Needs to re-use existing MySQL Enterprise Needs campus SSO (CAS 2.x) End users get user1-like 'root'

5. Drupal On-Demand What my team figured was needed ('Good artists

   borrow. Great artists steal') High-scalability Re-use scalability work from others Optimize caching at framework Optimize caching at reverse-proxy Optimize caching at db-layer

6. Drupal On-Demand What my team figured was needed Version control

   system This will be highly customized Drupal Multiple ITS teams committing code SFTPing to and fro is error prone Likely need vendor branch goodness

7. Drupal On-Demand What my team figured was needed Deployment tools

   Reach out and touch New customers quickly Customized 'Yale code' updates Module code bumps Drupal core patches for vulns

8. Drupal On-Demand What my team knew would fail Dedicated pizzaboxes

   Doesnt scale Datacenters out of space/power Customers are cash-strapped tar xvf and handing keys to users Not enough 'service' People can have that now in places And theyre not happy

9. Drupal On-Demand What my team knew would fail Extensive human

   involvement s/Extensive/Expensive/ Error prone, even with rote docs Shared databases Table prefixing Hack attack FUD => info disclosure

10. Drupal On-Demand What my team knew would fail Drupal multi-site

    No account for disparate codebases Yale is paranoid about change control 'Set and forget' lest you need to touch What if Yale Drupal forks? More Yale infrastructure to support? Hack attack FUD => info disclosure We hear its brittle

11. Drupal On-Demand What my team decided upon Shared codebase 'Code

    up; content down' Customized bits flow upstream Templates, modules, etc dev => tst => prd Merges + deploys Opt not to push/pull This allows us to commit emergency breakfixes to prd and bring back down when the storm clears

12. Drupal On-Demand What my team decided upon 'Code up; content

    down' Content bits flow downstream Pages/nodes, module configs, etc prd => {dev,tst,sbx} Refresh + optional deploy Great for testing away from production!

13. Drupal On-Demand At the frontend: how Yale did it Apache

    => Name-based vhosts SSL a no-go Combine with wildcard DNS *.sites.domain.com ; *.web.domain.com Greatly diminishes turnaround PHP5 => open_basedir restriction Stops hack attack directory traversal

14. Drupal On-Demand At the frontend: how Yale did it Pressflow

    is an awesome Drupal 'fork' Performance DB replication support Memcache plug-and-play Scalability/Availability squid/varnish support MySQL-optimized PHP5-optimized Testing SimpleTest support

15. Drupal On-Demand At the frontend: how Yale did it Alternative

    PHP cache (APC) Optimizes PHP intermediate code Caches data/compiled bytecode Ramcache module at f5 load-balancers Drupal App servers already 'behind' Dedicated ASICs are --omg-optimized

16. Drupal On-Demand At the frontend: how Yale did it Memcached

    + memcache PHP extension + memcache Drupal module Opt-in service for high-throughput users Standardized on 9 memcached bins per site (default, block, content, filter, form, menu, page, update, views) Fast (fnv > crc32), yet complex Cfengine puts opt-in code where needed Stands up memcached bins Points Pressflow to the bins Site's bins flushed on deploy Plan/manage port splay for these customers

17. Drupal On-Demand At the backend: how Yale did it Drupal

    node attachments (binary files) NetApp filer NFS exports to app servers Broken up per environment (dev, prd, etc) open_basedir locks sites to their place Keeps $site1 files from leaking via PHP powering $site2, etc Scales well across horizontal nodes Drupal wont write-collide Reads available at all, instantly Big win; some sites were glorified smb shares!

18. Drupal On-Demand Behind the scenes: how Yale did it Lots

    of tools let us build en-masse Apache Ant Yale loves J2EE already Potential for Maven later Capistrano Maturing, but who wants all the ruby? :) Aegir Immature, but promising

19. Drupal On-Demand Behind the scenes: how Yale did it Apache

    Ant won out Dev deploys by webteam on-demand! Prd deploys by release manager on change schedule! Hash algorithm => MySQL bits Allows divination at Ant runtime No one knows, no one cares 'Seed' Ant to allow later authn changes Creds can always leak!

20. Drupal On-Demand Behind the scenes: how Yale did it Bazaar

    DVCS Handles vendor branches well No svn_load_dirs! Easy to punt to webteam Yale Drupal easily most extensive vendor drop install on-campus Easily lets webdev team GTD Mac => MAMP ; Win => XAMP Easy to use, well supported at both DVCS also a boon for infrastructure Were also users (utilities in bzr)

21. Drupal On-Demand What powers Drupal Architecture f5 LAMP MySQL NetApp

    Ant Bazaar

22. Drupal On-Demand Behind the scenes: how Yale did it Drush

    (DRUpal SHell) CLI tool availabile to PHP >= 5.2 users Our Ant work is married to Drush Cache flushes Getting/setting variables stored in MySQL Stages authz for new customers Maintains authz with lifecycle management Stands up Drupal 'Features'

23. Drupal On-Demand Behind the scenes: how Yale did it Drupal

    Features drupal.org community module Allows bootstrap of vetted mods+cfgs Combined with strongarm, enforces cfg Great for standup; terrible for teardown

24. Drupal On-Demand Drush+Features rules at enabling mods+cfgs!

25. Drupal On-Demand So thats how we got from this ...

26. Drupal On-Demand ... to this. In under a minute.

27. Drupal On-Demand Ant workflow for new namespaces Export Drupal branch

    from bzr Bootstrap Apache DocRoot on filesystem Push+uncompress branch bits Clean up 'restricted' theme data Bootstrap settings.php from templates + Ant runtime Use MySQL stored procedures to create database + set ACLs Push Pressflow schema into db Bootstrap Apache VirtualHost config from templates + Ant runtime Configure files/ to point to NetApp

28. Drupal On-Demand Ant workflow for new namespaces Do the 'switch'

    (point current => this revno) Export utilities branch from bzr Drush: Stand up Yale features Drush: Set $site_name Drush: Blast $drupal_private_key Drush: Flush Drupal cache Drush: Push autorole data Bop Apache

29. Drupal On-Demand Ant workflow for existing namespaces Export Drupal branch

    from bzr Validate bootstrapped Apache DocRoot on filesystem Push+uncompress branch bits Clean up 'restricted' theme data Validate bootstrapped settings.php from templates + Ant runtime Validate bootstrapped Apache VirtualHost config from templates + Ant runtime Validate configured files/ to point to NetApp Take paranoid database snapshot Drush: $site_offline => 1 ('offline')

30. Drupal On-Demand Ant workflow for existing namespaces Do the 'switch'

    (point current => this revno; point prior => prior revno) Export utilities branch from bzr Drush: Update schema if needed (module bumps, etc.) Drush: Flush Drupal cache Drush: Enforce pushed autorole data Bop Apache Drush: $site_offline => 0 ('online')

31. Drupal On-Demand "But what if youre bricked/WSOD post-deploy?" Do the

    'switch' back (point current => prior revno) Restore paranoid database snapshot Viola! Now prd is restored; leave no trace ... "The fire is out. Now what?" Refresh prd db => dev db Deploy prd revno => dev You now have 'prd' ... but in 'dev'! Debug, test, etc.

32. Drupal On-Demand What my team learned Ant can do way

    more Lifecycle mgmt of deploys on app servers Vuln? Reach out and touch everyone Call MySQL stored procedures Create DB Stand up MySQL authz Recycle authz should we leak Automates deployment

33. Drupal On-Demand What my team learned Ant can do way

    more Juggle performance variables? Convert MyISAM to InnoDB? Reinstantiate admin role to stripped users? Strip admin role from gone/rogue users?

34. Drupal On-Demand Drupal Ant dependencies, August 2009

35. Drupal On-Demand Drupal Ant dependencies, October 2009

36. Drupal On-Demand Drupal Ant dependencies, December 2009

37. Drupal On-Demand Drupal Ant dependencies, February 2010

38. Drupal On-Demand Looks scary ... but its not :) We

    wrote it, maintained it, documented it Internals easily taught; new staffer jumped in Because of this, we keep extending it Some of our acrobatics included: Updating every namespace in minutes for vulns Converting every namespace erroneously using MyISAM to InnoDB (bug spotted post-launch) Switching running configs regarding performance vars within Pressflow (caches, etc) Recycling MySQL authn for every namespace when we had leak problems*

39. Drupal On-Demand What my team learned Front Ant Currently called

    interactively Continuous integration tools enable users Hudson is a clear winner GUI wrapper for cryptic Ant Stage Ant properties in Hudson jobs

40. Drupal On-Demand What my team learned More than ITS needs

    to commit to VCS Power users on-campus want read-write Maybe svn > bzr regarding authz? bzr shines with svn+ssh gids/facls == mess

41. Drupal On-Demand What my team learned PHP >=5.2 is a

    necessity for today's LAMP RHEL5.x provides 5.1 :( drupal.org code/mods want 5.2 drush wants 5.2 core is safe with 5.1 ... currently Yale built 5.2 for drush Drupal runs 5.1

42. Drupal On-Demand What my team learned Open-source reverse-proxies == 'easier'

    Deployment tools can easily reach out to: Varnish Pound Perbal iControl API via SDK for f5, but steep curve

43. Drupal On-Demand Where Yale Drupal is going Subversion is back

    Refresh lone svn branch nightly from bzr Give power-users branches Weve solved authz in svn Use externals to keep their branches to-date Use externals + authz for r-w where needed Templates => yes Modules => no Core => no Network-aware VCS == continuous integration

44. Drupal On-Demand Where Yale Drupal is going Hudson is coming

    Let webteam play build/release manager For institutional namespaces/sites Push power-user code to their sites on-commit Power-users hack/slash in their local MAMP Once baked, they commit back to svn Hudson watches their svn branches Hudson auto-deploys to power-user sites Keeps 'sandbox' in Yale infrastructure

45. Drupal On-Demand Where Yale Drupal is going Hudson is coming

    Field power-user merge requests Let webteam play branch manager Get power-user svn code back to bzr Easy with bzr-svn plugin Webteam plays release manager to push up

46. Drupal On-Demand Where Yale Drupal is going Remember 'Code up,

    content down'? Provide DB slices down to power-users Enables power-user MAMP/XAMP Enables webteam MAMP/XAMP Still evaluating risk of portablity

47. Drupal On-Demand Kudos to Four Kitchens Austin-based Drupal consultant shop

    One of their own is the author of Pressflow Helped in design and build of early Yale Drupal I dont speak for Yale, but I highly-recommend them Yale/4K code given back to drupal.org

48. Drupal On-Demand Questions? Ask away. Drupalers at Yale want to:

    Help others facing similar challenges Be public with our work Perhaps give it away We already did with some Many Edus, One Wheel