SECURING THE DEVOPS LIFECYCLE Jarret Raim, BSides Austin, April 2012 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! WHO AM I? Jarret Raim – Security Intrapreneur 2! DEVELOPER ACADEMIC APPLICATION SECURITY SECURITY CONSULTANT SECURITY PRODUCTS [email protected]! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! RACKSPACE® HOSTING The Service Leader in Cloud Computing 3! 172,000+ CUSTOMERS 4,000+ RACKERS 9 GLOBAL DATA CENTERS 120 + COUNTRIES 2008, 2010, 2011 & 2012 LEADER IN GARTNER'S MAGIC QUADRANT FOR MANAGED HOSTING 40%   FORTUNE® 100 OF THE WE SERVE RAX 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 4! OUR VISION To be recognized as one of the World’s greatest service companies. “ ” 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! RAX CLOUD APPROACH Open source orchestration, management & provisioning cloud platform ! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! Applications OPENSTACK™ Management Layer that Adds Automation and Control Creates Pools of Resources Automates The Network USERS ADMINS CLOUD OPERATING SYSTEM Connects to apps via APIs Self-service Portals for users Stage II Cloud Data Center Stage III Cloud Federation Stage I Server Virtualization 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! §  Scalable Web Apps §  Test, Development, Production §  SaaS/Resellers/Web Developers Dedicated AND Cloud Two independent applications DEDICATED •  Customizable •  Compliance •  High Performance APP 1 CLOUD HOSTING •  Flexible •  Elastic •  Pay-as-you-go APP 2 HYBRID HOSTING RAX HYBRID CLOUD SOLUTIONS 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! PROVIDER DC! CUSTOMER SITE! RACKSPACE LOCATIONS! 8! DEDICATED! PUBLIC CLOUD! PRIVATE CLOUD! PRIVATE CLOUD! PUBLIC CLOUD! Rackspace Provides! The Fanatical Support! •  One Control Panel across OpenStack connected clouds! •  One Fanatical Support Team! •  Our Cloud, Your Cloud, Partner Hosted OpenStack Cloud! •  Global Reach! THE FUTURE: FANATICAL SUPPORT ANYWHERE 8 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 9! SECURING THE DEVOPS LIFECYCLE 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 10! • Cycle time for software is getting shorter! • Continuous delivery is a goal! • Scanning windows are not viable! ! THE PROBLEM 10 "DevOps" is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/ infrastructure) professionals. 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 11! • Automated software testing! • Automated operational infrastructure! • Automated security testing! THE SOLUTION 11 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 12! AUTOMATING 12 • Declarative configuration language! • Plain-text configuration in source control! • Fully programmatic, no manual interactions! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 13! CHEF 13 1.  Solo! 2.  Server! 3.  Hosted! 4.  Private Hosted! Node! Node! Node! Node! Node! Node! Node! Node! Node! Node! Node! Node! Node! Node! Node! Racker! Server / Hosted / Private! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 14! COOKBOOKS 14 • Most major software packages have cookbooks! • You will have to write your own / customize! • Good place to spend security cycles! - Merge patches upstream for extra points.! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 15! GROUPING & TAGGING 15 • Tagging your servers applies the required set of recipes! • A base set of recipes is common! • Each server will have multiple tags set at bootstrap time! Node! Node! Node! Node! DB! Node! Node! Node! Node! Cache! Node! Node! Node! Node! Web! Apache! Monitoring! MySql! Memcache! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 16! LIMITATIONS 16 • Focus on single machines! • A multi-box configuration is based on copying existing configurations! • No support for implicit application or environment configuration! • Applications include more than just servers! • Images have security issues! Web! Web! Web! Cloud Load Balancer! Memcached! Database as a Service! Web! Cloud Files! CDN! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 17! CHECKMATE 17 A system to build generic application configurations! Architect   • Templates   • Ques2ons   Contractor   • Decomposi2on   • Orchestra2on   Inspector   • Verifica2on   • Due  Diligence   

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 18! ARCHITECTURE 18 Message   Queue   Architect   Contractor   Inspector   Checkmate   Web   Message   Queue   Compute   Storage   Load   Balancer   Database   Hadoop   Caching   • Components communicate through a common queue! • Each provisioning component is independent! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 19! ARCHITECT 19 Template! Generic Provider Definitions! Architecture Questions! Scaling Factors! base:      name:  wordpress  large      environment-­‐name:  {tenantId}-­‐ wordpress-­‐large     providers:      -­‐  rackspace:          -­‐  compute:  &rax-­‐cloud-­‐servers              endpoint:  https://...          -­‐  loadbalancer:  &rax-­‐lbaas              endpoint:  https://...          -­‐  database:  &rax-­‐dbaas              endpoint:  https://...          -­‐  common:          vendor:  rackspace          credentials:              -­‐  token:  {token}   

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 20! ARCHITECT 20 Template! Generic Provider Definitions! Architecture Questions! Scaling Factors! • Requests per hour?! • Budget! • High availability! • Disaster resistant! • SSL! • Backup! • CDN! …! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 21! ARCHITECT 21 Template! Generic Provider Definitions! Architecture Questions! Scaling Factors! tiers:   -­‐  name:  web      resource:  &loadbalancer          min-­‐occur:  1          type:  loadbalancer          connection:  public              port:  [80,  443]              allow:  all              isolation:  none      resource:  &webheads          min-­‐occur:  2          type:  compute          os:  Ubuntu  11.10          memory-­‐min:  2Gb          memory-­‐max:  4Gb          configs:          -­‐  wordpress-­‐mp              attributes:              -­‐  role:  web              connection:  *database   

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 22! ARCHITECT 22 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 23! CONTRACTOR 23 • Takes Architect’s plan and builds it! • Task Decomposition! - Uses standard workflow patterns! • Orchestration / Ordering! • Status Reporting! • Farms out tasks to sub- contractors! Our current implementation uses an open source Python workflow engine, SpiffWorkflow.! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 24! INSPECTOR 24 • Takes Architect’s plan & contractor’s output! • Focuses on checking for code compliance! - Not perfection, bare minimums! • Can include multiple facets! - Security! - Scalability! - Compliance! Our current implementation includes WP Scan for WordPress and the Nikto vulnerability scanner.! 

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 25! INSPECTOR 25 +  Server:  Apache/2.2.12  (Ubuntu)   +  No  CGI  Directories  found  (use  '-­‐C  all'  to  force  check  all   possible  dirs)   +  Apache/2.2.12  appears  to  be  outdated  (current  is  at  least  Apache/ 2.2.17).  Apache  1.3.42  (final  release)  and  2.0.64  are  also  current.   +  ETag  header  found  on  server,  inode:  12534048,  size:  317,  mtime:   0x4b9436dbea280   +  Allowed  HTTP  Methods:  GET,  HEAD,  POST,  OPTIONS     +  OSVDB-­‐3268:  /icons/:  Directory  indexing  found.   +  OSVDB-­‐3233:  /icons/README:  Apache  default  file  found.   +  6448  items  checked:  0  error(s)  and  5  item(s)  reported     

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! 26! INSPECTOR 26 [!]  The  WordPress  "http://-­‐-­‐-­‐.com/readme.html"  file  exists.   [!]  WordPress  version  3.1  identified  from  meta  generator.     [+]  Enumerating  installed  plugins...Checking  for  2394  total   plugins   [+]  We  found  2  plugins:   Name:  disqus-­‐comment-­‐systemLocation:     Name:  wordpress-­‐popular-­‐postsLocation:       [+]  There  were  1  vulnerabilities  identified  from  the  plugin   names:     [!]  ["WordPress  Plugin  Disqus  Comment  System  <=  2.68  Reflected   Cross-­‐Site  Scripting  (XSS)"]*     

RACKSPACE® HOSTING | WWW.RACKSPACE.COM ! Architect   • Templates   • Ques2ons   Contractor   • Decomposi2on   • Orchestra2on   Inspector   • Verifica2on   • Due  Diligence   Monitor   • Trending   • Thresholding   27! FUTURE WORK 27 

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218! US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM! RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM! ANY QUESTIONS?