High-Scalability and High-Availability at ImmobilienScout24

High-Scalability and High-Availability at ImmobilienScout24

0b5d242584112741f7c14c9a2e1a299d?s=128

André Neubauer

October 12, 2010
Tweet

Transcript

  1. André Neubauer, Oliver Schmitz-Hennemann | ImmobilienScout24 High-Scalability and High-Availability at

    ImmobilienScout24
  2. I‘m André Neubauer. You can find me at… • Team Leader

    IT Development • Researcher, Speaker, Author #web #devops #mobileWeb #android #ar andre.neubauer@... twitter.com/devpg devpg.com
  3. I‘m Oliver Schmitz-Hennemann. You can find me at… schmitz@is24.de twitter.com/OSchmi

    • Head of IT Architecture • OrgDev, agile evangelist, inconvenient
  4. Facts • World’s largest Internet market place for real estate (revenue

    2009) Source: propertyportalwatch.com • 11 years in the Internet business • ~ 500 employees • Agile (Scrum) since 2008
  5. Facts Business Architecture Platform Vendor Private Professional Search ~95% Misc

  6. Facts – Trends >1.500.000.000 >99,9%

  7. Facts • ~ 5 m unique users/ month • >1.500 page views/

    second (peak) • >700 (virtual) instances on 100 hosts • 100 physical systems additionally • 2 data centers • ~1.5 m lines of code
  8. How Could We Achieve That? Luck Blood Sweat

  9. How Could We Achieve That? Individuals Technology Process

  10. Individuals • Cross-teams • Software craftsmanship • Responsibility

  11. Individuals – Cross-Teams PM Dev QA Arc Ops

  12. Individuals - Software Craftsmanship • Reliable and maintainable systems

  13. Individuals - Responsibility • Scouty principle/ continuous refactoring Code Quality Speed

    Time
  14. PROCESS

  15. Process • Continuous product development • Team architecture review • Application Monitoring • Incident

    Management
  16. Process – Continuous Product Development • No challenge to work in

    short iterations • Create a shippable product increment is more difficult (technical as well as on business) • Arc violations break the build
  17. Process – Team Architecture Review • Decide on architectural changes • Balance

    interests • Consensus decision • Each member stands by the decision
  18. Process – Application Monitoring Fail early is okay but should

    know why
  19. Process – Incident Management • Name a officer on duty • Decide

    on defense conditions • Inform the management • Set up a task force and a war room • Understand the symptom • Decide on next steps • Fix it! Nothing beats an incident!
  20. Technology -  Infrastructure -  Software architecture -  ARC2010 -  Database

    -  Scale -  Environment
  21. Technology – Infrastructure

  22. Technology – Infrastructure

  23. Technology – Architecture ServicePool Corba Skeletons Search Index MT UTIL

    Business Object ServiceImpl Web Tier Apache Tomcat Web UTIL ConfigServlet Web Service Service Wrappers Corba Stubs web modules servlet filters Portallogic JSP Dialog Workflow
  24. Technology – ARC2010 -  Reduce complexity -  Standards, guidelines

  25. Technology – Database -  Oracle Enterprise 8, 9, 10 (+4

    RO), 11 -  Currently 2 node RAC -  Load not a problem, except full table/ index scans -  Performance issues mostly „minor“ traffic usecases
  26. Technology – Database -  Never tried searching on DB - 

    History says, we could not afford a suitable machine in the beginning -  Multi attribute, complex sorting
  27. Technology – Scale -  Memory based architecture since 1999 - 

    Legacy in Memory search index -  NoSQL – Result cache based on a standard java.util.HashMap
  28. Technology – Scale -  Just being replaced with Lucene/ SOLR

    -  Free text search
  29. Technology - Environment -  Different stages -  Development (rapid response,

    continuous build) -  Testing (quality assurance) -  Load-testing -  Production
  30. Technology – Failover

  31. Technology Caching/ CDN Failover RW-Split … Great Stuff … …

    but adds complexity to your application.
  32. Individuals Technology Process

  33. Lessons Learned • Hire excellent people • Stay agile, be open for

    innovation • Process should be a helpful tool • Remind debugging rules • Problems change with scale • KISS & keep complexity low
  34. WHAT‘S NEXT?

  35. What‘s Next? • DevOps • Continous Deployment -  Functional application split up

  36. Continuous Deployment • It‘s not about deploying every change • Deliver a

    business value • “Feature after acceptance by PO will be LIVE on our portal within 24h” [CTO]
  37. THANK YOU FOR YOUR ATTENTION! ANY QUESTIONS?

  38. Process (Backup) “Process or processing typically describes the act of

    taking something through an established and usually routine set of procedures to convert it from one form to another […]” [Wikipedia] • Should be a useful tool –  No waste –  Outweigh complexity