High-Scalability and High-Availability at ImmobilienScout24

Transcript

1. André Neubauer, Oliver Schmitz-Hennemann | ImmobilienScout24 High-Scalability and High-Availability at ImmobilienScout24
2. I‘m André Neubauer. You can find me at… • Team Leader IT Development • Researcher, Speaker, Author #web #devops #mobileWeb #android #ar andre.neubauer@... twitter.com/devpg devpg.com
3. I‘m Oliver Schmitz-Hennemann. You can find me at… schmitz@is24.de twitter.com/OSchmi • Head of IT Architecture • OrgDev, agile evangelist, inconvenient
4. Facts • World’s largest Internet market place for real estate (revenue 2009) Source: propertyportalwatch.com • 11 years in the Internet business • ~ 500 employees • Agile (Scrum) since 2008
5. Facts Business Architecture Platform Vendor Private Professional Search ~95% Misc
6. Facts – Trends >1.500.000.000 >99,9%
7. Facts • ~ 5 m unique users/ month • >1.500 page views/ second (peak) • >700 (virtual) instances on 100 hosts • 100 physical systems additionally • 2 data centers • ~1.5 m lines of code
8. How Could We Achieve That? Luck Blood Sweat
9. How Could We Achieve That? Individuals Technology Process
10. Individuals • Cross-teams • Software craftsmanship • Responsibility
11. Individuals – Cross-Teams PM Dev QA Arc Ops
12. Individuals - Software Craftsmanship • Reliable and maintainable systems
13. Individuals - Responsibility • Scouty principle/ continuous refactoring Code Quality Speed Time
14. PROCESS
15. Process • Continuous product development • Team architecture review • Application Monitoring • Incident Management
16. Process – Continuous Product Development • No challenge to work in short iterations • Create a shippable product increment is more difficult (technical as well as on business) • Arc violations break the build
17. Process – Team Architecture Review • Decide on architectural changes • Balance interests • Consensus decision • Each member stands by the decision
18. Process – Application Monitoring Fail early is okay but should know why
19. Process – Incident Management • Name a officer on duty • Decide on defense conditions • Inform the management • Set up a task force and a war room • Understand the symptom • Decide on next steps • Fix it! Nothing beats an incident!
20. Technology -  Infrastructure -  Software architecture -  ARC2010 -  Database -  Scale -  Environment
21. Technology – Infrastructure
22. Technology – Infrastructure
23. Technology – Architecture ServicePool Corba Skeletons Search Index MT UTIL Business Object ServiceImpl Web Tier Apache Tomcat Web UTIL ConfigServlet Web Service Service Wrappers Corba Stubs web modules servlet filters Portallogic JSP Dialog Workflow
24. Technology – ARC2010 -  Reduce complexity -  Standards, guidelines
25. Technology – Database -  Oracle Enterprise 8, 9, 10 (+4 RO), 11 -  Currently 2 node RAC -  Load not a problem, except full table/ index scans -  Performance issues mostly „minor“ traffic usecases
26. Technology – Database -  Never tried searching on DB -  History says, we could not afford a suitable machine in the beginning -  Multi attribute, complex sorting
27. Technology – Scale -  Memory based architecture since 1999 -  Legacy in Memory search index -  NoSQL – Result cache based on a standard java.util.HashMap
28. Technology – Scale -  Just being replaced with Lucene/ SOLR -  Free text search
29. Technology - Environment -  Different stages -  Development (rapid response, continuous build) -  Testing (quality assurance) -  Load-testing -  Production
30. Technology – Failover
31. Technology Caching/ CDN Failover RW-Split … Great Stuff … … but adds complexity to your application.
32. Individuals Technology Process
33. Lessons Learned • Hire excellent people • Stay agile, be open for innovation • Process should be a helpful tool • Remind debugging rules • Problems change with scale • KISS & keep complexity low
34. WHAT‘S NEXT?
35. What‘s Next? • DevOps • Continous Deployment -  Functional application split up
36. Continuous Deployment • It‘s not about deploying every change • Deliver a business value • “Feature after acceptance by PO will be LIVE on our portal within 24h” [CTO]
37. THANK YOU FOR YOUR ATTENTION! ANY QUESTIONS?
38. Process (Backup) “Process or processing typically describes the act of taking something through an established and usually routine set of procedures to convert it from one form to another […]” [Wikipedia] • Should be a useful tool –  No waste –  Outweigh complexity