Web application performance tuning beyond Xmx

The slides from my "Web application performance tuning beyond Xmx" talk

Transcript

1. @YourTwitterHandle #DV14 #YourTag @MilenDyankov #DV14 #beyondXmx Web application performance tuning beyond Xmx Milen Dyankov
2. This talk is not about JVM and GC tuning ! Disappointed? You shouldn't be, there are plenty of other talks out there!
3. This talk is not about Liferay either ! However you may want to check it out anyway! It is a great platform!
4. This is not a university course This is not a university course on performance tuning! on performance tuning! I'm just sharing my personal I'm just sharing my personal experience with you! experience with you!
5. performance tuning expert
7. Customer's statement: “It is very slow! You need to improve the performance!” What would you do?
8. Customer's statement: “It is very slow! You need to improve the performance!” What would you do? Check the CPU and Check the CPU and memory usage memory usage Check the Check the JVM parameters JVM parameters Check the database Check the database for slow queries for slow queries
9. Check the CPU and Check the CPU and memory usage memory usage Check the Check the JVM parameters JVM parameters Check the database Check the database for slow queries for slow queries While those are valid points, they are all the WRONG answer! Customer's statement: “It is very slow! You need to improve the performance!” What would you do?
10. Ask to provide Ask to provide numbers! numbers! Ask to provide Ask to provide expectations! expectations! Ask to provide Ask to provide units! units! Customer's statement: “It is very slow! You need to improve the performance!” What would you do?
11. Ask to provide Ask to provide numbers! numbers! Ask to provide Ask to provide expectations! expectations! Ask to provide Ask to provide units! units! Customer's statement: “It is very slow! You need to improve the performance!” What would you do? Make sure everyone uses the same definitions and units to describe both the status and the expectations!
12. “It can only serve N users, we expect it to serve MxN users”
13. “It can only serve N users, we expect it to serve MxN users” N users accessing the system N users accessing the system at random times during given period! at random times during given period! N users accessing the system N users accessing the system as a result of marketing campaign! as a result of marketing campaign! N users accessing the system N users accessing the system every day between 8:00 and 8:15! every day between 8:00 and 8:15! consider these
14. “It can only serve N users, we expect it to serve MxN users” N users accessing the system N users accessing the system at random times during given period! at random times during given period! N users accessing the system N users accessing the system as a result of marketing campaign! as a result of marketing campaign! N users accessing the system N users accessing the system every day between 8:00 and 8:15! every day between 8:00 and 8:15! consider these Don't confuse ability to scale with performance tuning! Use unambiguous or less ambiguous units!
15. Unambiguous Ambiguous Business Technical users users concurrent users concurrent users concurrent sessions concurrent sessions concurrent requests concurrent requests requests per second requests per second (M/K)bytes per second (M/K)bytes per second But keep in mind: sometimes being too technical leads to the wrong decisions! Use unambiguous or less ambiguous units!
16. Consider this statement: Consider this statement: “N needs to travel from town A to town B. What is the fastest possible way?” But keep in mind: sometimes being too technical leads to the wrong decisions!
17. Consider this statement: “N needs to travel from town A to town B. What is the fastest possible way?” M x But keep in mind: sometimes being too technical leads to the wrong decisions! Refinement: N is a group of M passengers
18. Refinement: N is a group of M passengers Consider this statement: “N needs to travel from town A to town B. What is the fastest possible way?” But keep in mind: sometimes being too technical leads to the wrong decisions!
19. How to do performance tuning?
20. It's kind of like … … drawing an Owl! How to do performance tuning?
21. • First draw 2 ellipses! It's kind of like … … drawing an Owl! How to do performance tuning?
22. • First draw 2 ellipses! • Then draw the rest of the damn Owl! It's kind of like … … drawing an Owl! How to do performance tuning?
23. units = defineUnits(); expectations = defineExpectations(units); results = measurePerformance(); while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); results = measurePerformance(); } Enough analogies! Let's try with an algorithm: How to do performance tuning?
24. units = defineUnits(); expectations = defineExpectations(units); results = measurePerformance(); while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); results = measurePerformance(); } Enough analogies! Let's try with an algorithm: How to do performance tuning?
25. units = defineUnits(); expectations = defineExpectations(units); results = measurePerformance(); while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); results = measurePerformance(); } Enough analogies! Let's try with an algorithm: How to do performance tuning?
26. Application measurePerformance(); measurePerformance();
27. Application measurePerformance(); measurePerformance();
28. units = defineUnits(); expectations = defineExpectations(units); - results = measurePerformance() + data = collectData(); + results = analyze(data) while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); - results = measurePerformance() + data = collectData(); + results = analyze(data) } Let's refine the algorithm: Application measurePerformance(); measurePerformance();
29. units = defineUnits(); expectations = defineExpectations(units); - results = measurePerformance() + data = collectData(); + results = analyze(data) while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); - results = measurePerformance() + data = collectData(); + results = analyze(data) } Let's refine the algorithm: Application measurePerformance(); measurePerformance();
30. Browser Network Server JVM Container Application Browser Network . . . collectData(); collectData();
31. Browser Network Server JVM Container Application Browser Network . . . Take into account your target group! Respect “Cache-Control” and “Expires” headers ! Home page is not the main entrance anymore! Consider search results and referrals! Take ONLY what you REALLY need! Measure the impact of 3rd party code (analytics, ads, …)! collectData(); collectData();
32. Browser Network Server JVM Container Application Browser Network . . . You are NOT the only creature in the space and time! Understand who else is using the same network, what for and when! A network is, what the name says – a net, not a line! There may be many networks and interfaces involved! Network traffic goes in two directions! No problem on the way in, does NOT mean no problem on the way out! (or vice-versa) collectData(); collectData();
33. Browser Network Server JVM Container Application Browser Network . . . “Server” is one of the most ambiguous words ! Amazon, Jelastic, … are servers as well … but different! Virtual resources are … well, VIRTUAL! “… 3 VMs (8GB of RAM each) … oh the server has 16GB of RAM” Operating System will only tell you what you ask for ! nice, top, htop, atop, dstat, vmstat, ifstat, iostat, … collectData(); collectData();
34. Browser Network Server JVM Container Application Browser Network . . . Know what your workers are doing! Regular thread dumps help find long running threads! Garbage collection of course! Everyone knows that! -verbose:gc -Xloggc:... -XX:+PrintGCDetails Tools are nice but they also consume resources! Don't measure having the profiler attached! collectData(); collectData();
35. Browser Network Server JVM Container Application Browser Network . . . Policies, politics, preferences and common sense! Don't fight against windmills! Sessions tend to grow, last forever and … … replicate! Security, audits, transactions … , Security, audits, transactions … , … … and interceptors in general! and interceptors in general! collectData(); collectData();
36. Browser Network Server JVM Container Application Browser Network . . . Text Object XML Object JSON … ! → → → → → Simultaneous translation is not an easy job! Many a mickle makes a muckle! Many a mickle makes a muckle! Number of calls is as important as the time the call takes! Number of calls is as important as the time the call takes! Cache stats – hits, misses, invalidation, overflows, ... Not caching anything? Great! You just found the bottleneck! collectData(); collectData();
37. Browser Network Server JVM Container Application Browser Network . . . Databases External storages Frameworks 3rd party services . . . Last, but not least, pay attention to all those little tiny things that makes an Owl look like an Owl! Oh and here is something that is valid for about everything we covered so far: You can really make use of another pair of eyes! collectData(); collectData();
38. “It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong.” Richard P. Feynman Do I really have to do all that? You better do! Here is why:
39. Do I really have to do all that? You better do! Here is why:
40. Even the guys behind BIG names sometimes don't do it right! 2014, Louvre Museum, Paris, France Do I really have to do all that? You better do! Here is why:
41. THE USUAL SUSPECTS is a movie title not a problem solving strategy! Do I really have to do all that? You better do! Here is why:
42. units = defineUnits(); expectations = defineExpectations(units); - results = measurePerformance() + data = collectData(); + results = analyze(data) while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); - results = measurePerformance() + data = collectData(); + results = analyze(data) } Back to the algorithm:
43. units = defineUnits(); expectations = defineExpectations(units); - results = measurePerformance() + data = collectData(); + results = analyze(data) while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); - results = measurePerformance() + data = collectData(); + results = analyze(data) } Back to the algorithm:
44. 1 2 3 analyze( analyze(data data); );
45. The Base Rate Fallacy When people have both generic and specific information, they tend to make judgments of probability based entirely upon specific information, leaving out the base rate. analyze( analyze(data data); );
46. Congruence bias A hypothesis is tested only by thinking of results that would be found if that hypothesis is true. This heuristic, which many people seem to use, ignores alternative hypotheses analyze( analyze(data data); );
47. units = defineUnits(); expectations = defineExpectations(units); - results = measurePerformance() + data = collectData(); + results = analyze(data) while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); - results = measurePerformance() + data = collectData(); + results = analyze(data) } Back to the algorithm:
48. units = defineUnits(); expectations = defineExpectations(units); - results = measurePerformance() + data = collectData(); + results = analyze(data) while (!results.match(expectations)) { component = findLowestPerforming(results); bottleneck = findBottleneck (component); remove(bottleneck); - results = measurePerformance() + data = collectData(); + results = analyze(data) } Back to the algorithm:
49. I'm sorry! I don't have any universal solution yet! But I will be happy to give you some links, tips and warnings if you don't mind! remove( remove(bottleneck bottleneck); );
50. Browser Network Server JVM Container Application Browser Network . . . “I sometimes go into a client site and got 3X improvements on performance just with some tuning of the operating system and the ways VMs are running.” Martin Thompson http://www.infoq.com/presentations/top-10-performance-myths Step out of the comfort zone! remove( remove(bottleneck bottleneck); );
51. Browser Network Server JVM Container Application Browser Network . . . Avoid tuning by folklore! remove( remove(bottleneck bottleneck); ); Don't copy JVM settings from various blogs! They are not meant to be for you!
52. Browser Network Server JVM Container Application Browser Network . . . Blocking vs Nonblocking Cache Data structures ( the famoust ArrayList vs LinkedList interview question is for a reason ) Method complexity ( StringBuilder vs String.concat ) Logging Know what the code you use actually do! remove( remove(bottleneck bottleneck); );
53. Browser Network Server JVM Container Application Browser Network . . . Going multi-thread ! Going asynchronous ! Going modular! Going multi-vm ! Going distributed! Going off-heap! Get (more) things done! remove( remove(bottleneck bottleneck); );
55. SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS Tuning the unknown! “A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools.” The Hitchhiker's Guide to the Galaxy Douglas Adams
56. There are 2382 configuration properties in Liferay 6.2 Make it configurable! SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS Tuning the unknown!
57. Document it ! There are 6896 lines of comments documenting Liferay 6.2 properties! SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS Tuning the unknown!
58. Provide reasonable defaults ! Example: # # If the user can unzip compressed HTTP content, # the GZip filter will zip up the HTTP content # before sending it to the user. This will speed up # page rendering for users that are on dial up. # com.liferay.portal.servlet.filters.gzip.GZipFilter=true SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS Tuning the unknown!
59. Provide performance results for most common cases! SPECIAL CASE FOR FRAMEWORK AND LIBRARY DEVELOPERS Tuning the unknown!
60. @YourTwitterHandle #DV14 #YourTag @MilenDyankov #DV14 #beyondXmx Photo: http://www.flickr.com/photos/wwworks/4759535950/ http://www.liferay.com @Liferay http://www.facebook.com/Liferay milen.dyankov@liferay.com http://www.liferay.com/web/milen.dyankov/ @milendyankov @LiferayPL