Good Server Engage User Group Conference, Eindhoven March 2016 Serdar Basegmez - Developi - @serdar_basegmez William Malchisky Jr. - ESS - @BillMalchisky
Creative Commons BY-NC-SA 4.0 2 • Preface • Chapter I - The Beginning • Chapter 2 - Searching for Clues • Chapter 3 - CreaSng a Solid PlaTorm • Chapter 4 - The SoUside of Performance Gains • The Final Chapter - Results Our Story in Forty Minutes
Creative Commons BY-NC-SA 4.0 3 "Ladies and Gentlemen. The story you are about to see is true; the names have been changed to protect the innocent." --Dragnet For example... Acme Corporation is now referred to as Acme, Inc. Disclaimer
Creative Commons BY-NC-SA 4.0 4 • What we will cover • Problem analysis • TroubleshooSng skills • Best pracSces • The performance impact of subopSmal applicaSons • What we omi[ed • Boring, rambling, dry, lectures • Useless drivel Se^ng ExpectaSons
Creative Commons BY-NC-SA 4.0 5 • Preface • Chapter I - The Beginning • Chapter 2 - Searching for Clues • Chapter 3 - CreaSng a Solid PlaTorm • Chapter 4 - The SoUside of Performance Gains • The Final Chapter - Results Our Story in Forty Minutes
Creative Commons BY-NC-SA 4.0 6 • "We're having a problem. Can you help?" • "Absolutely. What's happening?" • "Our mission criScal DB is really $%&@#$^& our users. It's way too slow. It takes less Sme to reboot [Windows 3.1 on an i386 with 32MB RAM] than to open a document." • "Any idea what changed?" • "We don't know. We have not touched the box." Customer Calls
Creative Commons BY-NC-SA 4.0 8 • Preface • Chapter I - The Beginning • Chapter 2 - Searching for Clues • Chapter 3 - CreaSng a Solid PlaTorm • Chapter 4 - The SoUside of Performance Gains • The Final Chapter - Results Our Story in Forty Minutes
Creative Commons BY-NC-SA 4.0 9 • While waiSng for access... request the following • Helps establish the level of criScality "Round Up the Usual Suspects" notes.ini log.sf sh tasks top vmstat iosys df -h User to server ping results mount swapon -s Server NAB DB copy, sans users
Creative Commons BY-NC-SA 4.0 11 • Run DCT - returned a few items, but nothing applicable to the performance issue experienced • Check Domino stats • Located a key issue - needle in haystack • SAI fluctuated wildly, frequently, plummeSng to 18% for minutes on end • Locate any recent NSD files for analysis Data, Data Everywhere
Creative Commons BY-NC-SA 4.0 12 • Watch the server when nobody else does • Lots of strange things happen on servers overnight • Observed the system processing over one million records in :15 twice a week, at different Smes • For example… no one at Acme, Inc. knew this occurred or why Pro Tip on Data CollecSon
Creative Commons BY-NC-SA 4.0 13 • Swap space 50% of installed memory • Memory was under 1GB for mission criScal server • Several key DBs contained 100k+ docs • CombinaSon created page faulSng plague further eroding performance • System properly patched • Free space adequate IniSal Data Analysis - OS
Creative Commons BY-NC-SA 4.0 14 • Obvious but important data points • Server layout • Where items located • Recognized server.id file • Server tasks • Contrast to sh tasks requested earlier • No obvious problems IniSal Data Analysis - Notes.ini
Creative Commons BY-NC-SA 4.0 15 • Agents running all hours of the night and day • Agents running from DBs acSvely being compacted • Agents running from DBs when updall and fixup running • Not all scheduled agents needed to run all weekend IniSal Data Analysis - Amgr
Creative Commons BY-NC-SA 4.0 16 • Compact sSll running when updall Program fires-off • Compact never finished before execuSon Sme ceiling hit • LeU largest DBs in a completely subopSmal state • Connected to servers that did not exist • Scheduled replicaSon documents • Significant delays with replica synchronizaSon • Ensured data never properly synchronized across domain • Certain connecSon documents only covered two DBs IniSal Data Analysis - Log.sf
Creative Commons BY-NC-SA 4.0 17 • Several big DBs last fixup completed two years ago • Most heavily used files 30-75% Used • Many views means clicking one forces a new index build • No design, document, or a[achment compression • Design server task ciSng non-existent templates IniSal Data Analysis - DBs
Creative Commons BY-NC-SA 4.0 18 • Preface • Chapter I - The Beginning • Chapter 2 - Searching for Clues • Chapter 3 - CreaSng a Solid PlaTorm • Chapter 4 - The SoUside of Performance Gains • The Final Chapter - Results Our Story in Forty Minutes
Creative Commons BY-NC-SA 4.0 19 • Swap space - No set rule these days • 1.5x - 2.0x RAM is good rule of thumb • Memory - 4GB per processor on busy servers • VMware se^ngs if available • Avoid temptaSon of too many processors • Review parSSons and free space Tier 1 - OS
Creative Commons BY-NC-SA 4.0 20 • Check that previous made system changes sSck • Unfamiliar servers can exhibit odd behavior • Check Technotes for any recent performance issues • Once OS is working, check to ensure that virtualizaSon is opSmal AddiSonal OS ConsideraSons
Creative Commons BY-NC-SA 4.0 21 • Space properly Program Documents • Avoid overlap with agents and other Programs • Pause agent schedule during maintenance • Schedule a weekend to complete first full maintenance • First full compact will take much longer than you realize • Create maintenance schedule of tasks agreed to by business line managers • Ensures all needed jobs are available when needed Tier 2 - Domino
Creative Commons BY-NC-SA 4.0 22 • Review all enabled Domino features to ensure that they funcSon properly • Simple configuraSon miscues can impact negaSvely • Cluster replicaSon unable to locate a cluster member • DNS errors create lookup delays • Remove unneeded, deprecated network ports AddiSonal Items to Fix
Creative Commons BY-NC-SA 4.0 23 • Preface • Chapter I - The Beginning • Chapter 2 - Searching for Clues • Chapter 3 - CreaSng a Solid PlaTorm • Chapter 4 - The SoUside of Performance Gains • The Final Chapter - Results Our Story in Forty Minutes
Creative Commons BY-NC-SA 4.0 24 • Domino Admin handled the first level treatment • Server performs well, but not good enough • Triangulated the issue to a mission-criScal applicaSon • Now what? Where are We?
Creative Commons BY-NC-SA 4.0 26 • There is no magic pill for finding a performance issue • Many problems are circumstanSal • Depends on who/when/how… • RepeaSng the problem on a controlled environment • Need for Proof! • The most difficult part of the task • Need to be systemaKcal Developers vs Performance Issues
Creative Commons BY-NC-SA 4.0 28 Methodology Research ✤ Symptoms (e.g. logs, performance data, etc.) ✤ Story (e.g. user input) ✤ Application code Hypothesis ✤ Speculation on possible reasons ✤ Search for ‘Usual Suspects’ Experiment ✤ Testing for possible reasons Analyze ✤ Check symptoms if fixed Conclusion ✤ Issue validated and proved to be fixed.
Creative Commons BY-NC-SA 4.0 29 • What to collect, based on the symptom; • CPU/memory load, hangs, spikes, crashes, etc. • All the Sme, the same Sme everyday or random? • Experienced by specific users? • We are looking for a pa[ern between incidents. Research & Assessment
Creative Commons BY-NC-SA 4.0 30 Log/NSD/Semaphore files Server configuraSon (inc. notes.ini) Server monitoring and staSsScs data Web logs (for web applicaSon issues) XPages and OSGi logs (for XPages specific issues) ApplicaSon and dependencies Data CollecSon Checklist
Creative Commons BY-NC-SA 4.0 31 • SomeSmes, even opening in DDE may cause issues! • e.g. XPages components are automaScally built • ApplicaSon code might have side effects • e.g. UpdaSng on another data source, adding audit logs, performance degradaSon on the server, etc. • There will be dependencies • Once isolated, we can start inspecSon… Isolate the ApplicaSon
Creative Commons BY-NC-SA 4.0 32 • Database corrupSons • @Today/@Now in views • Code snippets acSng like an admin • UpdaSng views, replicaSng databases, running server commands, etc. • Code snippets using the worst pracSces • Search in a large database, wrong looping, etc. • Anything that fits into the pa[ern if there is one • e.g. An agent matching the incident Sming Usual Suspects
Creative Commons BY-NC-SA 4.0 34 • Deeper invesSgaSon needs a teaming effort • Admins and Developers should collaborate • A test setup to simulate the producSon environment • Intensive / Controlled debugging sessions in limited Sme windows • Sharing experSse • ExperimenSng on producSon should be the last resort • Once a repeatable error found, cooperate for a soluSon Team Up!
Creative Commons BY-NC-SA 4.0 35 • JVM Crash with the HTTP task • Random Smes • No pa[ern in the log • Memory dumps point a leak in the JVM Heap • Inspected XPages applicaSons, nothing found • Triangulated the problem into one XPages app, following clues in intensive debugging on memory • Isolated the applicaSon for a load test, nothing found • Increased logging, to collect more data, no hope! Example Case - Analysis
Creative Commons BY-NC-SA 4.0 36 • Checked the server configuraSon and noSced • Logging data incomplete • Removed exclusions • New logs pointed the problem • Searching soUware crawling a specific page • Page generates state data, fills up the memory • Simulated the same crash on the test environment • One line of code fixed the issue Example Case - ResoluSon
Creative Commons BY-NC-SA 4.0 37 • A mission criScal applicaSon at a bank • Web applicaSon with 2000+ users • CPU spikes and random hangs, mostly aUernoon • Logs are clear, no crashes, no error messages • Isolated the applicaSon, inspected the ‘usual suspects’ • Found a web agent updaSng a view! • Triangulated the problem using web logs and SEMDEBUG • But, cannot validate the issue on the test environment… Another Case - Analysis
Creative Commons BY-NC-SA 4.0 38 • Cooperated with the Domino Admin • Detailed assessment on the server configuraSon • We found the issue! • “ServerTasksAt14” running an updall task. • Another Program file running Updall on a specific database, every 30 minutes • Applied to the test plaTorm, validated by a load test • Problem solved! Another Case - ResoluSon
Creative Commons BY-NC-SA 4.0 39 • Preface • Chapter I - The Beginning • Chapter 2 - Searching for Clues • Chapter 3 - CreaSng a Solid PlaTorm • Chapter 4 - The SoUside of Performance Gains • The Final Chapter - Results Our Story in Forty Minutes
Creative Commons BY-NC-SA 4.0 40 • Page faults reduced to zero • General DB usage and administraSon tasks work well • SAI now over 80% • Weird overnight (agent) system operaSons resolved • Key DBs have 93% used space threshold now • All DBs compressed: design, documents, all a[achments • Program documents, agent schedules all adjusted: finish, no overlap Quality Analysis Yields Quality Results
Creative Commons BY-NC-SA 4.0 43 • You can get help inspecSng applicaSons and servers! • They have also helped Engage! Bonus Slide cooperteam MartinScott teamstudio Ytria
Creative Commons BY-NC-SA 4.0 45 • IBM Champion (2011 - 2016) • EffecSve SoUware SoluSons, LLC • Co-founder of Linuxfest at Lotusphere/Connect • Speaker at 20+ Lotus/IBM related events/LUGs • Co-authored two IBM Redbooks • Co-wrote the IBM EducaSon AdministraSon cerSficaSon track for Domino 8.5 William Malchisky Jr.
sbasegmez
shotashiratori
ogaclejapan
jinoketani
minorun365
oracle4engineer
ytaka23
mii3king
kniino
scalar
rayuron
mitchan
sat
keathley
reverentgeek
mraible
orderedlist
addyosmani
zakiyama
soudai
rasmusluckow
jakevdp
marktimemedia
csswizardry
brad_frost
