ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®

Java and in particular OSGi are now very important parts of the Notes/Domino app dev model. In this session, you will learn what techniques can be utilized to process background jobs for XPages applications. Whether you want to replace your existing agents with Domino OSGi Tasklet Services (DOTS) or use Eclipse Jobs to run time-consuming routines without interrupting the use of your application - we will show you real life examples of why and how. You should also consider attending this session to hear about some suprises you don't want to miss...

Transcript

1. UKLUG 2012 – Cardiff, Wales Background Tasks An XPager's Guide to Process Server-Side Jobs on IBM® Domino® Serdar Basegmez | Developi / OpenNTF Frank van der Linden | e-office / OpenNTF
2. UKLUG 2012 – Cardiff, Wales Agenda  Introduction  DOTS  Basics  Running, Developing and Deploying DOTS  Programmability Tips and Tricks  Asynchronous Processing  Developing Jobs  Alternative Approaches  Tips and Tricks  Q&A
3. UKLUG 2012 – Cardiff, Wales Serdar Basegmez •IBM Collaboration Solutions Champion (2011-2013) •Owner of Developi Information Systems (Istanbul, Turkey) •Founder/Co-leader of LUGTR – Turkish Lotus User Group •Bilingual Blogger at LotusNotus.com (Turkish/English) •OpenNTF Guy in Turkey • Contributor in XSnippets and CollaborationToday.info • Member Director at OpenNTF Board •IBM Notes/Domino / SBT Design Partner •Expertise on XPages, IBM Domino, IBM Connections, etc.
4. UKLUG 2012 – Cardiff, Wales Frank van der Linden •IBM Collaboration Solutions Champion (2013) •XPages developer at e-office (Netherlands) •Founder/Co-leader of XPages Meetup (Netherlands) •Blogger at domino-weblog.com •OpenNTF • Contributor in XSnippets and CollaborationToday.info • Member Director at OpenNTF Board •IBM Notes/Domino / SBT Design Partner •Expertise on XPages, IBM Domino, IBM Connections, etc.
5. UKLUG 2012 – Cardiff, Wales Introduction
6. UKLUG 2012 – Cardiff, Wales What’s the problem? •XPages is great! • We all know it... •But... • XPages are based on interaction. • Users demand, server replies back. •What if… • Processing takes too long? • You need housekeeping? • You have scheduled reports? • You need to send reminders? • Your application needs batch processing? • You require unattended tasks?
7. UKLUG 2012 – Cardiff, Wales Background Processing is Possible! Unattended Tasks • Domino OSGi Tasklet Services • Alternative to Domino Agents Asynchronous Processing • Threads and Jobs • Multi-threaded Applications
8. UKLUG 2012 – Cardiff, Wales Domino OSGi Tasklet Services
9. UKLUG 2012 – Cardiff, Wales Meet DOTS – your new best friend •Domino OSGi Tasklet Services: Next Generation Agents for Domino •DOTS provides ability to develop and run OSGi level server Tasklets for Domino • Easy deployment. • Tooling with Eclipse IDE and its Java and plugins editors and Java debugger. • Leverage existing OSGi assets (within the company or open source). • Tasklets can be run: • Manually (local and remote calls) • Scheduled or on server start •OpenNTF project by IBM since April, 2011 (replaced JAVADDIN project) •Included (but not supported) in IBM Domino 9 Social Edition
10. UKLUG 2012 – Cardiff, Wales What is a Tasklet? •Component to process a step in your business logic • Lightweight addin task. • DOTS Tasklets are simply Java methods within OSGi bundles. • Declared as OSGi extension point •Tasklet is functionally less capable than a Server Task Tasks DOTS Tasklets Loaded once, run continuously Loaded once, run when needed Manages own threads DOTS manages threading Should be native (or encapsulated) Runs inside DOTS container Unmanaged security Managed security Hard to develop Easy to develop
11. UKLUG 2012 – Cardiff, Wales DOTS vs. Agents •DOTS Tasklets outperforms over Java Agents! AMGR launches a Thread JVM Initializes Prepare Java and JNI interfaces Load Agent Bytecode Run! Run! Java Agent: DOTS tasklet: Everything is ready to run > tell amgr run "test\XPagescrash.nsf" 'LongJobAgent' 09.11.2012 19:38:39 JVM: Java Virtual Machine initialized. 09.11.2012 19:38:39 AMgr: Start executing agent 'LongJobAgent' in 'test\XPagescrash.nsf' 09.11.2012 19:38:39 Agent Manager: Agent printing: 181349 09.11.2012 19:41:02 Agent Manager: Agent printing: 2227 09.11.2012 19:41:02 Agent Manager: Agent printing: Finished in 143 secs... - 09.11.2012 19:41:02 AMgr: Agent 'LongJobAgent' in 'test\XPagescrash.nsf' completed execution > load dots > Listening for transport dt_socket at address: 8001 09.11.2012 19:42:40 Domino OSGi Tasklet Container started ( profile DOTS ) > 181349 > 2227 09.11.2012 19:43:22 [DOTS] (annotated) Finished in 41 secs...
12. UKLUG 2012 – Cardiff, Wales DOTS vs. Agents •Amgr is old and complicated • Security model problems • Limited functionality (like issues on class loading, reflections) •Range • Agents are database-dependent • DOTS runs server-wide •More flexible scheduling • Precise timings like 45 seconds • Programmatic control over scheduling •Multi-threading • DOTS tasklet might perform multi-threaded operations for more efficiency
13. UKLUG 2012 – Cardiff, Wales Examples on DOTS •IBM SmartCloud™ for Social Business (a.k.a. LotusLive) • Some part of back-end processing has been implemented using DOTS •CollaborationToday.info • Feed Aggregation is implemented using DOTS •Other possible uses... • Workflow Engine implementations • Social Media Monitoring • Database Monitoring • Application Architecture Assessment • Big Data • Search and Discovery
14. UKLUG 2012 – Cardiff, Wales Runtime Environment •IBM Domino 9.0 Social Edition provides DOTS Runtime • via Embedded Experience Add-on •For older versions, • DOTS should be installed into IBM Domino server • Refer to BP207 slides from IBM Connect 2013
15. UKLUG 2012 – Cardiff, Wales Development Environment •DOTS Tasklets are simply Java methods • Designated classes as OSGi bundles. • Deployed as Eclipse plug-ins. •Eclipse IDE is needed for development and debugging. • DDE can be used with lots of unsupported configuration changes. • “Unsupported”  No need for an adventure  •Refer to BP207 slides from IBM Connect 2013
16. UKLUG 2012 – Cardiff, Wales Deployment Methods •DOTS tasklets are packaged as a «Plug-in» •Plug-ins can be deployed into OSGi by... • Referencing PDE tool to the Eclipse Run Configuration • For testing & development... • Putting JAR file into OSGi plug-ins folder, • Beg your Admin for file-level access to the server! • Referencing an update site • Using DOTS Profiles, any update site can be used (NSF, HTTP…)
17. UKLUG 2012 – Cardiff, Wales Demo Photo: http://www.flickr.com/photos/mjtmail/
18. UKLUG 2012 – Cardiff, Wales Deep Dive into DOTS
19. UKLUG 2012 – Cardiff, Wales You Like Annotation? •DOTS also provides annotated declarations within Java classes • Remember, you still need to create an extension point in «plugin.xml» @RunOnStart public void runOnStart( IProgressMonitor monitor ){ logMessage("Annotated onStart method"); } @Run( id="manual") public void runManual( String[] args, IProgressMonitor monitor ){ logMessage("Annotated run method with id=manual"); } @RunOnStart @RunEvery( every=60, unit=RunUnit.second ) @HungPossibleAfter( timeInMinutes=1 ) public void runEvery60seconds( IProgressMonitor monitor ){ logMessage( "Called from annotated method every 60 seconds"); } <plugin> <extension point="com.ibm.dots.task"> <task class="com.ic13.samples.helloworld.AnnotatedTasklet" id="someid" /> </extension> </plugin>
20. UKLUG 2012 – Cardiff, Wales A Powerful Feature: DOTS Profiles •A DOTS profile is a specific DOTS container. • Each profile defines «Tasklet Container» that has different settings for; • Remote Controller, Security Configuration, Specific plug-in container •DOTS support multiple profiles for different needs. • Tasklet used occasionally, for testing or with special security •Profiles can be stored in XML files or Notes Databases • It can be any Notes Database like names.nsf. •Every profile has a unique name (Message Queue Name – mqName) • Profiles accept separate tell commands, like «tell profileName run XYZ» • DOTS loaded with a default profile named as «DOTS»
21. UKLUG 2012 – Cardiff, Wales Programmability Tips: Logging •Logging • Try not to use «System.out.println». (No output to log.nsf) • Instead, use «logMessage(...)» and «logException(...)» •ServerConsole helps. • If your code works outside the tasklet, you don’t have «logMessage(...)» • Use «com.ibm.dots.task.ServerConsole» class for console output. •You know “XPages OpenLog Logger” project? • It utilize OpenLog database for your Java code. • Find Paul S.Withers and buy a beer for him.
22. UKLUG 2012 – Cardiff, Wales Programmability Tips: Parameters •Parameterization • Manual tasks can get parameters through Console or Plug-in manifest (plugin.xml) • «tell dots run XYZ param1 param2». • «args[n]», «getNthArgument(n)» or «getKeyedArgument(key)» can be used. • For other tasks, you have multiple options. • «notes.ini» settings • Argument Resolver (refer to DOTS samples) • Profile parameterization extension via DXL (refer to DOTS readme.pdf)
23. UKLUG 2012 – Cardiff, Wales Programmability Tips: Parameters •Profile Configuration Document:
24. UKLUG 2012 – Cardiff, Wales Remote Controller for DOTS •Remote Controller • DOTS might accept remote commands from other applications (e.g. XPages apps) • Remote commands are accepted from localhost only. • Listener should be configured from Profile document. Source: DOTS Readme File
25. UKLUG 2012 – Cardiff, Wales Progress Monitoring for DOTS •Progress Monitoring • «IProgressMonitor monitor» argument passed on tasklet methods. • Very useful for tasklets running long time. • Progress monitor can be used to... • Inform DOTS task manager about how much tasklet has completed. • Aware of situation that might break execution. Vector<?> views = db.getViews(); monitor.beginTask( "List the views", views.size() );  Init with # of steps for ( Object view : views ){ if ( monitor.isCanceled() ){ break;  Break if canceled. } logMessage("\t" + ((View)view).getName() ); try { Thread.sleep( 1000 ); } catch (InterruptedException e) { e.printStackTrace(); } monitor.worked( 1 );  One step completed }
26. UKLUG 2012 – Cardiff, Wales Threads and DOTS Source: Lotusphere 2012 – SHOW112 session by David Taieb, Paul Fiore, Elizabeth Sawyer Three Two different threads
27. UKLUG 2012 – Cardiff, Wales Threads and DOTS •Implications on running: • One thread for manual tasklets? • The first tasklet you call will run, • The second one will run at the sametime • One thread for scheduled tasklets? • Only one scheduled tasklet run at any time. • Conflicting tasklet will be deferred once. • What about RunOnStart Tasklets? • More than one RunOnStart tasklets will run at the same time!
28. UKLUG 2012 – Cardiff, Wales Threads and DOTS •Best Practices for multiple long-running tasklets • Need some attention for synchronization issues • Use Progress Monitoring for cancellations • Too many scheduled tasklets  Unpredictable scheduling • You can use multiple profiles for different purposes • Each profiles has its own service threads! • Scheduled tasklets might fall into deadlocks! • Deadlocks are not resolved automatically. • while(true) is your enemy… • Some Java methods are black holes. • e.g. HttpURLConnection has infinite timeout by default.
29. UKLUG 2012 – Cardiff, Wales Threads and DOTS •In terms of instances: • Threads run tasklets with reflection. • Scheduled and manual tasklets differ a bit: • Manual (and RunOnStart) Tasklets: • Thread creates an instance from the tasklet class • Calls the method and destroys the instance. • Each time you run a manual tasklet, it’s a different Java object! • Scheduled Tasklets: • Thread creates an instance for the first time, • Saves the instance for the next time. • Each time it runs a scheduled tasklet, it’s the same Java object!
30. UKLUG 2012 – Cardiff, Wales Threads and DOTS •Instantiation may be a problem in some cases: Bundle Bundle.start  Initialize the Queue Bundle.stop  Save the Queue QueueManager (singleton object) readNextFeed() refreshQueue() readFeed(id) Tasklet Class @RunEvery( every=2, unit=RunUnit.minute ) Sched1()  qm.readNextFeed() @RunEvery( every=60, unit=RunUnit.minute ) Sched2()  qm.refreshQueue() @Run( id="refreshQueue" ) Manual1(args)  qm.refreshQueue() @Run( id="refreshFeed" ) Manual21(args)  qm.readFeed(args[0])
31. UKLUG 2012 – Cardiff, Wales Problems of DOTS •DOTS runs in a different OSGi level than HTTP. • So there is almost no connection between DOTS and XPages • You can start DOTS tasks from XPages, using remote controller socket • Refer to Readme documentation for configuration and a basic example • Some possibilities • Communicate over Notes Document • Using REST API to send/receive information between DOTS and XPages •OSGi has different architecture for Class Loading • Some Open Source APIs might have issues (e.g. Rome project) • You need to have some level of OSGi knowledge in case •Out of Support Contract for now. • DOTS tasklet functionality is for internal use in Domino 9
32. UKLUG 2012 – Cardiff, Wales Asynchronous Processing
33. UKLUG 2012 – Cardiff, Wales Asynchronous Processing for XPages •The Problem • XSP Engine provides a single-threaded operation for users. •Implications • User has to wait for completion of long-running tasks • Worse, timeout will cause trouble… • Sometimes, it’s meaningless for user to wait • Throw into the basket and forget • Low performance • Varies for different cases •Solution • Launch a parallel thread for a non-blocking asynchronous task
34. UKLUG 2012 – Cardiff, Wales Asynchronous Processing •Basic Problem • We have two needs to run a Java code for Domino • A Session object • A [Notes] context • These are provided for us in XPages, DOTS, Agents, etc. • … for a particular thread… • There is a planned life cycle for every threads •Fortunately, we can create a context • It’s like dancing on a thin ice… Photo: http://www.flickr.com/photos/ell-r-brown/
35. UKLUG 2012 – Cardiff, Wales Demo Photo: http://www.flickr.com/photos/mjtmail/
36. UKLUG 2012 – Cardiff, Wales Formal Approach •ThreadSessionExecutor has been introduced in 8.5.3 • Simpler, cleaner • However, it #FAILS for Domino 9. • Expecting a fix for 9.0.1
37. UKLUG 2012 – Cardiff, Wales An Eclipse Job Job class Modification #1
38. UKLUG 2012 – Cardiff, Wales An Eclipse Job Run Method
39. UKLUG 2012 – Cardiff, Wales An Eclipse Job •Running any code out of these boundaries will fail • Threads cannot work out of Notes Context… A static method to start the job
40. UKLUG 2012 – Cardiff, Wales Other Approaches • Stephan H. Wissel posted a thread management bean for XPages • http://www.wissel.net/blog/d6plinks/SHWL-99U64Q •Provides • An abstract class • Managed Bean for Background Jobs • Very simple to use • No need to design separate Job classes for every task • Can be easily embedded into applications
41. UKLUG 2012 – Cardiff, Wales Other Approaches • Nathan T.Freeman has ported more classes for Threading • Within OpenNTF Domino API project (OpenNTF API is optional) • https://github.com/OpenNTF/...BackgroundRunnable.java •Provides an abstract class •Better for long-running threads •Session for different users (including server) •For DOTS and XPages •Will be extended for Callable and Future interfaces
42. UKLUG 2012 – Cardiff, Wales Multi-threading in DOTS •Similar idea can be applied for DOTS tasklets •Why? • Improved Performance • A job can be divided atomic parts (vertically or horizontally) and run asynchronously • Better Functionality • Developing your own scheduling system • Shifting Paradigm • Design a tasklet that acts as a service provider (e.g. FTP Server)
43. UKLUG 2012 – Cardiff, Wales Multi-threading in DOTS •Method: • A “RunOnStart” tasklet, continously running. • A Thread / Job manager adapted • Session creation is easier, just like a Java Application • Use NotesThread and NotesFactory • Test, test, test… •For sophisticated examples: • “DOTS” Source code • “DOTS Extended” project source code
44. UKLUG 2012 – Cardiff, Wales Important Points •Jobs and Threads will run on behalf of the authenticated user •Use Java responsibly… • TRY-CATCH is your best friend. • Recycle always… (except Session…) • Use Logging •If something doesn’t work right, • Restart HTTP task, Clean the project •Keep Job and Thread objects clean • Perform the real task in your own classes. • Pass the session object, throw and catch exception in case •Always perform harsh tests before production
45. UKLUG 2012 – Cardiff, Wales
46. UKLUG 2012 – Cardiff, Wales Some Important Resources •IC13 - BP207 Session by Serdar Basegmez & Bruce Elgort • http://bit.ly/DOTS2013 •OpenNTF Project: Threads and Jobs • http://www.openntf.org/internal/home.nsf/project.xsp?action=openDocument&na me=Threads%20and%20Jobs •Java concurrency (multi-threading) – tutorial by Lars Vogel • http://www.vogella.com/articles/JavaConcurrency/article.html •Eclipse Jobs API • http://www.eclipse.org/articles/Article-Concurrency/jobs-api.html •Stephan H. Wissel - NotesSessions, XPages and Threads • http://www.wissel.net/blog/d6plinks/SHWL-99U64Q •CollaborationToday.info FeedMonster Module – on Github • https://github.com/OpenNTF/collaborationtoday/tree/master/DOTSFeedMonster
47. UKLUG 2012 – Cardiff, Wales Q&A
48. UKLUG 2012 – Cardiff, Wales Thank you! Twitter: @flinden68 Blog: http://www.domino-weblog.nl Twitter: @sbasegmez Blog: http://lotusnotus.com