Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®

sbasegmez
September 02, 2013

ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®

Java and in particular OSGi are now very important parts of the Notes/Domino app dev model. In this session, you will learn what techniques can be utilized to process background jobs for XPages applications. Whether you want to replace your existing agents with Domino OSGi Tasklet Services (DOTS) or use Eclipse Jobs to run time-consuming routines without interrupting the use of your application - we will show you real life examples of why and how. You should also consider attending this session to hear about some suprises you don't want to miss...

sbasegmez

September 02, 2013
Tweet

More Decks by sbasegmez

Other Decks in Programming

Transcript

  1. UKLUG 2012 – Cardiff, Wales Background Tasks An XPager's Guide

    to Process Server-Side Jobs on IBM® Domino® Serdar Basegmez | Developi / OpenNTF Frank van der Linden | e-office / OpenNTF
  2. UKLUG 2012 – Cardiff, Wales Agenda  Introduction  DOTS

     Basics  Running, Developing and Deploying DOTS  Programmability Tips and Tricks  Asynchronous Processing  Developing Jobs  Alternative Approaches  Tips and Tricks  Q&A
  3. UKLUG 2012 – Cardiff, Wales Serdar Basegmez •IBM Collaboration Solutions

    Champion (2011-2013) •Owner of Developi Information Systems (Istanbul, Turkey) •Founder/Co-leader of LUGTR – Turkish Lotus User Group •Bilingual Blogger at LotusNotus.com (Turkish/English) •OpenNTF Guy in Turkey • Contributor in XSnippets and CollaborationToday.info • Member Director at OpenNTF Board •IBM Notes/Domino / SBT Design Partner •Expertise on XPages, IBM Domino, IBM Connections, etc.
  4. UKLUG 2012 – Cardiff, Wales Frank van der Linden •IBM

    Collaboration Solutions Champion (2013) •XPages developer at e-office (Netherlands) •Founder/Co-leader of XPages Meetup (Netherlands) •Blogger at domino-weblog.com •OpenNTF • Contributor in XSnippets and CollaborationToday.info • Member Director at OpenNTF Board •IBM Notes/Domino / SBT Design Partner •Expertise on XPages, IBM Domino, IBM Connections, etc.
  5. UKLUG 2012 – Cardiff, Wales What’s the problem? •XPages is

    great! • We all know it... •But... • XPages are based on interaction. • Users demand, server replies back. •What if… • Processing takes too long? • You need housekeeping? • You have scheduled reports? • You need to send reminders? • Your application needs batch processing? • You require unattended tasks?
  6. UKLUG 2012 – Cardiff, Wales Background Processing is Possible! Unattended

    Tasks • Domino OSGi Tasklet Services • Alternative to Domino Agents Asynchronous Processing • Threads and Jobs • Multi-threaded Applications
  7. UKLUG 2012 – Cardiff, Wales Meet DOTS – your new

    best friend •Domino OSGi Tasklet Services: Next Generation Agents for Domino •DOTS provides ability to develop and run OSGi level server Tasklets for Domino • Easy deployment. • Tooling with Eclipse IDE and its Java and plugins editors and Java debugger. • Leverage existing OSGi assets (within the company or open source). • Tasklets can be run: • Manually (local and remote calls) • Scheduled or on server start •OpenNTF project by IBM since April, 2011 (replaced JAVADDIN project) •Included (but not supported) in IBM Domino 9 Social Edition
  8. UKLUG 2012 – Cardiff, Wales What is a Tasklet? •Component

    to process a step in your business logic • Lightweight addin task. • DOTS Tasklets are simply Java methods within OSGi bundles. • Declared as OSGi extension point •Tasklet is functionally less capable than a Server Task Tasks DOTS Tasklets Loaded once, run continuously Loaded once, run when needed Manages own threads DOTS manages threading Should be native (or encapsulated) Runs inside DOTS container Unmanaged security Managed security Hard to develop Easy to develop
  9. UKLUG 2012 – Cardiff, Wales DOTS vs. Agents •DOTS Tasklets

    outperforms over Java Agents! AMGR launches a Thread JVM Initializes Prepare Java and JNI interfaces Load Agent Bytecode Run! Run! Java Agent: DOTS tasklet: Everything is ready to run > tell amgr run "test\XPagescrash.nsf" 'LongJobAgent' 09.11.2012 19:38:39 JVM: Java Virtual Machine initialized. 09.11.2012 19:38:39 AMgr: Start executing agent 'LongJobAgent' in 'test\XPagescrash.nsf' 09.11.2012 19:38:39 Agent Manager: Agent printing: 181349 09.11.2012 19:41:02 Agent Manager: Agent printing: 2227 09.11.2012 19:41:02 Agent Manager: Agent printing: Finished in 143 secs... - 09.11.2012 19:41:02 AMgr: Agent 'LongJobAgent' in 'test\XPagescrash.nsf' completed execution > load dots > Listening for transport dt_socket at address: 8001 09.11.2012 19:42:40 Domino OSGi Tasklet Container started ( profile DOTS ) > 181349 > 2227 09.11.2012 19:43:22 [DOTS] (annotated) Finished in 41 secs...
  10. UKLUG 2012 – Cardiff, Wales DOTS vs. Agents •Amgr is

    old and complicated • Security model problems • Limited functionality (like issues on class loading, reflections) •Range • Agents are database-dependent • DOTS runs server-wide •More flexible scheduling • Precise timings like 45 seconds • Programmatic control over scheduling •Multi-threading • DOTS tasklet might perform multi-threaded operations for more efficiency
  11. UKLUG 2012 – Cardiff, Wales Examples on DOTS •IBM SmartCloud™

    for Social Business (a.k.a. LotusLive) • Some part of back-end processing has been implemented using DOTS •CollaborationToday.info • Feed Aggregation is implemented using DOTS •Other possible uses... • Workflow Engine implementations • Social Media Monitoring • Database Monitoring • Application Architecture Assessment • Big Data • Search and Discovery
  12. UKLUG 2012 – Cardiff, Wales Runtime Environment •IBM Domino 9.0

    Social Edition provides DOTS Runtime • via Embedded Experience Add-on •For older versions, • DOTS should be installed into IBM Domino server • Refer to BP207 slides from IBM Connect 2013
  13. UKLUG 2012 – Cardiff, Wales Development Environment •DOTS Tasklets are

    simply Java methods • Designated classes as OSGi bundles. • Deployed as Eclipse plug-ins. •Eclipse IDE is needed for development and debugging. • DDE can be used with lots of unsupported configuration changes. • “Unsupported”  No need for an adventure  •Refer to BP207 slides from IBM Connect 2013
  14. UKLUG 2012 – Cardiff, Wales Deployment Methods •DOTS tasklets are

    packaged as a «Plug-in» •Plug-ins can be deployed into OSGi by... • Referencing PDE tool to the Eclipse Run Configuration • For testing & development... • Putting JAR file into OSGi plug-ins folder, • Beg your Admin for file-level access to the server! • Referencing an update site • Using DOTS Profiles, any update site can be used (NSF, HTTP…)
  15. UKLUG 2012 – Cardiff, Wales You Like Annotation? •DOTS also

    provides annotated declarations within Java classes • Remember, you still need to create an extension point in «plugin.xml» @RunOnStart public void runOnStart( IProgressMonitor monitor ){ logMessage("Annotated onStart method"); } @Run( id="manual") public void runManual( String[] args, IProgressMonitor monitor ){ logMessage("Annotated run method with id=manual"); } @RunOnStart @RunEvery( every=60, unit=RunUnit.second ) @HungPossibleAfter( timeInMinutes=1 ) public void runEvery60seconds( IProgressMonitor monitor ){ logMessage( "Called from annotated method every 60 seconds"); } <plugin> <extension point="com.ibm.dots.task"> <task class="com.ic13.samples.helloworld.AnnotatedTasklet" id="someid" /> </extension> </plugin>
  16. UKLUG 2012 – Cardiff, Wales A Powerful Feature: DOTS Profiles

    •A DOTS profile is a specific DOTS container. • Each profile defines «Tasklet Container» that has different settings for; • Remote Controller, Security Configuration, Specific plug-in container •DOTS support multiple profiles for different needs. • Tasklet used occasionally, for testing or with special security •Profiles can be stored in XML files or Notes Databases • It can be any Notes Database like names.nsf. •Every profile has a unique name (Message Queue Name – mqName) • Profiles accept separate tell commands, like «tell profileName run XYZ» • DOTS loaded with a default profile named as «DOTS»
  17. UKLUG 2012 – Cardiff, Wales Programmability Tips: Logging •Logging •

    Try not to use «System.out.println». (No output to log.nsf) • Instead, use «logMessage(...)» and «logException(...)» •ServerConsole helps. • If your code works outside the tasklet, you don’t have «logMessage(...)» • Use «com.ibm.dots.task.ServerConsole» class for console output. •You know “XPages OpenLog Logger” project? • It utilize OpenLog database for your Java code. • Find Paul S.Withers and buy a beer for him.
  18. UKLUG 2012 – Cardiff, Wales Programmability Tips: Parameters •Parameterization •

    Manual tasks can get parameters through Console or Plug-in manifest (plugin.xml) • «tell dots run XYZ param1 param2». • «args[n]», «getNthArgument(n)» or «getKeyedArgument(key)» can be used. • For other tasks, you have multiple options. • «notes.ini» settings • Argument Resolver (refer to DOTS samples) • Profile parameterization extension via DXL (refer to DOTS readme.pdf)
  19. UKLUG 2012 – Cardiff, Wales Remote Controller for DOTS •Remote

    Controller • DOTS might accept remote commands from other applications (e.g. XPages apps) • Remote commands are accepted from localhost only. • Listener should be configured from Profile document. Source: DOTS Readme File
  20. UKLUG 2012 – Cardiff, Wales Progress Monitoring for DOTS •Progress

    Monitoring • «IProgressMonitor monitor» argument passed on tasklet methods. • Very useful for tasklets running long time. • Progress monitor can be used to... • Inform DOTS task manager about how much tasklet has completed. • Aware of situation that might break execution. Vector<?> views = db.getViews(); monitor.beginTask( "List the views", views.size() );  Init with # of steps for ( Object view : views ){ if ( monitor.isCanceled() ){ break;  Break if canceled. } logMessage("\t" + ((View)view).getName() ); try { Thread.sleep( 1000 ); } catch (InterruptedException e) { e.printStackTrace(); } monitor.worked( 1 );  One step completed }
  21. UKLUG 2012 – Cardiff, Wales Threads and DOTS Source: Lotusphere

    2012 – SHOW112 session by David Taieb, Paul Fiore, Elizabeth Sawyer Three Two different threads
  22. UKLUG 2012 – Cardiff, Wales Threads and DOTS •Implications on

    running: • One thread for manual tasklets? • The first tasklet you call will run, • The second one will run at the sametime • One thread for scheduled tasklets? • Only one scheduled tasklet run at any time. • Conflicting tasklet will be deferred once. • What about RunOnStart Tasklets? • More than one RunOnStart tasklets will run at the same time!
  23. UKLUG 2012 – Cardiff, Wales Threads and DOTS •Best Practices

    for multiple long-running tasklets • Need some attention for synchronization issues • Use Progress Monitoring for cancellations • Too many scheduled tasklets  Unpredictable scheduling • You can use multiple profiles for different purposes • Each profiles has its own service threads! • Scheduled tasklets might fall into deadlocks! • Deadlocks are not resolved automatically. • while(true) is your enemy… • Some Java methods are black holes. • e.g. HttpURLConnection has infinite timeout by default.
  24. UKLUG 2012 – Cardiff, Wales Threads and DOTS •In terms

    of instances: • Threads run tasklets with reflection. • Scheduled and manual tasklets differ a bit: • Manual (and RunOnStart) Tasklets: • Thread creates an instance from the tasklet class • Calls the method and destroys the instance. • Each time you run a manual tasklet, it’s a different Java object! • Scheduled Tasklets: • Thread creates an instance for the first time, • Saves the instance for the next time. • Each time it runs a scheduled tasklet, it’s the same Java object!
  25. UKLUG 2012 – Cardiff, Wales Threads and DOTS •Instantiation may

    be a problem in some cases: Bundle Bundle.start  Initialize the Queue Bundle.stop  Save the Queue QueueManager (singleton object) readNextFeed() refreshQueue() readFeed(id) Tasklet Class @RunEvery( every=2, unit=RunUnit.minute ) Sched1()  qm.readNextFeed() @RunEvery( every=60, unit=RunUnit.minute ) Sched2()  qm.refreshQueue() @Run( id="refreshQueue" ) Manual1(args)  qm.refreshQueue() @Run( id="refreshFeed" ) Manual21(args)  qm.readFeed(args[0])
  26. UKLUG 2012 – Cardiff, Wales Problems of DOTS •DOTS runs

    in a different OSGi level than HTTP. • So there is almost no connection between DOTS and XPages • You can start DOTS tasks from XPages, using remote controller socket • Refer to Readme documentation for configuration and a basic example • Some possibilities • Communicate over Notes Document • Using REST API to send/receive information between DOTS and XPages •OSGi has different architecture for Class Loading • Some Open Source APIs might have issues (e.g. Rome project) • You need to have some level of OSGi knowledge in case •Out of Support Contract for now. • DOTS tasklet functionality is for internal use in Domino 9
  27. UKLUG 2012 – Cardiff, Wales Asynchronous Processing for XPages •The

    Problem • XSP Engine provides a single-threaded operation for users. •Implications • User has to wait for completion of long-running tasks • Worse, timeout will cause trouble… • Sometimes, it’s meaningless for user to wait • Throw into the basket and forget • Low performance • Varies for different cases •Solution • Launch a parallel thread for a non-blocking asynchronous task
  28. UKLUG 2012 – Cardiff, Wales Asynchronous Processing •Basic Problem •

    We have two needs to run a Java code for Domino • A Session object • A [Notes] context • These are provided for us in XPages, DOTS, Agents, etc. • … for a particular thread… • There is a planned life cycle for every threads •Fortunately, we can create a context • It’s like dancing on a thin ice… Photo: http://www.flickr.com/photos/ell-r-brown/
  29. UKLUG 2012 – Cardiff, Wales Formal Approach •ThreadSessionExecutor has been

    introduced in 8.5.3 • Simpler, cleaner • However, it #FAILS for Domino 9. • Expecting a fix for 9.0.1
  30. UKLUG 2012 – Cardiff, Wales An Eclipse Job •Running any

    code out of these boundaries will fail • Threads cannot work out of Notes Context… A static method to start the job
  31. UKLUG 2012 – Cardiff, Wales Other Approaches • Stephan H.

    Wissel posted a thread management bean for XPages • http://www.wissel.net/blog/d6plinks/SHWL-99U64Q •Provides • An abstract class • Managed Bean for Background Jobs • Very simple to use • No need to design separate Job classes for every task • Can be easily embedded into applications
  32. UKLUG 2012 – Cardiff, Wales Other Approaches • Nathan T.Freeman

    has ported more classes for Threading • Within OpenNTF Domino API project (OpenNTF API is optional) • https://github.com/OpenNTF/...BackgroundRunnable.java •Provides an abstract class •Better for long-running threads •Session for different users (including server) •For DOTS and XPages •Will be extended for Callable and Future interfaces
  33. UKLUG 2012 – Cardiff, Wales Multi-threading in DOTS •Similar idea

    can be applied for DOTS tasklets •Why? • Improved Performance • A job can be divided atomic parts (vertically or horizontally) and run asynchronously • Better Functionality • Developing your own scheduling system • Shifting Paradigm • Design a tasklet that acts as a service provider (e.g. FTP Server)
  34. UKLUG 2012 – Cardiff, Wales Multi-threading in DOTS •Method: •

    A “RunOnStart” tasklet, continously running. • A Thread / Job manager adapted • Session creation is easier, just like a Java Application • Use NotesThread and NotesFactory • Test, test, test… •For sophisticated examples: • “DOTS” Source code • “DOTS Extended” project source code
  35. UKLUG 2012 – Cardiff, Wales Important Points •Jobs and Threads

    will run on behalf of the authenticated user •Use Java responsibly… • TRY-CATCH is your best friend. • Recycle always… (except Session…) • Use Logging •If something doesn’t work right, • Restart HTTP task, Clean the project •Keep Job and Thread objects clean • Perform the real task in your own classes. • Pass the session object, throw and catch exception in case •Always perform harsh tests before production
  36. UKLUG 2012 – Cardiff, Wales Some Important Resources •IC13 -

    BP207 Session by Serdar Basegmez & Bruce Elgort • http://bit.ly/DOTS2013 •OpenNTF Project: Threads and Jobs • http://www.openntf.org/internal/home.nsf/project.xsp?action=openDocument&na me=Threads%20and%20Jobs •Java concurrency (multi-threading) – tutorial by Lars Vogel • http://www.vogella.com/articles/JavaConcurrency/article.html •Eclipse Jobs API • http://www.eclipse.org/articles/Article-Concurrency/jobs-api.html •Stephan H. Wissel - NotesSessions, XPages and Threads • http://www.wissel.net/blog/d6plinks/SHWL-99U64Q •CollaborationToday.info FeedMonster Module – on Github • https://github.com/OpenNTF/collaborationtoday/tree/master/DOTSFeedMonster
  37. UKLUG 2012 – Cardiff, Wales Thank you! Twitter: @flinden68 Blog:

    http://www.domino-weblog.nl Twitter: @sbasegmez Blog: http://lotusnotus.com