Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Engage 2022: The Superpower of Integrating External APIs for Notes and Domino Apps

Engage 2022: The Superpower of Integrating External APIs for Notes and Domino Apps

Session De13 from Engage User Group Conference 2022 with Thilo Volprich

"In the jungle, no application walks alone. Sooner or later, applications will need to interact with other internal and external applications.
Luckily, Notes/Domino is a powerful platform to access data from other systems and provides wide range of possibilities to interact with different APIs.
This session focuses on how to architect various integration scenarios to implement secure and robust solutions for your Notes/Domino platform.
Whether you use LotusScript or Java, come to this session to see the best practices with real-world use cases.



May 25, 2022

More Decks by sbasegmez

Other Decks in Programming


  1. #engageug DE13 The Superpower of Integrating External APIs 

    Notes and Domino Apps Serdar Basegmez, Developi Thilo Volprich, Team Technology ENGAGE 2022
  2. #engageug Who is speaking to you today? ▪ CEO of

    Team Technology GmbH ▪ IBM Champion 2018 - 2019 ▪ HCL Ambassador 2019-2021 ▪ Loving Domino ▪ Doing that stuff since 2003 Thilo Volprich ▪ Developer/Half-blooded Admin ▪ Developi UK ▪ OpenNTF Board ▪ Notes/Domino since 1999 ▪ IBM Champion Alumni (2011-2018) ▪ HCL Ambassador (2020-2022) Serdar Basegmez
  3. #engageug Our session today ๏What is in our session ๏

    A lot :-) ๏ Collective experience with customer projects ๏ Our (very personal) recommendations ๏ Some small code examples that can be easily used ๏ A list of helpers (software and libraries) ๏ Space for your questions and exchange of experience (all together) ๏What is not in our session ๏ Solutions to all your problems ๏ Only a selection of interfaces. The topic is far too big
  4. #engageug What is an API An application programming interface (API)

    is an interface or communication protocol between a client and a server intended to simplify the building of client-side software. We focus on consuming APIs from other systems! (Source: en.Wikipedia.org). „ „
  5. #engageug Authentication Planning ๏Questions to ask… ๏ Process ๏ Who

    knows the password, who sign in, who changes, how often? ๏ Method ๏ Username/Password, API key, OpenID/OAUTH, Obscurity? ๏ LAN? WAN? Internet? Public/Private Cloud? ๏ Potential eavesdroppers? Security precautions? ๏ What else? ๏ Any regulations to comply? ๏ Test-QA-Dev-Production staging?
  6. #engageug Authentication Planning ๏API Authentication at a glance… ๏ No

    Secret ๏ Authentication not needed for some reason ๏ Shared Secret (Credentials) ๏ We know a “thing” (password, api key, etc.) ๏ Generated Secret (Tokens) ๏ We know a “thing” to acquire a token ๏ Delegated Access (Tokens) ๏ Someone knows a “thing” to provide us a token Where to keep these “thing”s?
  7. #engageug Keep in mind! Forget about security if credentials are

    crappy! Source: https://www.flickr.com/photos/30478819@N08/29613520138
  8. #engageug Best practise around security ๏Always run your code on

    the server ๏Clients and Users should not store/use the keys ๏ Beware of … ๏ LotusScript Debug ๏ Document Properties ๏ Malicious code to access data ๏Never trust your local network. Always use secure channels (Use SSL!)
  9. #engageug Storing Credentials / Keys ๏Create a profile form, e.g.

    frm.fancyAPI ๏ Add a readers field ๏ Add yourself (Admin) and the Signer of the database ๏Create an agent, e.g. ag.fancyAPI ๏ Set security settings ๏ Run on behalf of [Database signer] (select the signer) ๏Run the code ๏ Run your Agent or script ๏ Call ag_fancyAPI ๏ Within ag_fancyAPI use ๏ db.GetProfileDocCollection(“frm.fancyAPI”).getfirstdocument
  10. #engageug Best practise around security ๏Additionally, you can use encrypted

    field ๏ Every ID file has Public/Private keys ๏ Including servers! ๏ So, native support for encryption! ๏ Example: “WebSSOConfig” (Public Addressbook) ๏The lifecycle of credentials ๏ Plan ahead! ๏ Do they expire? How often? Do they need changing? ๏Please, RTFM!
  11. #engageug Easy but helpful settings Notes.ini ๏ Disable SSLv3 if

    you are still pre 9.0.1 FP9 ๏ Disable_SSLv3=1 ๏ Disable TLSv1 ๏ SSL_DISABLE_TLS_10=1 ๏ HTTPDisableServerHeader=1 ๏ HSTS (HTTP Strict Transport Security) ๏ Protects against protocol downgrade attacks ๏ Declares browsers should only interact using HTTPS ๏ It’s actually a HTTP Response Header set via notes.ini
  12. #engageug HTTP Response Headers HTTP Response Headers: the most important

    once: ๏ Strict-Transport-Security ๏ X-Frame-Options ๏ X-Content-Type-Options ๏ Content-Security-Policy Set them for every hostname or IP your server is reachable from the internet.
  13. #engageug Strict-Transport-Security Strict Transport Security ๏ Force clients only to

    connect via HTTPS ๏ Set completely in notes.ini ๏ Enabled by default since 9.0.1 FP3 IF2 ๏ but: to short: ๏ HTTP_HSTS_MAX_AGE= 17280000 ๏ and: does not include subdomains ๏ HTTP_HSTS_INCLUDE_SUBDOMAINS=1
  14. #engageug X-Content-Type-Options X-Content-Type Options ๏ Reduces exposure to drive-by downloads

    and the risks of user uploaded content that, with clever naming, could be treated as a different content- type, like an executable. ๏ X-Content-Type-Options nosniff ๏ Domino 9: need to be set ๏ Domino 10: active by default ๏ can be disabled by notes.ini parameter starting with 10.0.1 FP4 ๏ HTTP_DISABLE_X_CONTENT_TYPE_OPTIONS_NOSNIFF=1. ๏ Domino 11: active by default ๏ same as above
  15. #engageug Content-Security-Policy Content-Security-Policy ๏ More complex only needed if you

    provide input fields ๏ Defines approved sources of content that the browser may load ๏ Can be an effective countermeasure to Cross Site Scripting (XSS) attacks ๏ You can whitelist approved sources ๏ Details about the options: ๏ https://scotthelme.co.uk/content-security-policy-an-introduction/
  16. #engageug Domino Administration Domino Administration basics ๏ Update to the

    latest version of domino to be more secure! ๏ Do the minimum on Domino ๏ Disable Port 80 or at least redirect it to 443 ๏ Hide your server type ๏ It’s 2022: use at least TLS 1.2 ๏ Protect your users/customers with HTTP-Headers ๏ More security requires a Reverse Proxy ๏ They are built for that See all details here. Source: https://en.rnug.ru/wp-content/uploads/2021/01/rnug-letsmakeyourdominowebserverrocksolid.pdf
  17. #engageug API Integration at a Glance ๏User-Initiated ๏ Notes Client

    / Web / Mobile ๏ User Interaction ๏ Implementation: ๏ Notes Client Actions ๏ Agents (LS / Java) ๏ XPages App (Java) ๏ OSGi Plugins (Java) ๏Unattended ๏ Scheduled / Event-triggered ๏ No user interaction / Background ๏ Implementation: ๏ Agents (LS / Java) ๏ Agent → XPages (Java) ๏ DOTS (Java) ๏ XOTS (Java)
  18. #engageug Language Selection ๏Java ๏ XPages ๏ Managed Beans, SSJS

    ๏ XOTS ๏ Plugins ๏ HTTP, DOTS ๏ Agents ๏ Java Agents ๏LotusScript ๏ Notes Client ๏ Actions, Events, etc. ๏ Agents ๏ Client-Side or Server-Side ๏JavaScript (limited) ๏ Web Browser ๏ SSJS ๏ App. Dev. Pack (?)
  19. #engageug Java Agents ๏Java agents? Really Why? ๏ We still

    need them! ๏ Unattended run (scheduled agents) ๏ RunOnServer ๏Try to avoid, if possible ๏ Compatibility problems ๏ Memory Leak with JAR files ๏ Tip: Put your JAR files into /jvm/lib/ext ๏ (Very) Low Performance
  20. #engageug Scheduling XPages Code ๏Why schedule XPages code? ๏ Agents

    are far from ideal ๏ Reuse the same codebase ๏ Modern(ish) development environment ๏How? ๏ Write your code ๏ /mydb.nsf/myAgent.xsp ๏ Create a scheduled agent ๏ Call “https://myserver.com/mydb.nsf/myAgent.xsp”
  21. #engageug DOTS ๏DOTS: Domino OSGi Tasklet Services ๏ Server Tasklets

    for Domino ๏ Tasklet: A lightweight server task ๏ Run background tasks in a lightweight scalable container ๏ Separate JVM and OSGi Container ๏ Various triggers ๏ Manual (console commands / socket trigger) ๏ Scheduled or on server start ๏ Triggered (Hooks through data events)
  22. #engageug DOTS ๏Some History ๏ 2010 - OpenNTF project contributed

    by IBM ๏ 2013 - Added to IBM Domino 9 Social Edition ๏ OpenSocial add-on ๏ Out of support (Internal Use Only) ๏ 2018 - Removed in Domino 10 ๏ 2021 - Came back into Domino 12 ๏ Installed with the Core product ๏ Support: Windows 64 and Linux 64 ๏ Upgraded to Eclipse OSGi 4.6.2 (Neon 2)
  23. #engageug Lotus Script / Agents ๏Lotus Script / Agents ๏

    Can do a lot (domino is able to work with other systems) ๏ Read / Write files ๏ Read / Write to SQL databases ๏ Read / Write SOAP Requests ๏ Read / Write REST APIs ๏ Remote control of other programs ๏ Can be invoked by ๏ Time: Scheduled Agent ๏ User: Action Button ๏ User: Events (eg opening a document) ๏Conclusion ๏ Usually works well. Implemented using non modern technology.
  24. #engageug Real World Examples ๏Scheduling XPages Code ๏ Hybrid Application

    ๏ 80% XPages, 20% Notes ๏ Documents to be sent a remote service over REST API ๏ New content and critical changes => Upload immediately ๏ Other changes => Queue and Upload next hour ๏ Upload code complications ๏ Java SDK supplied, fails in Java agents ๏ Upload should be scheduled ๏ Single document upload triggered by XPages and/or Notes Client ๏All possible complications in a single project…
  25. #engageug Real World Examples ๏Scheduling XPages Code ๏ Upload business

    logic ๏ Java classes embedded in managed beans ๏ Utilise SDK and Apache HttpComponents ๏ XPages code ๏ Multimode: Upload specific document or process queue ๏ Java Agents ๏ Notes client can trigger Java agent to upload single document ๏ Scheduled Java agent runs every hour to trigger upload ๏ Security ๏ XPages code should only be triggered by a Java Agent ๏ Precautions: IP limitations, authentication, temporary backend document, etc.
  26. #engageug Real World Examples ๏Uploading Data via FTP ๏ Problem:

    ๏ Newsletters and Campaign Communication ๏ Forced migration to a cloud-based e-mail marketing service ๏ Upload text files to a Secure FTP server ๏ Transient Customer data needed daily and weekly ๏ Custom Campaign data to be uploaded instantly ๏ Queries and Uploads take too much time and resource ๏ Security is the top-priority! ๏ Solution: ๏ DOTS Tasklets
  27. #engageug Real World Examples ๏Uploading Data via FTP ๏ Tasklet

    can handle long-running upload process ๏ Watching queue every minute ๏ Very small footprint for queue monitoring ๏ Reuse Java code already developed before User builds a target query for the campaign Predefined target lists for newsletters UploadJob Queue DOTS Tasklet (Scheduled / Manual) •Fetch next UploadJob •Run query •Convert to CSV file •Compress •Upload to SCP Server Upload Job #1 Upload Job #2 … Upload Job #N
  28. #engageug Real world examples ๏Expense Workflow - Accounting App Integration

    ๏ Very common scenario ๏ Expense form ๏ Reads and caches Project codes, Customers, etc. ๏ Writes back approved expense amounts ๏ Various Solutions ๏ SQL calls to Stored Procedures ๏ LotusScript using a DLL for integration ๏ Web Services (SOAP, REST, etc) ๏ Text file exchange
  29. #engageug I am confused!?* What should I use? ๏Whenever you

    can, use an REST JSON API ๏ Instead of interacting with databases directly (LSX, ODBC) ๏ Instead of file transfers (Export / Import) ๏ And yes, instead of using HEI ๏ Use JAVA. Lotus Script implementation is far behind! ๏But keep in mind ๏ APIs can change very frequently (e.g. Microsoft M365) ๏ You don’t have control on details ๏ You don’t know the details (but most of the time, you are not interested in)
  30. #engageug The question of life, the universe and all the

    rest There are only two hard things in Computer Science: - Cache invalidation and - Naming things (Source: Phil Karlton (Former Netscape developper)) „ „
  31. #engageug Caching: How to ๏Caching: Main challenges ๏ When to

    provide ๏ When you request the same data multiple times ๏ The data you receive do not change very often or you know exactly when ๏ Time required to obtain the data is high ๏ YES, all of them! ๏ When to clean (cache invalidation) ๏ As early as possible without loosing the advantages ๏ Time based ๏ Data change recognised
  32. #engageug Java Libraries ๏Utilise Libraries ๏ Practical Development ๏ OpenNTF

    Domino API ๏ Apache Commons ๏ Network Connections (REST APIs) ๏ Performance, compatibility, simplicity, protection from implementation changes ๏ HttpComponents (Apache Commons) ๏ Misc. ๏ POI (Java API for Microsoft Documents)
  33. #engageug OSGi Plugins ๏Make use of OSGi Plugins ๏ Larger

    integration projects ๏ Multiple applications/business processes ๏ Broader functionality (UI modules, server-wide caching etc.) ๏ Easier implementations for various scenarios ๏ e.g. OAuth authentication ๏ Libraries ๏ XPages run in a secure container → Compatibility problems ๏ Jackson, Gson, some JDBC-wrappers, etc.
  34. #engageug Logging ๏Use modern logging ๏ Log Levels ๏ Less

    is more, more is not enough… ๏ Consider stages ๏ Dev/Test and Production ๏ Tip: notes.ini ๏ Try OpenLog by Julian Robichaux ๏ Lotusscript: OpenLog ๏ XPages: XPages OpenLog Logger (Paul Withers) ๏ Java (XPages, DOTS, OSGi): XLogback (Serdar Basegmez)
  35. #engageug JSON Processing using JAVA ๏What to process? How to

    process? ๏ Create/Consume JSON String ๏ Maps, Arrays, etc. ๏ IBM Commons, preloaded in Domino ๏ JSON ↔ Java Object conversions ๏ Direct Conversion between POJO and JSON ๏ Gson, Jackson, etc. ๏ JSON Streaming ๏ Read/Write large files/network streams ๏ Optimised CPU/Memory usage ๏ Gson, Jackson, etc.
  36. #engageug JSON Processing ๏Java Options ๏ IBM Commons JSON Library

    ๏ Included in Domino ๏ Encapsulated Maps and Lists ๏ Gson (Google) ๏ Mid-weight, easy to use ๏ Object mapping, POJO conversion, streaming, etc. ๏ Jackson (FasterXML) ๏ More features (JAX-RS provider, language bindings, etc.) ๏ Tip: Gson and Jackson use Reflection ๏ Encapsulate into plugin ๏ Disable security (java.pol file)
  37. #engageug JSON Processing using Javascript ๏How to process? ๏ Basic

    functionality of Javascript ๏ Very fast ๏ Very easy ๏ JSON ↔ JS Object conversions ๏ JSON.stringify ๏ JSON.parse
  38. #engageug JSON Processing using Lotus Script ๏How to process? ๏

    Two main classes to work with ๏ NotesJSONNavigator (reading) ๏ NotesJSONObject (writing) ๏ JSON ↔ JS Object conversions ๏ You have to run through the object ๏ No direct access ๏ Avoid to use it, if you work with large objects ๏ Buggy Has some difficulties ๏ Slow Needs time for processing and traversing ๏ Alternative (if you really have to, maybe) ๏ ls.snapps.JSONReader from OpenNTF
  39. #engageug Java Tricks ๏JavaOptionsFile ๏ Adding JVM Options using notes.ini

    ๏ Setting TLS protocols (Important for Java Agents) ๏ Additional debugging, tweak third party libraries ๏ Client and Server ๏ Alternative Form: JavaUserOptionsFile ๏ Only custom options (after “-D”) ๏ DOTS uses a different JVM ๏ DOTS_JavaOptionsFile
  40. #engageug Java Tricks ๏SSL and TLS issues ๏ Issues with

    HTTPS connections in Java agents ๏ Before Domino 9.0.1FP7, ๏ Use JavaOptionsFile to force TLS1.2 ๏ Technote KB0028812 ๏ SSL Handshake Errors ๏ Certification Authority information outdated ๏ Add trusted certificates to CACERTS ๏ Technote KB0035853 ๏ Much older versions ๏ Try your luck with Bouncy Castle
  41. #engageug Security Assessment / test websites Check your server ๏

    SSL Labs (www.ssllabs.com) ๏ Grade change in January 2020 if you provide TLS 1.0/1.1
  42. #engageug Security Assessment / test websites ๏ https://securityheaders.com/ ๏ Not

    every header is for every site ๏ Check your site functionality! HTTP Response Headers can break your site