Engage 2022: The Superpower of Integrating External APIs for Notes and Domino Apps

Session De13 from Engage User Group Conference 2022 with Thilo Volprich

"In the jungle, no application walks alone. Sooner or later, applications will need to interact with other internal and external applications.
Luckily, Notes/Domino is a powerful platform to access data from other systems and provides wide range of possibilities to interact with different APIs.
This session focuses on how to architect various integration scenarios to implement secure and robust solutions for your Notes/Domino platform.
Whether you use LotusScript or Java, come to this session to see the best practices with real-world use cases."

(25/May/2022)

Transcript

1. #engageug DE13 The Superpower of Integrating External APIs 
 for Notes and Domino Apps Serdar Basegmez, Developi Thilo Volprich, Team Technology ENGAGE 2022
2. #engageug Who is speaking to you today? ▪ CEO of Team Technology GmbH ▪ IBM Champion 2018 - 2019 ▪ HCL Ambassador 2019-2021 ▪ Loving Domino ▪ Doing that stuff since 2003 Thilo Volprich ▪ Developer/Half-blooded Admin ▪ Developi UK ▪ OpenNTF Board ▪ Notes/Domino since 1999 ▪ IBM Champion Alumni (2011-2018) ▪ HCL Ambassador (2020-2022) Serdar Basegmez
3. #engageug Our session today ๏What is in our session ๏ A lot :-) ๏ Collective experience with customer projects ๏ Our (very personal) recommendations ๏ Some small code examples that can be easily used ๏ A list of helpers (software and libraries) ๏ Space for your questions and exchange of experience (all together) ๏What is not in our session ๏ Solutions to all your problems ๏ Only a selection of interfaces. The topic is far too big
4. #engageug What is an API An application programming interface (API) is an interface or communication protocol between a client and a server intended to simplify the building of client-side software. We focus on consuming APIs from other systems! (Source: en.Wikipedia.org). „ „
5. #engageug Security SECURITY
6. #engageug Authentication Planning ๏Questions to ask… ๏ Process ๏ Who knows the password, who sign in, who changes, how often? ๏ Method ๏ Username/Password, API key, OpenID/OAUTH, Obscurity? ๏ LAN? WAN? Internet? Public/Private Cloud? ๏ Potential eavesdroppers? Security precautions? ๏ What else? ๏ Any regulations to comply? ๏ Test-QA-Dev-Production staging?
7. #engageug Authentication Planning ๏API Authentication at a glance… ๏ No Secret ๏ Authentication not needed for some reason ๏ Shared Secret (Credentials) ๏ We know a “thing” (password, api key, etc.) ๏ Generated Secret (Tokens) ๏ We know a “thing” to acquire a token ๏ Delegated Access (Tokens) ๏ Someone knows a “thing” to provide us a token Where to keep these “thing”s?
8. #engageug Keep in mind! Forget about security if credentials are crappy! Source: https://www.flickr.com/photos/30478819@N08/29613520138
9. #engageug Best practise around security ๏Always run your code on the server ๏Clients and Users should not store/use the keys ๏ Beware of … ๏ LotusScript Debug ๏ Document Properties ๏ Malicious code to access data ๏Never trust your local network. Always use secure channels (Use SSL!)
10. #engageug Storing Credentials / Keys ๏Create a profile form, e.g. frm.fancyAPI ๏ Add a readers field ๏ Add yourself (Admin) and the Signer of the database ๏Create an agent, e.g. ag.fancyAPI ๏ Set security settings ๏ Run on behalf of [Database signer] (select the signer) ๏Run the code ๏ Run your Agent or script ๏ Call ag_fancyAPI ๏ Within ag_fancyAPI use ๏ db.GetProfileDocCollection(“frm.fancyAPI”).getfirstdocument
11. #engageug Best practise around security ๏Additionally, you can use encrypted field ๏ Every ID file has Public/Private keys ๏ Including servers! ๏ So, native support for encryption! ๏ Example: “WebSSOConfig” (Public Addressbook) ๏The lifecycle of credentials ๏ Plan ahead! ๏ Do they expire? How often? Do they need changing? ๏Please, RTFM!
12. #engageug Domino How to secure your Domino server Essentials
13. #engageug Easy but helpful settings Notes.ini ๏ Disable SSLv3 if you are still pre 9.0.1 FP9 ๏ Disable_SSLv3=1 ๏ Disable TLSv1 ๏ SSL_DISABLE_TLS_10=1 ๏ HTTPDisableServerHeader=1 ๏ HSTS (HTTP Strict Transport Security) ๏ Protects against protocol downgrade attacks ๏ Declares browsers should only interact using HTTPS ๏ It’s actually a HTTP Response Header set via notes.ini
14. #engageug HTTP Response Headers HTTP Response Headers: the most important once: ๏ Strict-Transport-Security ๏ X-Frame-Options ๏ X-Content-Type-Options ๏ Content-Security-Policy Set them for every hostname or IP your server is reachable from the internet.
15. #engageug Strict-Transport-Security Strict Transport Security ๏ Force clients only to connect via HTTPS ๏ Set completely in notes.ini ๏ Enabled by default since 9.0.1 FP3 IF2 ๏ but: to short: ๏ HTTP_HSTS_MAX_AGE= 17280000 ๏ and: does not include subdomains ๏ HTTP_HSTS_INCLUDE_SUBDOMAINS=1
16. #engageug X-Content-Type-Options X-Content-Type Options ๏ Reduces exposure to drive-by downloads and the risks of user uploaded content that, with clever naming, could be treated as a different content- type, like an executable. ๏ X-Content-Type-Options nosniff ๏ Domino 9: need to be set ๏ Domino 10: active by default ๏ can be disabled by notes.ini parameter starting with 10.0.1 FP4 ๏ HTTP_DISABLE_X_CONTENT_TYPE_OPTIONS_NOSNIFF=1. ๏ Domino 11: active by default ๏ same as above
17. #engageug Content-Security-Policy Content-Security-Policy ๏ More complex only needed if you provide input fields ๏ Defines approved sources of content that the browser may load ๏ Can be an effective countermeasure to Cross Site Scripting (XSS) attacks ๏ You can whitelist approved sources ๏ Details about the options: ๏ https://scotthelme.co.uk/content-security-policy-an-introduction/
18. #engageug Domino Administration Domino Administration basics ๏ Update to the latest version of domino to be more secure! ๏ Do the minimum on Domino ๏ Disable Port 80 or at least redirect it to 443 ๏ Hide your server type ๏ It’s 2022: use at least TLS 1.2 ๏ Protect your users/customers with HTTP-Headers ๏ More security requires a Reverse Proxy ๏ They are built for that See all details here. Source: https://en.rnug.ru/wp-content/uploads/2021/01/rnug-letsmakeyourdominowebserverrocksolid.pdf
19. #engageug Integration Methods
20. #engageug API Integration at a Glance ๏User-Initiated ๏ Notes Client / Web / Mobile ๏ User Interaction ๏ Implementation: ๏ Notes Client Actions ๏ Agents (LS / Java) ๏ XPages App (Java) ๏ OSGi Plugins (Java) ๏Unattended ๏ Scheduled / Event-triggered ๏ No user interaction / Background ๏ Implementation: ๏ Agents (LS / Java) ๏ Agent → XPages (Java) ๏ DOTS (Java) ๏ XOTS (Java)
21. #engageug Language Selection ๏Java ๏ XPages ๏ Managed Beans, SSJS ๏ XOTS ๏ Plugins ๏ HTTP, DOTS ๏ Agents ๏ Java Agents ๏LotusScript ๏ Notes Client ๏ Actions, Events, etc. ๏ Agents ๏ Client-Side or Server-Side ๏JavaScript (limited) ๏ Web Browser ๏ SSJS ๏ App. Dev. Pack (?)
22. #engageug Java Agents ๏Java agents? Really Why? ๏ We still need them! ๏ Unattended run (scheduled agents) ๏ RunOnServer ๏Try to avoid, if possible ๏ Compatibility problems ๏ Memory Leak with JAR files ๏ Tip: Put your JAR files into /jvm/lib/ext ๏ (Very) Low Performance
23. #engageug Scheduling XPages Code ๏Why schedule XPages code? ๏ Agents are far from ideal ๏ Reuse the same codebase ๏ Modern(ish) development environment ๏How? ๏ Write your code ๏ /mydb.nsf/myAgent.xsp ๏ Create a scheduled agent ๏ Call “https://myserver.com/mydb.nsf/myAgent.xsp”
24. #engageug DOTS ๏DOTS: Domino OSGi Tasklet Services ๏ Server Tasklets for Domino ๏ Tasklet: A lightweight server task ๏ Run background tasks in a lightweight scalable container ๏ Separate JVM and OSGi Container ๏ Various triggers ๏ Manual (console commands / socket trigger) ๏ Scheduled or on server start ๏ Triggered (Hooks through data events)
25. #engageug DOTS ๏Some History ๏ 2010 - OpenNTF project contributed by IBM ๏ 2013 - Added to IBM Domino 9 Social Edition ๏ OpenSocial add-on ๏ Out of support (Internal Use Only) ๏ 2018 - Removed in Domino 10 ๏ 2021 - Came back into Domino 12 ๏ Installed with the Core product ๏ Support: Windows 64 and Linux 64 ๏ Upgraded to Eclipse OSGi 4.6.2 (Neon 2)
26. #engageug Lotus Script / Agents ๏Lotus Script / Agents ๏ Can do a lot (domino is able to work with other systems) ๏ Read / Write files ๏ Read / Write to SQL databases ๏ Read / Write SOAP Requests ๏ Read / Write REST APIs ๏ Remote control of other programs ๏ Can be invoked by ๏ Time: Scheduled Agent ๏ User: Action Button ๏ User: Events (eg opening a document) ๏Conclusion ๏ Usually works well. Implemented using non modern technology.
27. #engageug Real World Examples REAL WORLD EXAMPLES
28. #engageug Real World Examples ๏Scheduling XPages Code ๏ Hybrid Application ๏ 80% XPages, 20% Notes ๏ Documents to be sent a remote service over REST API ๏ New content and critical changes => Upload immediately ๏ Other changes => Queue and Upload next hour ๏ Upload code complications ๏ Java SDK supplied, fails in Java agents ๏ Upload should be scheduled ๏ Single document upload triggered by XPages and/or Notes Client ๏All possible complications in a single project…
29. #engageug Real World Examples ๏Scheduling XPages Code ๏ Upload business logic ๏ Java classes embedded in managed beans ๏ Utilise SDK and Apache HttpComponents ๏ XPages code ๏ Multimode: Upload specific document or process queue ๏ Java Agents ๏ Notes client can trigger Java agent to upload single document ๏ Scheduled Java agent runs every hour to trigger upload ๏ Security ๏ XPages code should only be triggered by a Java Agent ๏ Precautions: IP limitations, authentication, temporary backend document, etc.
30. #engageug Real World Examples ๏Uploading Data via FTP ๏ Problem: ๏ Newsletters and Campaign Communication ๏ Forced migration to a cloud-based e-mail marketing service ๏ Upload text files to a Secure FTP server ๏ Transient Customer data needed daily and weekly ๏ Custom Campaign data to be uploaded instantly ๏ Queries and Uploads take too much time and resource ๏ Security is the top-priority! ๏ Solution: ๏ DOTS Tasklets
31. #engageug Real World Examples ๏Uploading Data via FTP ๏ Tasklet can handle long-running upload process ๏ Watching queue every minute ๏ Very small footprint for queue monitoring ๏ Reuse Java code already developed before User builds a target query for the campaign Predefined target lists for newsletters UploadJob Queue DOTS Tasklet (Scheduled / Manual) •Fetch next UploadJob •Run query •Convert to CSV file •Compress •Upload to SCP Server Upload Job #1 Upload Job #2 … Upload Job #N
32. #engageug Real world examples ๏Expense Workflow - Accounting App Integration ๏ Very common scenario ๏ Expense form ๏ Reads and caches Project codes, Customers, etc. ๏ Writes back approved expense amounts ๏ Various Solutions ๏ SQL calls to Stored Procedures ๏ LotusScript using a DLL for integration ๏ Web Services (SOAP, REST, etc) ๏ Text file exchange
33. #engageug Real world SQLInterface class
34. #engageug Real world SQLInterface class: Init
35. #engageug Real world SQLInterface class: run sql
36. #engageug Real world SQLInterface class: demo of usage
37. #engageug I am confused!?* What should I use? ๏Whenever you can, use an REST JSON API ๏ Instead of interacting with databases directly (LSX, ODBC) ๏ Instead of file transfers (Export / Import) ๏ And yes, instead of using HEI ๏ Use JAVA. Lotus Script implementation is far behind! ๏But keep in mind ๏ APIs can change very frequently (e.g. Microsoft M365) ๏ You don’t have control on details ๏ You don’t know the details (but most of the time, you are not interested in)
38. #engageug Common tasks Common tasks
39. #engageug The question of life, the universe and all the rest There are only two hard things in Computer Science: - Cache invalidation and - Naming things (Source: Phil Karlton (Former Netscape developper)) „ „
40. #engageug Caching: How to ๏Caching: Main challenges ๏ When to provide ๏ When you request the same data multiple times ๏ The data you receive do not change very often or you know exactly when ๏ Time required to obtain the data is high ๏ YES, all of them! ๏ When to clean (cache invalidation) ๏ As early as possible without loosing the advantages ๏ Time based ๏ Data change recognised
41. #engageug Caching: Javascript Example
42. #engageug Java Libraries ๏Utilise Libraries ๏ Practical Development ๏ OpenNTF Domino API ๏ Apache Commons ๏ Network Connections (REST APIs) ๏ Performance, compatibility, simplicity, protection from implementation changes ๏ HttpComponents (Apache Commons) ๏ Misc. ๏ POI (Java API for Microsoft Documents)
43. #engageug OSGi Plugins ๏Make use of OSGi Plugins ๏ Larger integration projects ๏ Multiple applications/business processes ๏ Broader functionality (UI modules, server-wide caching etc.) ๏ Easier implementations for various scenarios ๏ e.g. OAuth authentication ๏ Libraries ๏ XPages run in a secure container → Compatibility problems ๏ Jackson, Gson, some JDBC-wrappers, etc.
44. #engageug Logging ๏Use modern logging ๏ Log Levels ๏ Less is more, more is not enough… ๏ Consider stages ๏ Dev/Test and Production ๏ Tip: notes.ini ๏ Try OpenLog by Julian Robichaux ๏ Lotusscript: OpenLog ๏ XPages: XPages OpenLog Logger (Paul Withers) ๏ Java (XPages, DOTS, OSGi): XLogback (Serdar Basegmez)
45. #engageug JSON Processing using JAVA ๏What to process? How to process? ๏ Create/Consume JSON String ๏ Maps, Arrays, etc. ๏ IBM Commons, preloaded in Domino ๏ JSON ↔ Java Object conversions ๏ Direct Conversion between POJO and JSON ๏ Gson, Jackson, etc. ๏ JSON Streaming ๏ Read/Write large files/network streams ๏ Optimised CPU/Memory usage ๏ Gson, Jackson, etc.
46. #engageug JSON Processing ๏Java Options ๏ IBM Commons JSON Library ๏ Included in Domino ๏ Encapsulated Maps and Lists ๏ Gson (Google) ๏ Mid-weight, easy to use ๏ Object mapping, POJO conversion, streaming, etc. ๏ Jackson (FasterXML) ๏ More features (JAX-RS provider, language bindings, etc.) ๏ Tip: Gson and Jackson use Reflection ๏ Encapsulate into plugin ๏ Disable security (java.pol file)
47. #engageug JSON Processing using Javascript ๏How to process? ๏ Basic functionality of Javascript ๏ Very fast ๏ Very easy ๏ JSON ↔ JS Object conversions ๏ JSON.stringify ๏ JSON.parse
48. #engageug JSON Processing using Lotus Script ๏How to process? ๏ Two main classes to work with ๏ NotesJSONNavigator (reading) ๏ NotesJSONObject (writing) ๏ JSON ↔ JS Object conversions ๏ You have to run through the object ๏ No direct access ๏ Avoid to use it, if you work with large objects ๏ Buggy Has some difficulties ๏ Slow Needs time for processing and traversing ๏ Alternative (if you really have to, maybe) ๏ ls.snapps.JSONReader from OpenNTF
49. #engageug Java Tricks ๏JavaOptionsFile ๏ Adding JVM Options using notes.ini ๏ Setting TLS protocols (Important for Java Agents) ๏ Additional debugging, tweak third party libraries ๏ Client and Server ๏ Alternative Form: JavaUserOptionsFile ๏ Only custom options (after “-D”) ๏ DOTS uses a different JVM ๏ DOTS_JavaOptionsFile
50. #engageug Java Tricks ๏SSL and TLS issues ๏ Issues with HTTPS connections in Java agents ๏ Before Domino 9.0.1FP7, ๏ Use JavaOptionsFile to force TLS1.2 ๏ Technote KB0028812 ๏ SSL Handshake Errors ๏ Certification Authority information outdated ๏ Add trusted certificates to CACERTS ๏ Technote KB0035853 ๏ Much older versions ๏ Try your luck with Bouncy Castle
51. #engageug Litte helpers TOOLS / UTILITIES
52. #engageug Dev / Testing helper software / apps ๏Postman ๏PAW ๏SOAPUI ๏CURL ๏JSONLINT
53. #engageug Security Assessment / test websites Check your server ๏ SSL Labs (www.ssllabs.com) ๏ Grade change in January 2020 if you provide TLS 1.0/1.1
54. #engageug Security Assessment / test websites ๏ https://securityheaders.com/ ๏ Not every header is for every site ๏ Check your site functionality! HTTP Response Headers can break your site
55. Any questions?