Tweet
Tweet

Slide 1

Slide 1 text

#engageug DE13 The Superpower of Integrating External APIs 
 for Notes and Domino Apps Serdar Basegmez, Developi Thilo Volprich, Team Technology ENGAGE 2022

Slide 2

Slide 2 text

#engageug Who is speaking to you today? ▪ CEO of Team Technology GmbH ▪ IBM Champion 2018 - 2019 ▪ HCL Ambassador 2019-2021 ▪ Loving Domino ▪ Doing that stuff since 2003 Thilo Volprich ▪ Developer/Half-blooded Admin ▪ Developi UK ▪ OpenNTF Board ▪ Notes/Domino since 1999 ▪ IBM Champion Alumni (2011-2018) ▪ HCL Ambassador (2020-2022) Serdar Basegmez

Slide 3

Slide 3 text

#engageug Our session today ๏What is in our session ๏ A lot :-) ๏ Collective experience with customer projects ๏ Our (very personal) recommendations ๏ Some small code examples that can be easily used ๏ A list of helpers (software and libraries) ๏ Space for your questions and exchange of experience (all together) ๏What is not in our session ๏ Solutions to all your problems ๏ Only a selection of interfaces. The topic is far too big

Slide 4

Slide 4 text

#engageug What is an API An application programming interface (API) is an interface or communication protocol between a client and a server intended to simplify the building of client-side software. We focus on consuming APIs from other systems! (Source: en.Wikipedia.org). „ „

Slide 5

Slide 5 text

#engageug Security SECURITY

Slide 6

Slide 6 text

#engageug Authentication Planning ๏Questions to ask… ๏ Process ๏ Who knows the password, who sign in, who changes, how often? ๏ Method ๏ Username/Password, API key, OpenID/OAUTH, Obscurity? ๏ LAN? WAN? Internet? Public/Private Cloud? ๏ Potential eavesdroppers? Security precautions? ๏ What else? ๏ Any regulations to comply? ๏ Test-QA-Dev-Production staging?

Slide 7

Slide 7 text

#engageug Authentication Planning ๏API Authentication at a glance… ๏ No Secret ๏ Authentication not needed for some reason ๏ Shared Secret (Credentials) ๏ We know a “thing” (password, api key, etc.) ๏ Generated Secret (Tokens) ๏ We know a “thing” to acquire a token ๏ Delegated Access (Tokens) ๏ Someone knows a “thing” to provide us a token Where to keep these “thing”s?

Slide 8

Slide 8 text

#engageug Keep in mind! Forget about security if credentials are crappy! Source: https://www.flickr.com/photos/30478819@N08/29613520138

Slide 9

Slide 9 text

#engageug Best practise around security ๏Always run your code on the server ๏Clients and Users should not store/use the keys ๏ Beware of … ๏ LotusScript Debug ๏ Document Properties ๏ Malicious code to access data ๏Never trust your local network. Always use secure channels (Use SSL!)

Slide 10

Slide 10 text

#engageug Storing Credentials / Keys ๏Create a profile form, e.g. frm.fancyAPI ๏ Add a readers field ๏ Add yourself (Admin) and the Signer of the database ๏Create an agent, e.g. ag.fancyAPI ๏ Set security settings ๏ Run on behalf of [Database signer] (select the signer) ๏Run the code ๏ Run your Agent or script ๏ Call ag_fancyAPI ๏ Within ag_fancyAPI use ๏ db.GetProfileDocCollection(“frm.fancyAPI”).getfirstdocument

Slide 11

Slide 11 text

#engageug Best practise around security ๏Additionally, you can use encrypted field ๏ Every ID file has Public/Private keys ๏ Including servers! ๏ So, native support for encryption! ๏ Example: “WebSSOConfig” (Public Addressbook) ๏The lifecycle of credentials ๏ Plan ahead! ๏ Do they expire? How often? Do they need changing? ๏Please, RTFM!

Slide 12

Slide 12 text

#engageug Domino How to secure your Domino server Essentials

Slide 13

Slide 13 text

#engageug Easy but helpful settings Notes.ini ๏ Disable SSLv3 if you are still pre 9.0.1 FP9 ๏ Disable_SSLv3=1 ๏ Disable TLSv1 ๏ SSL_DISABLE_TLS_10=1 ๏ HTTPDisableServerHeader=1 ๏ HSTS (HTTP Strict Transport Security) ๏ Protects against protocol downgrade attacks ๏ Declares browsers should only interact using HTTPS ๏ It’s actually a HTTP Response Header set via notes.ini

Slide 14

Slide 14 text

#engageug HTTP Response Headers HTTP Response Headers: the most important once: ๏ Strict-Transport-Security ๏ X-Frame-Options ๏ X-Content-Type-Options ๏ Content-Security-Policy Set them for every hostname or IP your server is reachable from the internet.

Slide 15

Slide 15 text

#engageug Strict-Transport-Security Strict Transport Security ๏ Force clients only to connect via HTTPS ๏ Set completely in notes.ini ๏ Enabled by default since 9.0.1 FP3 IF2 ๏ but: to short: ๏ HTTP_HSTS_MAX_AGE= 17280000 ๏ and: does not include subdomains ๏ HTTP_HSTS_INCLUDE_SUBDOMAINS=1

Slide 16

Slide 16 text

#engageug X-Content-Type-Options X-Content-Type Options ๏ Reduces exposure to drive-by downloads and the risks of user uploaded content that, with clever naming, could be treated as a different content- type, like an executable. ๏ X-Content-Type-Options nosniff ๏ Domino 9: need to be set ๏ Domino 10: active by default ๏ can be disabled by notes.ini parameter starting with 10.0.1 FP4 ๏ HTTP_DISABLE_X_CONTENT_TYPE_OPTIONS_NOSNIFF=1. ๏ Domino 11: active by default ๏ same as above

Slide 17

Slide 17 text

#engageug Content-Security-Policy Content-Security-Policy ๏ More complex only needed if you provide input fields ๏ Defines approved sources of content that the browser may load ๏ Can be an effective countermeasure to Cross Site Scripting (XSS) attacks ๏ You can whitelist approved sources ๏ Details about the options: ๏ https://scotthelme.co.uk/content-security-policy-an-introduction/

Slide 18

Slide 18 text

#engageug Domino Administration Domino Administration basics ๏ Update to the latest version of domino to be more secure! ๏ Do the minimum on Domino ๏ Disable Port 80 or at least redirect it to 443 ๏ Hide your server type ๏ It’s 2022: use at least TLS 1.2 ๏ Protect your users/customers with HTTP-Headers ๏ More security requires a Reverse Proxy ๏ They are built for that See all details here. Source: https://en.rnug.ru/wp-content/uploads/2021/01/rnug-letsmakeyourdominowebserverrocksolid.pdf

Slide 19

Slide 19 text

#engageug Integration Methods

Slide 20

Slide 20 text

#engageug API Integration at a Glance ๏User-Initiated ๏ Notes Client / Web / Mobile ๏ User Interaction ๏ Implementation: ๏ Notes Client Actions ๏ Agents (LS / Java) ๏ XPages App (Java) ๏ OSGi Plugins (Java) ๏Unattended ๏ Scheduled / Event-triggered ๏ No user interaction / Background ๏ Implementation: ๏ Agents (LS / Java) ๏ Agent → XPages (Java) ๏ DOTS (Java) ๏ XOTS (Java)

Slide 21

Slide 21 text

#engageug Language Selection ๏Java ๏ XPages ๏ Managed Beans, SSJS ๏ XOTS ๏ Plugins ๏ HTTP, DOTS ๏ Agents ๏ Java Agents ๏LotusScript ๏ Notes Client ๏ Actions, Events, etc. ๏ Agents ๏ Client-Side or Server-Side ๏JavaScript (limited) ๏ Web Browser ๏ SSJS ๏ App. Dev. Pack (?)

Slide 22

Slide 22 text

#engageug Java Agents ๏Java agents? Really Why? ๏ We still need them! ๏ Unattended run (scheduled agents) ๏ RunOnServer ๏Try to avoid, if possible ๏ Compatibility problems ๏ Memory Leak with JAR files ๏ Tip: Put your JAR files into /jvm/lib/ext ๏ (Very) Low Performance

Slide 23

Slide 23 text

#engageug Scheduling XPages Code ๏Why schedule XPages code? ๏ Agents are far from ideal ๏ Reuse the same codebase ๏ Modern(ish) development environment ๏How? ๏ Write your code ๏ /mydb.nsf/myAgent.xsp ๏ Create a scheduled agent ๏ Call “https://myserver.com/mydb.nsf/myAgent.xsp”

Slide 24

Slide 24 text

#engageug DOTS ๏DOTS: Domino OSGi Tasklet Services ๏ Server Tasklets for Domino ๏ Tasklet: A lightweight server task ๏ Run background tasks in a lightweight scalable container ๏ Separate JVM and OSGi Container ๏ Various triggers ๏ Manual (console commands / socket trigger) ๏ Scheduled or on server start ๏ Triggered (Hooks through data events)

Slide 25

Slide 25 text

#engageug DOTS ๏Some History ๏ 2010 - OpenNTF project contributed by IBM ๏ 2013 - Added to IBM Domino 9 Social Edition ๏ OpenSocial add-on ๏ Out of support (Internal Use Only) ๏ 2018 - Removed in Domino 10 ๏ 2021 - Came back into Domino 12 ๏ Installed with the Core product ๏ Support: Windows 64 and Linux 64 ๏ Upgraded to Eclipse OSGi 4.6.2 (Neon 2)

Slide 26

Slide 26 text

#engageug Lotus Script / Agents ๏Lotus Script / Agents ๏ Can do a lot (domino is able to work with other systems) ๏ Read / Write files ๏ Read / Write to SQL databases ๏ Read / Write SOAP Requests ๏ Read / Write REST APIs ๏ Remote control of other programs ๏ Can be invoked by ๏ Time: Scheduled Agent ๏ User: Action Button ๏ User: Events (eg opening a document) ๏Conclusion ๏ Usually works well. Implemented using non modern technology.

Slide 27

Slide 27 text

#engageug Real World Examples REAL WORLD EXAMPLES

Slide 28

Slide 28 text

#engageug Real World Examples ๏Scheduling XPages Code ๏ Hybrid Application ๏ 80% XPages, 20% Notes ๏ Documents to be sent a remote service over REST API ๏ New content and critical changes => Upload immediately ๏ Other changes => Queue and Upload next hour ๏ Upload code complications ๏ Java SDK supplied, fails in Java agents ๏ Upload should be scheduled ๏ Single document upload triggered by XPages and/or Notes Client ๏All possible complications in a single project…

Slide 29

Slide 29 text

#engageug Real World Examples ๏Scheduling XPages Code ๏ Upload business logic ๏ Java classes embedded in managed beans ๏ Utilise SDK and Apache HttpComponents ๏ XPages code ๏ Multimode: Upload specific document or process queue ๏ Java Agents ๏ Notes client can trigger Java agent to upload single document ๏ Scheduled Java agent runs every hour to trigger upload ๏ Security ๏ XPages code should only be triggered by a Java Agent ๏ Precautions: IP limitations, authentication, temporary backend document, etc.

Slide 30

Slide 30 text

#engageug Real World Examples ๏Uploading Data via FTP ๏ Problem: ๏ Newsletters and Campaign Communication ๏ Forced migration to a cloud-based e-mail marketing service ๏ Upload text files to a Secure FTP server ๏ Transient Customer data needed daily and weekly ๏ Custom Campaign data to be uploaded instantly ๏ Queries and Uploads take too much time and resource ๏ Security is the top-priority! ๏ Solution: ๏ DOTS Tasklets

Slide 31

Slide 31 text

#engageug Real World Examples ๏Uploading Data via FTP ๏ Tasklet can handle long-running upload process ๏ Watching queue every minute ๏ Very small footprint for queue monitoring ๏ Reuse Java code already developed before User builds a target query for the campaign Predefined target lists for newsletters UploadJob Queue DOTS Tasklet (Scheduled / Manual) •Fetch next UploadJob •Run query •Convert to CSV file •Compress •Upload to SCP Server Upload Job #1 Upload Job #2 … Upload Job #N

Slide 32

Slide 32 text

#engageug Real world examples ๏Expense Workflow - Accounting App Integration ๏ Very common scenario ๏ Expense form ๏ Reads and caches Project codes, Customers, etc. ๏ Writes back approved expense amounts ๏ Various Solutions ๏ SQL calls to Stored Procedures ๏ LotusScript using a DLL for integration ๏ Web Services (SOAP, REST, etc) ๏ Text file exchange

Slide 33

Slide 33 text

#engageug Real world SQLInterface class

Slide 34

Slide 34 text

#engageug Real world SQLInterface class: Init

Slide 35

Slide 35 text

#engageug Real world SQLInterface class: run sql

Slide 36

Slide 36 text

#engageug Real world SQLInterface class: demo of usage

Slide 37

Slide 37 text

#engageug I am confused!?* What should I use? ๏Whenever you can, use an REST JSON API ๏ Instead of interacting with databases directly (LSX, ODBC) ๏ Instead of file transfers (Export / Import) ๏ And yes, instead of using HEI ๏ Use JAVA. Lotus Script implementation is far behind! ๏But keep in mind ๏ APIs can change very frequently (e.g. Microsoft M365) ๏ You don’t have control on details ๏ You don’t know the details (but most of the time, you are not interested in)

Slide 38

Slide 38 text

#engageug Common tasks Common tasks

Slide 39

Slide 39 text

#engageug The question of life, the universe and all the rest There are only two hard things in Computer Science: - Cache invalidation and - Naming things (Source: Phil Karlton (Former Netscape developper)) „ „

Slide 40

Slide 40 text

#engageug Caching: How to ๏Caching: Main challenges ๏ When to provide ๏ When you request the same data multiple times ๏ The data you receive do not change very often or you know exactly when ๏ Time required to obtain the data is high ๏ YES, all of them! ๏ When to clean (cache invalidation) ๏ As early as possible without loosing the advantages ๏ Time based ๏ Data change recognised

Slide 41

Slide 41 text

#engageug Caching: Javascript Example

Slide 42

Slide 42 text

#engageug Java Libraries ๏Utilise Libraries ๏ Practical Development ๏ OpenNTF Domino API ๏ Apache Commons ๏ Network Connections (REST APIs) ๏ Performance, compatibility, simplicity, protection from implementation changes ๏ HttpComponents (Apache Commons) ๏ Misc. ๏ POI (Java API for Microsoft Documents)

Slide 43

Slide 43 text

#engageug OSGi Plugins ๏Make use of OSGi Plugins ๏ Larger integration projects ๏ Multiple applications/business processes ๏ Broader functionality (UI modules, server-wide caching etc.) ๏ Easier implementations for various scenarios ๏ e.g. OAuth authentication ๏ Libraries ๏ XPages run in a secure container → Compatibility problems ๏ Jackson, Gson, some JDBC-wrappers, etc.

Slide 44

Slide 44 text

#engageug Logging ๏Use modern logging ๏ Log Levels ๏ Less is more, more is not enough… ๏ Consider stages ๏ Dev/Test and Production ๏ Tip: notes.ini ๏ Try OpenLog by Julian Robichaux ๏ Lotusscript: OpenLog ๏ XPages: XPages OpenLog Logger (Paul Withers) ๏ Java (XPages, DOTS, OSGi): XLogback (Serdar Basegmez)

Slide 45

Slide 45 text

#engageug JSON Processing using JAVA ๏What to process? How to process? ๏ Create/Consume JSON String ๏ Maps, Arrays, etc. ๏ IBM Commons, preloaded in Domino ๏ JSON ↔ Java Object conversions ๏ Direct Conversion between POJO and JSON ๏ Gson, Jackson, etc. ๏ JSON Streaming ๏ Read/Write large files/network streams ๏ Optimised CPU/Memory usage ๏ Gson, Jackson, etc.

Slide 46

Slide 46 text

#engageug JSON Processing ๏Java Options ๏ IBM Commons JSON Library ๏ Included in Domino ๏ Encapsulated Maps and Lists ๏ Gson (Google) ๏ Mid-weight, easy to use ๏ Object mapping, POJO conversion, streaming, etc. ๏ Jackson (FasterXML) ๏ More features (JAX-RS provider, language bindings, etc.) ๏ Tip: Gson and Jackson use Reflection ๏ Encapsulate into plugin ๏ Disable security (java.pol file)

Slide 47

Slide 47 text

#engageug JSON Processing using Javascript ๏How to process? ๏ Basic functionality of Javascript ๏ Very fast ๏ Very easy ๏ JSON ↔ JS Object conversions ๏ JSON.stringify ๏ JSON.parse

Slide 48

Slide 48 text

#engageug JSON Processing using Lotus Script ๏How to process? ๏ Two main classes to work with ๏ NotesJSONNavigator (reading) ๏ NotesJSONObject (writing) ๏ JSON ↔ JS Object conversions ๏ You have to run through the object ๏ No direct access ๏ Avoid to use it, if you work with large objects ๏ Buggy Has some difficulties ๏ Slow Needs time for processing and traversing ๏ Alternative (if you really have to, maybe) ๏ ls.snapps.JSONReader from OpenNTF

Slide 49

Slide 49 text

#engageug Java Tricks ๏JavaOptionsFile ๏ Adding JVM Options using notes.ini ๏ Setting TLS protocols (Important for Java Agents) ๏ Additional debugging, tweak third party libraries ๏ Client and Server ๏ Alternative Form: JavaUserOptionsFile ๏ Only custom options (after “-D”) ๏ DOTS uses a different JVM ๏ DOTS_JavaOptionsFile

Slide 50

Slide 50 text

#engageug Java Tricks ๏SSL and TLS issues ๏ Issues with HTTPS connections in Java agents ๏ Before Domino 9.0.1FP7, ๏ Use JavaOptionsFile to force TLS1.2 ๏ Technote KB0028812 ๏ SSL Handshake Errors ๏ Certification Authority information outdated ๏ Add trusted certificates to CACERTS ๏ Technote KB0035853 ๏ Much older versions ๏ Try your luck with Bouncy Castle

Slide 51

Slide 51 text

#engageug Litte helpers TOOLS / UTILITIES

Slide 52

Slide 52 text

#engageug Dev / Testing helper software / apps ๏Postman ๏PAW ๏SOAPUI ๏CURL ๏JSONLINT

Slide 53

Slide 53 text

#engageug Security Assessment / test websites Check your server ๏ SSL Labs (www.ssllabs.com) ๏ Grade change in January 2020 if you provide TLS 1.0/1.1

Slide 54

Slide 54 text

#engageug Security Assessment / test websites ๏ https://securityheaders.com/ ๏ Not every header is for every site ๏ Check your site functionality! HTTP Response Headers can break your site

Slide 55

Slide 55 text

Any questions?