Engage 2019: Your Data in the Major Leagues: A Practical and Updated Guide to RESTful Domino v10

Engage 2019 session with Tom Van Aken (https://twitter.com/vanakentom):

Your Data in the Major Leagues: A Practical and Updated Guide to RESTful Domino v10

Things are moving fast with many new tools and toys in our hands. While Domino v10 creates a whole new world of possibilities, we still need to interact with other internal and external applications. One of the key technologies to do that is the RESTful services and we have an updated toolbox now. Regardless of your experience, come to this session to see what options are available to you, what pitfalls you may experience and how to break down the borders between your applications and others. We will present the basic concepts and best practices, allowing you to walk away with the techniques that you will need to take your applications to the next level. Last but not the least, we will show you practical examples on how you provide RESTful services for other applications and how you can use REST data to enrich your applications using the the new capabilities.

Speakers:
Serdar Basegmez , Developi Information Systems
Tom Van Aken, GroupWave

Transcript

1. #engageug Your Data in the Major Leagues: A Prac6cal and Updated Guide to RESTful Domino v10 Serdar Basegmez Developi serdar_basegmez vanakentom Tom Van Aken GroupWave
2. #engageug Serdar Basegmez • IBM Champion Alumni (2011 - 2018) • Developi InformaCon Systems, London • Notes/Domino/XPages Developer, half-blooded admin! • Member Director at OpenNTF Board • Blog: LotusNotus.com [English / Turkish] • Blogger and Podcaster on ScienCfic SkepCcism / CriCcal Thinking serdar_basegmez
3. #engageug Tom Van Aken • First Cme IBM Champion in 2019 • Groupwave, KonCch - Belgium • Notes Domino / Xpage developer • Blog: vanakentom.wordpress.com [Dutch] vanakentom
4. #engageug Agenda • RESTful Architecture • PracCcal ImplicaCons and Advantages of RESTful Services • Ways to Provide REST Services for IBM Domino • Consuming REST Services within IBM Domino • Wrap-up
5. #engageug RESTful Web Services Source: https://en.wikipedia.org/wiki/Representational_state_transfer Representa)onal state transfer (REST) is an architectural style used for web development. Systems and sites designed using this style aim for fast performance, reliability and the ability to scale (to grow and easily support extra users). To achieve these goals, developers work with reusable components that can be managed and updated without affec)ng the system as a whole while it is running.
6. #engageug History
7. #engageug Source: https://speakerdeck.com/jeffschenck/rest-easy-api-security-done-right User Interface Business Logic Datastore Front-end Back-end ASP, PHP, CGI, Web Agents, JSP, etc. ← HTML, CSS, JavaScript Forms → Old School Web ApplicaCons
8. #engageug Web ApplicaCons Evolving User Interface Business Logic Datastore Front-end Back-end Async web apps, Ruby on Rails, Django, JSF, XPages, etc. ← HTML, CSS, JavaScript Forms, AJAX →
9. #engageug Web ApplicaCons Evolving User Interface Business Logic Datastore Front-end Back-end Modern Web frameworks, Angular.js, React.js, etc. ← HTML, CSS, JavaScript ← REST →
10. #engageug [Web] ApplicaCons Evolving User Interface Business Logic Datastore Mobile Applications Back-end Modern Web frameworks, Angular.js, React.js, etc. ← HTML, CSS, JavaScript ← REST → Front-end
11. #engageug [Web] ApplicaCons Evolving User Interface Business Logic Datastore Mobile Applications Back-end Modern Web frameworks, Angular.js, React.js, etc. ← HTML, CSS, JavaScript ← REST → Front-end Microservice Microservice Microservice
12. #engageug [Web] ApplicaCons Evolving Back-end Modern Web frameworks, Angular.js, React.js, etc. ← HTML, CSS, JavaScript ← REST → User Interface Business Logic Datastore Mobile Applications Front-end External Apps Microservice Microservice Microservice Integration
13. #engageug RESTful, Everywhere! Solid Architecture Well-defined practices Widespread use Easily consumable Scalable
14. #engageug The ConversaCon Makes Sense! Source: http://www.bizcoder.com/a-fresh-coat-of-rest-paint-on-a-soap-stack
15. #engageug The ConversaCon Makes Sense! GET /twink/contacts/DLEY-ACLH6Y HTTP/1.1 Host: homer.developi.info Cache-Control: no-cache { "zip": "13202", "state": "NY", "lastName": "Abbate", "middle": "J", "country": "US", "emailAddress": "Jessica.J.Abbate@trashymail.com", "number": "DLEY-ACLH6Y", "city": "Syracuse", "firstName": "Jessica" }
16. #engageug The ConversaCon Makes Sense! hPp://appserver.company.com/apps/contacts.nsf/ GiveMeTheContactWeNeedPleaseAgent?OpenAgent&id=1522 or… hPp://appserver.company.com/api/contacts/1522
17. #engageug Conventions on URLs GET hPp://appserver.company.com/api/contacts GET hPp://appserver.company.com/api/contacts/UK/London POST hPp://appserver.company.com/api/contacts Retrieve Contacts / Create a new Contact…
18. #engageug Conventions on URLs GET hPp://appserver.company.com/api/contacts/1522 PUT hPp://appserver.company.com/api/contacts/1522 DELETE hPp://appserver.company.com/api/contacts/1522 Retrieve/Update/Delete the Contact resource with id=1522…
19. #engageug URI GET PUT POST DELETE /contacts/ List Contacts Replace Contacts Create New Contact Delete Contacts /contacts/id Retrieve a Contact Replace a Contact N/A (generally) Delete a Contact Source: https://en.wikipedia.org/wiki/Representational_state_transfer
20. #engageug Different Conventions GET hPps://api.twiPer.com/1.1/statuses/show.json?id=1234567890 Retrieve the Tweet with id=1234567890…
21. #engageug RESTful Services for IBM Domino Applica6ons
22. #engageug Some InspiraCon • New Front-ends and Enhanced UX • JS Frameworks (e.g. Angular, React), Mobile Apps, etc. • Grids, quick searches, becer type-aheads, etc. • Richer experiences with chatbots, AI, etc. • IntegraCon for third party sites/applicaCons • Pull exchange rates, get weather info • Get customers from CRM, create leads • Upload to Box, Send to Slack… • AutomaCng processes using APIs • Push data to the AccounCng system… • Access to ERP, check the inventory, iniCate a new PO User Experience Business Processes Integration
23. #engageug Providing RESTful Services Domino Access Services Extension Library REST Components Hardcoded Solutions (Web Agents, XAgents, Servlets) Advanced J2EE (JAX-RS) SmartNSF Node.js #OpenNTF #Domino10
24. #engageug Domino Access Services IBM Supported Fully Func6onal REST API Access to Data / Calendar / Mail* / Freebusy* / Directory** * Mail and Freebusy access is part of the core product since 9.0.1FP10 / ** Directory access is enabled with the ExtLib version of OpenNTF Caching / Na6ve Security (ACL, Reader/Author fields, etc.) Server-level / Database-level / Design-level control Enable on Server Enable For Database Enable For Views
25. #engageug Domino Access Services • OpenAPI definiCons (Swagger) are hosted on OpenNTF! • hcps://github.com/OpenNTF/das-api-specs
26. #engageug Domino Access Services • Drawbacks: • Weak control over the data! • No checkpoints on CRUD, No coding involved… • Weak control over the access. • No place for business logic! • Consumer has to implement all the business logic. • Exposes internals • You should trust consumers and the environment…
27. #engageug REST Components (ExtLib) • Provided and Supported by IBM • Customizable component version of the DAS • Computed/Filtered columns, Custom search, etc. • Event model helps building business logic on top of REST model • Custom REST Service • Write your own SSJS or Java bean • Write your CSJS rouCnes for async access • Remote Service / JSON-RPC • Dojo support for single page model
28. #engageug REST Components (ExtLib) • Setup REST component(s) on your page. • Minimal coding, no administrator needed. Add to your XPage Add a Service Configure Options
29. #engageug REST Components (ExtLib) • Drawbacks: • Easy to slip into a spageh code! • Java beans are your friends… • Not opCmized for performance and scalability • Difficult to follow RESTful URL ConvenCon https://someserver.domain.com/database.nsf/somepage.xsp/service/…
30. #engageug Hardcoding (Web agents, XAgents, Servlets…) • Old school way; sCll quite useful for some cases. • Great if you have pre-exisCng code (e.g. Lotusscript libraries, etc.) • Customizable, flexible and simple way to create any service
31. #engageug Hardcoding (Web agents, XAgents, Servlets…) • Drawbacks: • Hardcode everything… • e.g. Header/parameter extracCon • Very easy to slip into a spageh code! • Error handling / proper tesCng needed • Difficult to follow RESTful URL ConvenCon https://someserver.domain.com/database.nsf/xagent.xsp?… https://someserver.domain.com/database.nsf/someagent?OpenAgent&…
32. #engageug J2EE (JAX-RS) Methods • JAX-RS: ‘Java-ish’ way to define RESTful services • Create JAX-RS based REST services on top of OSGi plugins. • Complete Java soluCon, extensible with custom providers • OpCons • Apache Wink 1.1.2 (DAS uses!) • Jakarta EE • RestEasy, Apache CXF, Jersey, etc. • Code reusability outside IBM Domino world.
33. #engageug J2EE (JAX-RS) Methods On Apache Wink… hcps://speakerdeck.com/sbasegmez/iconuk-2016-rest-assured... On Jakarta EE… (by Jesse Gallagher) hcps://github.com/OpenNTF/org.opennp.xsp.jakartaee
34. #engageug J2EE (JAX-RS) Methods • Drawbacks: • Plugin only • Difficult if you are not familiar, Takes Cme to learn • Overkill? • Not suitable for small projects and simple needs • Tool selecCon is criCcal. • Apache Wink is old school • IntegraCng alternaCves might be difficult
35. #engageug SmartNSF • OpenNTF Plugin • Beta-7 version. • Developers: • ChrisCan Güdemann • MarCn Jinoch • Define your REST Services from DDE! • Uses Groovy • Auto-generate OpenAPI definiCons Looking for Contributors!
36. #engageug Node.js • Just arrived: DQL + Domino AppDev Pack: • Domino Query Language • AppDev Pack: Proton Task / domino-db.js • Countless opportuniCes in the most popular framework • Different possibiliCes for the architecture • Wide range of opCons for tooling https://insights.stackoverflow.com/survey/2019
37. #engageug Node.js IBM Domino Server Proton NSF Node.js Layer domino-db.js GRPC Application ……… Application REST New Add-on Domino v10 Routers for the RESTful service Might be on the same box or not MyRoutes.js
38. #engageug Providing RESTful Services on Domino Benefits Challenges Suggested When? Domino Access Services
 (DAS) Zero-setup True RESTful No Programming No Business Logic Exposes the Internals Simple data exchange ExtLib Components
 for REST Less Backend Code Minimal Setup Partial/Full Customization Error Handling Spaghetti Code Risk URL Convention Simple needs for a limited scope Hardcoding
 (XAgents, Web agents…) Tailor-made Any Language is OK Hardcoding Everything Spaghetti Code Risk URL Conventions Very specific needs for a limited scope J2EE / JAX-RS Tailor-made OSGi Benefits True RESTful Learning Curve Barrier to Entry Environment Setup Large scope implementation, API Design Node.js Tailor Made Node.js Benefits True RESTful Learning Curve Still Improving Environment Setup Modernization Projects SmartNSF Beta Less Backend Code True RESTful Learning Curve Limited Support Simple needs for a limited scope
39. #engageug Consuming RESTful Services …or Teaching your app to browse the Internet… Language Preference? Is it On-demand or Background? Server-side or Client-side? SDK? Authentication?
40. #engageug Language Preference • Server-side JavaScript • Already uses Java. Eliminate the middleman… • Java • XPages Managed Beans, Java Agents, OSGi Plugins, etc. • ConnecCng a remote • URLConnecCon (Core Java) • HcpComponents (Apache Commons) • JSON Processing • JsonJavaObject (IBM Commons) • Jackson, Gson, etc.
41. #engageug ConnecCng Java • Pulling exchange rates via Java (Agent or Managed Beans) public Double receiveEurUsdRate() throws Exception { CloseableHttpClient httpclient = HttpClients.createDefault(); HttpGet httpGet = new HttpGet("http://api.fixer.io/latest?base=USD"); CloseableHttpResponse response = httpclient.execute(httpGet); try { HttpEntity entity = response.getEntity(); // This is the worst practice ever! // We don't check anything! Everything can go wrong... JsonJavaObject ratesMap = (JsonJavaObject) JsonParser.fromJson(JsonJavaFactory.instanceEx, EntityUtils.toString(entity)); JsonJavaObject rates = ratesMap.getAsObject("rates"); // We can write values into a NotesDocument return rates.getAsDouble("EUR"); } finally { response.close(); } } * Networking is a restricted operation for Agents! Agent Security should be configured. Create an HTTP client Get a Response Convert to JSON
42. #engageug ConnecCng SSJS • Pulling exchange rates on an XPage acCon <xp:button id="button1" value="What is the exchange rate?"> <xp:eventHandler event="onclick" submit="true" refreshMode="partial" refreshId="computedField1"> <xp:this.action><![CDATA[#{javascript: viewScope.EurUsdRate = test.ExchangeRate.receiveEurUsdRate() }]]></xp:this.action> </xp:eventHandler> </xp:button> * This code will run multiple times for each page cycle. Always cache values! Call Java (via Managed Bean)
43. #engageug Reminder for Java • Consuming any REST Service (When needed or on a schedule) • Running from an Java Agent • Import Apache HcpComponents/IBM Commons • Configure Agent Security • Running from an XPage / Java or SSJS • Apache HcpComponents should be imported per NSF. • Time-outs and service hiccups should be considered • Values should always be cached!
44. #engageug Language Preference • Using SDK/Wrapper • Saves Cme and effort • Ower provided and supported by the vendor/community. • Box, IBM Watson, Salesforce, Quickbooks, etc. • AdapCng to Domino • CompaCbility issues. • Java version mismatch (prior to v9.0.1 FP9), Java security issues, etc.
45. #engageug Language Preference • Lotusscript • No internal support for Networking… • On Windows plaporm, we can use COM objects • Call Java via LS2J N O M O R E !
46. #engageug New CapabiliCes of Domino v10! • Create HTTP requests… • NotesHTTPRequest • Manipulate JSON… • NotesJSONNavigator • NotesJSONElement • NotesJSONArray • NotesJSONObject
47. #engageug NotesHTTPRequest • GET • POST • PUT • PATCH (V10.0.1) • DELETERESOURCE (V10.0.1) • Preferstrings • False: As Byte Array • True: As Text
48. #engageug Parsing JSON • NotesJSONNavigator • getElementByName • getElementByPointer • gepirstElement / getnextElement / getNthElement • preferUTF8 • False : gets String values • True : gets Byte Array in UTF-8 • NotesJSONElement • NotesJSONArray
49. #engageug Work has to be done • CreateJsonNavigator causes problems with CR/LF and internaConal characters • SoluCon=save file as UTF-8 and read as binary • JSONNavigator doesn’t handle internaConal characters well • Will be solved by FP2. • Only JSON parsing, No creaCon of JSON (yet?) • AlternaCve: ls.snapps.json… libraries • No REST compliant Agent NotaCon • Makes it less suitable for Providing REST services
50. #engageug Consuming RESTful Services Usage Examples XPages
 Java - SSJS Through SSJS or Java beans Call when needed Sending message to Slack Integrations to IBM Watson Social Media Interaction Agents Java - Lotusscript Scheduled agents Call by Notes Client Periodically pulling exchange rates Pushing data to remote service Training IBM Watson AI Using SDKs Access to services using libraries Call when needed Schedule / Trigger Accessing IBM Connections Integration to Box
51. #engageug Slide-storming on Details
52. #engageug Architecture… RESTful Remote Web Browser Domino Server Web App Agents XPages Notes Client On Demand REST Conversation Scheduled REST Conversation On Demand REST Conversation On Demand REST Conversation
53. #engageug Architecture Decision… • Who iniCates a RESTful conversaCon? When? How? • Trigger: On demand vs. Schedule vs. Event-based • How dynamic is the remote data? Can we use caching? • The remote can be accessed by the server or the client? • Intranet services cannot be accessed by remote users. • What about security? Who has the password for the remote? • SSO / OAuth can be a challenge in many cases. • Predefined credenCals might be exposed in some cases.
54. #engageug AuthenCcaCon • Key quesCon, depending on the remote service. • Who is going to be authenCcated and how? • Predefined credenCal or applicaCon key • Single Sign-on • Users’ own credenCals on remote (OAuth, Password…)
55. #engageug Some Tweaks • SSL Issue • TLS 1.2 is required in most cases. • Technote: hcps://www-01.ibm.com/support/docview.wss? uid=swg21985289 • HTTP Proxy/Gateway • Proxy through the Domino Server for CSJS requests (Mostly for security) • OpCons • IBM Domino provides HTTP-proxy servlet • ProgrammaCcally create a gateway using “Remote Service” component • TesCng • Postman / MockBin / RequestBin / hcpbin / JSON View
56. #engageug Resources • Serdar Başeğmez: Node.js demo for this session
 hcps://github.com/sbasegmez/Engage19Demo • Tom Van Aken: Rest calls and JSON Parsing in LotusScript
 hcps://vanakentom.wordpress.com/2019/01/15/rest-calls-and-json-parsing-in-lotusscript/ • Serdar Başeğmez: Demo for IBM Connect 2017 session
 hcps://github.com/sbasegmez/IC17RestDemo • Serdar Başeğmez: Apache Wink Template and Demo for Icon UK 2016
 hcps://github.com/sbasegmez/RestAssuredDemo • Graham Acres / Serdar Başeğmez: The Journey to Becoming a Social ApplicaCon Developer (IBM Connect 2014)
 hcps://speakerdeck.com/sbasegmez/bp308-the-journey-to-becoming-a-social-applicaCon-developer • Stephan H. Wissel: Custom REST service in XPages using a service bean
 hcps://wissel.net/blog/d6plinks/SHWL-9Q55QL
57. #engageug Resources (cont.) • Eric McCormick: Series on JSON Data with Java in XPages
 hcps://edm00se.io/json-with-java-in-xpages • Thomas Ladehoff: REST Services with the XPages Extension Library
 hcps://www.assono.de/blog/d6plinks/REST-Services-with-the-XPages-Extension-Library • Paul Withers: XPages OSGi Plugins series
 hcp://www.intec.co.uk/xpages-osgi-plugins-1-an-introducCon/ • John Cooper: Domino OSGI (Part 1) - Configuring Eclipse for XPages OSGI Plugins
 hcp://developmentblog.johnmcooper.co.uk/2014/05/configuring-eclipse-for-xpages-osgi-plugins-part1.html • Toby Samples: JAX-RS or THE way to do REST in Domino series
 hcps://tobysamples.wordpress.com/2015/04/28/jax-rs-or-the-way-to-do-rest-in-domino-part-1/ • Jesse Gallagher: Eclipse Tutorial for Domino Developers
 hcps://github.com/jesse-gallagher/eclipse-tutorial-oct2015/wiki/Java
58. #engageug Resources (cont.) • Ulrich Krause: NotesJsonNavigator, NotesJsonElement, NotesJsonArray, NotesJsonObject example
 hcps://www.eknori.de/2019-01-01/notesjsonnavigator-notesjsonelement-notesjsonarray-notesjsonobject-example/ • John Dalsgaard: REST Services in IBM Domino/XWork
 hcps://www.dalsgaard-data.eu/blog/rest-services-in-ibm-dominoxwork • Oliver Busse: First dive into Domino and Node.js
 hcp://oliverbusse.notesx.net/hp.nsf/blogpost.xsp? documentId=2E52&SessionID=00997C8066130BED21F9B372F01DADDA32214EAE • Oliver Busse: Domino, Proton, IAM, OAuth (Series)
 hcp://oliverbusse.notesx.net/hp.nsf/blogpost.xsp?documentId=2FEA • Sven Hasselbach: node.js, domino-db & Docker
 hcp://hasselba.ch/blog/?p=2670 • Jesse Gallagher: That Java Thing, Part 1: The Java Problem in the Community
 hcps://frosCllic.us/blog/thread.xsp?thread=That+Java+Thing
59. #engageug Resources (cont.) • Graham Acres / Heiko Voigt: Engage 2019 Demo of Domino v10, Proton and Node.js session 
 hcps://github.com/c3ug/engage2019democode • Troy Reimer: OpenNTF Project: LotusScript implementaCon of a reader and writer for JSON
 hcps://opennp.org/main.nsf/project.xsp?r=project/JSON%20LotusScript%20Classes
60. #engageug Thank you Serdar Basegmez Developi serdar_basegmez serdar.basegmez@developi.com lotusnotus.com Tom Van Aken GroupWave vanakentom tom.vanaken@groupwave.be vanakentom.wordpress.com