The ABCs of Connector Design

This session will demonstrate how to design and scope a TIBCO Cloud Integration Scribe connector using the software development kit, focusing on authentication, metadata, and operations. We will look at some common pitfalls and how best to deal with them. We will share examples of connector code, so you will be ready to begin your own connector project.

This deck prepared for TIBCO Scribe Conference 2018 @ Boston.

Transcript

1. The ABCs of Connector Design Vladimir Almaev https://aquivalabs.com
2. • Architect @ Aquiva Labs • 10 Years in Software Industry • 13 Connectors: • Scribe Platform API • MySQL • Scribe.Connector.AdoNet Library • Bunch of connectors based on it • valmaev@aquivalabs.com • @vladimir_almaev
3. Blood Language Design Patterns Code A LOT of Code
6. So, how to learn a connector design?
7. So, how to learn a connector design? By implementing a connector of course!
8. Agenda • Discovery • Connection • Metadata • Queries • Operations
9. 368c5ba Essentials Advanced
10. $ git log --oneline * cd38df8 Implement IConnector.Connect method in non-testable fashion * d45eb81 Exclude async methods from Guard Clause checking test in ScribeApiClient * 16c38e2 Define preprocessor variables to connector lifecycle diagram * 1b01a3c Add sequence diagram with connector lifecycle * b7c03e9 Add ScribeApiClient.CheckConnection which checks connectivity with API * 02e9620 Add ScribeApiClient which creates pre-configured HttpClient * dff6c1d Add link to conference website in readme * 73820bc Add readme * fcc917c Swap keys and values in Environment entry options * efd2cb6 Implement PreConnect to return serialized FormDefinition * 3ad1946 Implement ConnectorTypeId to return value from ScribeConnectorAttribute * 5959ab2 Implement IConnector on ScribeApiConnector in dummy way * 3b575c7 Add ScribeApiConnector class annotated with ScribeConnectorAttribute * 368c5ba Add initial solution 368c5ba
11. $ git log --oneline * cd38df8 Implement IConnector.Connect method in non-testable fashion * d45eb81 Exclude async methods from Guard Clause checking test in ScribeApiClient * 16c38e2 Define preprocessor variables to connector lifecycle diagram * 1b01a3c Add sequence diagram with connector lifecycle * b7c03e9 Add ScribeApiClient.CheckConnection which checks connectivity with API * 02e9620 Add ScribeApiClient which creates pre-configured HttpClient * dff6c1d Add link to conference website in readme * 73820bc Add readme * fcc917c Swap keys and values in Environment entry options * efd2cb6 Implement PreConnect to return serialized FormDefinition * 3ad1946 Implement ConnectorTypeId to return value from ScribeConnectorAttribute * 5959ab2 Implement IConnector on ScribeApiConnector in dummy way * 3b575c7 Add ScribeApiConnector class annotated with ScribeConnectorAttribute * 368c5ba Add initial solution 368c5ba github.com/valmaev/scribe-connector-demo
12. Discovery • Build a walking skeleton • ScribeConnector Attribute • IConnector
13. ❶ Annotate with ScribeConnector
14. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7
15. 3b575c7 [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } MEF • Inherits ExportAttribute • Implements IScribeConnectorMetadata • Additional Metadata
16. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Showed in Marketplace
17. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 GUID of the Connector • Feature Branches • Multiple Deployments of the same Connector • Can be useful in Testing
18. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Same as the Type 
 of the Connector
19. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Connector Version • Start with 0.0.0.0 • Same as AssemblyVersion and AssemblyFileVersion • Versioning Strategy
20. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Connector can support multiple Roles
21. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Connector can support multiple Roles Supported Legacy "Scribe.IS2.Source"
 "Scribe.IS2.Target" "Scribe.IS.Source"
 "Scribe.IS.Target" "Scribe.MS2.Source"
 "Scribe.MS2.Target" "Scribe.MS.Source"
 "Scribe.MS.Target" "Scribe.RS.Source"
 "Scribe.RS.Target"
22. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Connector can support multiple Roles Supported Legacy "Scribe.IS2.Source"
 "Scribe.IS2.Target" "Scribe.IS.Source"
 "Scribe.IS.Target" "Scribe.MS2.Source"
 "Scribe.MS2.Target" "Scribe.MS.Source"
 "Scribe.MS.Target" "Scribe.RS.Source"
 "Scribe.RS.Target" Integration Services (IS) • Integration Solution • On-Demand and Scheduled • Integration Event Solution • Triggered by HTTP request
23. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Cloud Agents • Set to true in most cases • Set to false if you have a strong reason
24. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Connection UI Type • Generic UI is for every developer • It's not available for
 3rd party developers
25. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Settings UI Type • Must be empty if Connector uses Generic UI • It's not available for
 3rd party developers
26. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7 Should be empty for 3rd party Connectors
27. [ScribeConnector( name: "TIBCO Scribe Platform API (Demo)", description: "Metaconnector which connects platform to itself.", connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", connectorType: typeof(ScribeApiConnector), connectorVersion: "0.0.0.0", supportedSolutionRoles: new[] {"Scribe.IS2.Source", "Scribe.IS2.Target"}, supportsCloud: true, connectionUITypeName: "ScribeOnline.GenericConnectionUI", connectionUIVersion: "1.0.0.0", settingsUITypeName: "", settingsUIVersion: "1.0.0.0", xapFileName: "")] public sealed class ScribeApiConnector { } 3b575c7
28. ❷ Implement IConnector
29. [ScribeConnector(/* ... */)] public sealed class ScribeApiConnector : IConnector { public Guid ConnectorTypeId { get; } public bool IsConnected { get; } public string PreConnect(IDictionary<string, string> properties) h> ""; public void Connect(IDictionary<string, string> properties) { } public void Disconnect() { } public IMetadataProvider GetMetadataProvider() h> null; public IEnumerable<DataEntity> ExecuteQuery(Query query) h> new DataEntity[0]; public OperationResult ExecuteOperation(OperationInput input) h> null; public MethodResult ExecuteMethod(MethodInput input) h> null; } 5959ab2
30. [ScribeConnector(/* ... */)] public sealed class ScribeApiConnector : IConnector { public Guid ConnectorTypeId { get; } public bool IsConnected { get; } public string PreConnect(IDictionary<string, string> properties) h> ""; public void Connect(IDictionary<string, string> properties) { } public void Disconnect() { } public IMetadataProvider GetMetadataProvider() h> null; public IEnumerable<DataEntity> ExecuteQuery(Query query) h> new DataEntity[0]; public OperationResult ExecuteOperation(OperationInput input) h> null; public MethodResult ExecuteMethod(MethodInput input) h> null; } 5959ab2
31. ❸ Choose ConnectorTypeId
32. [ScribeConnector( /* ... */ connectorTypeId: "e0e8461e-92ae-4b5f-980b-a2dd104b7d24", /* ... */)] public sealed class ScribeApiConnector : IConnector { public Guid ConnectorTypeId { get; } = Guid.Parse("e0e8461e-92ae-4b5f-980b-a2dd104b7d24"); } 3ad1946 Should be the same
34. [ScribeConnector( /* ... */ connectorTypeId: ConnectorTypeIdString, /* ... */)] public sealed class ScribeApiConnector : IConnector { private const string ConnectorTypeIdString = "e0e8461e-92ae-4b5f-980b-a2dd104b7d24"; public Guid ConnectorTypeId { get; } = Guid.Parse(ConnectorTypeIdString); } 3ad1946 You can use private const as an attribute argument
35. Connection • Compose UI • Establish Connection • Disconnect • Testability
36. [ScribeConnector(/* ... */)] public sealed class ScribeApiConnector : IConnector { public Guid ConnectorTypeId { get; } public bool IsConnected { get; } public string PreConnect(IDictionary<string, string> properties) h> ""; public void Connect(IDictionary<string, string> properties) { } public void Disconnect() { } public IMetadataProvider GetMetadataProvider() h> null; public IEnumerable<DataEntity> ExecuteQuery(Query query) h> new DataEntity[0]; public OperationResult ExecuteOperation(OperationInput input) h> null; public MethodResult ExecuteMethod(MethodInput input) h> null; } 5959ab2
40. public string PreConnect(IDictionary<string, string> properties) { var form = new FormDefinition { CompanyName = "Aquiva Labs", CryptoKey = CryptoKey, HelpUri = new Uri("https://aquivalabs.com"), Entries = new List<EntryDefinition>() }; return form.Serialize(); } efd2cb6 Generic UI • PreConnect return serialized FormDefinition
41. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; var form = new FormDefinition { CompanyName = "Aquiva Labs", CryptoKey = CryptoKey, HelpUri = new Uri("https://aquivalabs.com"), Entries = new List<EntryDefinition>() }; efd2cb6
42. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; var form = new FormDefinition { CompanyName = "Aquiva Labs", CryptoKey = CryptoKey, HelpUri = new Uri("https://aquivalabs.com"), Entries = new List<EntryDefinition>() }; efd2cb6 CryptoKey • Platform encrypts secrets stored in InputType.Password • IConnector.Connect should decrypt them using it
43. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; var form = new FormDefinition { CompanyName = "Aquiva Labs", CryptoKey = CryptoKey, HelpUri = new Uri("https://aquivalabs.com"), Entries = new List<EntryDefinition>() }; efd2cb6
44. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; var form = new FormDefinition { CompanyName = "Aquiva Labs", CryptoKey = CryptoKey, HelpUri = new Uri("https://aquivalabs.com"), Entries = new List<EntryDefinition> { /* Entries */ } }; efd2cb6
45. new EntryDefinition { Label = "Environment", PropertyName = "Environment", InputType = InputType.Text, IsRequired = false, Order = 0, Options = { ["QA"] = "https://qaendpoint.scribeqa.net", ["DEMO"] = "https://demoendpoint.scribeqa.net", ["SANDBOX"] = "https://sbendpoint.scribesoft.com", ["PRODUCTION"] = "https://api.scribesoft.com" } } efd2cb6 Label vs PropertyName • Labels shown to user • PropertyNames used in code
46. new EntryDefinition { Label = "Environment", PropertyName = "Environment", InputType = InputType.Text, IsRequired = false, Order = 0, Options = { ["QA"] = "https://qaendpoint.scribeqa.net", ["DEMO"] = "https://demoendpoint.scribeqa.net", ["SANDBOX"] = "https://sbendpoint.scribesoft.com", ["PRODUCTION"] = "https://api.scribesoft.com" } } efd2cb6
47. new EntryDefinition { Label = "Environment", PropertyName = "Environment", InputType = InputType.Password, IsRequired = false, Order = 0, Options = { ["QA"] = "https://qaendpoint.scribeqa.net", ["DEMO"] = "https://demoendpoint.scribeqa.net", ["SANDBOX"] = "https://sbendpoint.scribesoft.com", ["PRODUCTION"] = "https://api.scribesoft.com" } } efd2cb6
48. new EntryDefinition { Label = "Environment", PropertyName = "Environment", InputType = InputType.Text, IsRequired = false, Order = 0, Options = { ["QA"] = "https://qaendpoint.scribeqa.net", ["DEMO"] = "https://demoendpoint.scribeqa.net", ["SANDBOX"] = "https://sbendpoint.scribesoft.com", ["PRODUCTION"] = "https://api.scribesoft.com" } } efd2cb6
49. new EntryDefinition { Label = "Environment", PropertyName = "Environment", InputType = InputType.Text, IsRequired = true, Order = 0, Options = { ["QA"] = "https://qaendpoint.scribeqa.net", ["DEMO"] = "https://demoendpoint.scribeqa.net", ["SANDBOX"] = "https://sbendpoint.scribesoft.com", ["PRODUCTION"] = "https://api.scribesoft.com" } } efd2cb6 IsRequired • You can't test connection without filling Required entries TEST
50. efd2cb6 new EntryDefinition { Label = "Environment", PropertyName = "Environment", InputType = InputType.Text, IsRequired = true, Order = 0, Options = { ["QA"] = "https://qaendpoint.scribeqa.net", ["DEMO"] = "https://demoendpoint.scribeqa.net", ["SANDBOX"] = "https://sbendpoint.scribesoft.com", ["PRODUCTION"] = "https://api.scribesoft.com" } } IsRequired • You can't test connection without filling Required entries TEST
51. new EntryDefinition { Label = "Environment", PropertyName = "Environment", InputType = InputType.Text, IsRequired = false, Order = 0, Options = { ["QA"] = "https://qaendpoint.scribeqa.net", ["DEMO"] = "https://demoendpoint.scribeqa.net", ["SANDBOX"] = "https://sbendpoint.scribesoft.com", ["PRODUCTION"] = "https://api.scribesoft.com" } } efd2cb6 0 1 2
53. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8
54. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8
55. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8 [0] = "[Environment, https://sbendpoint.scribesoft.com]" [1] = "[Username, john@doe.com]" [2] = "[Pasword, W4WBPPYN58R+IzALc2PCzQuv]" [3] = "[ScribeOnline_ConnectionInfoDBId, f271e50f-87dc-47f1-a8f3-1c64447a297
57. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8 IsConnected • Add setter for the property • Change to true only after verifying properties
58. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8
59. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8
60. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8 Scribe API Client • Static class with: • Factory Methods • Extension methods for HttpClient which calls API
62. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8 Any problems with this code?
63. private const string CryptoKey = "3103dcf5-6d7c-4b56-8297-f9e449b57576"; public bool IsConnected { get; private set; } public void Connect(IDictionary<string, string> properties) { IsConnected = false; var baseAddress = new Uri(properties["Environment"] ?? ""); var username = properties["Username"]; var password = Decryptor.Decrypt_AesManaged( properties["Password"], CryptoKey); using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); IsConnected = true; } cd38df8 Any problems with this code? HARD TO UNIT TEST
64. public sealed class ScribeApiConnector : IConnector { private readonly HttpMessageHandler _httpMessageHandler; private HttpClient _httpClient; public ScribeApiConnector() : this(new HttpClientHandler()) { } public ScribeApiConnector(HttpMessageHandler httpMessageHandler) { if (httpMessageHandler uv null) throw new ArgumentNullException(nameof(httpMessageHandler)); _httpMessageHandler = httpMessageHandler; } } c60c47c Dependency Injection • Inject HttpMessageHandler via constructor • Possible to build custom request-response pipeline
65. public sealed class ScribeApiConnector : IConnector { private readonly HttpMessageHandler _httpMessageHandler; private HttpClient _httpClient; public ScribeApiConnector() : this(new HttpClientHandler()) { } public ScribeApiConnector(HttpMessageHandler httpMessageHandler) { if (httpMessageHandler uv null) throw new ArgumentNullException(nameof(httpMessageHandler)); _httpMessageHandler = httpMessageHandler; } } c60c47c Field for HttpClient • Instantiate it later in Connect • Lifetime of HttpClient now is tied to lifetime of ScribeApiConnector
66. public sealed class ScribeApiConnector : IConnector { private readonly HttpMessageHandler _httpMessageHandler; private HttpClient _httpClient; public ScribeApiConnector() : this(new HttpClientHandler()) { } public ScribeApiConnector(HttpMessageHandler httpMessageHandler) { if (httpMessageHandler uv null) throw new ArgumentNullException(nameof(httpMessageHandler)); _httpMessageHandler = httpMessageHandler; } } c60c47c Default Constructor • Keep default constructor for MEF • Instantiate dependencies 
 for Production run
67. public sealed class ScribeApiConnector : IConnector { private readonly HttpMessageHandler _httpMessageHandler; private HttpClient _httpClient; public ScribeApiConnector() : this(new HttpClientHandler()) { } public ScribeApiConnector(HttpMessageHandler httpMessageHandler) { if (httpMessageHandler uv null) throw new ArgumentNullException(nameof(httpMessageHandler)); _httpMessageHandler = httpMessageHandler; } } c60c47c
68. public void Connect(IDictionary<string, string> properties) { using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); } c60c47c
69. public void Connect(IDictionary<string, string> properties) { using (var httpClient = ScribeApiClient.Create(baseAddress, username, password)) httpClient.CheckConnection().GetAwaiter().GetResult(); _httpClient = ScribeApiClient.Create(baseAddress, username, password, _httpMessageHandler); _httpClient.CheckConnection().GetAwaiter().GetResult(); } c60c47c
70. [Fact] public void ScribeApiConnector_Connect_WithValidProperties_ShouldSetIsConnectedToTrue() { var httpHandler = new StubbedResponseHandler( new HttpResponseMessage( HttpStatusCode.OK)); var sut = new ScribeApiConnector(httpHandler); var input = new Dictionary<string, string> { ["Environment"] = "https://sbendpoint.scribesoft.com", ["Username"] = "foo", ["Password"] = Encryptor.Encrypt_AesManaged( value: "foobar", key: "3103dcf5-6d7c-4b56-8297-f9e449b57576") }; Assert.False(sut.IsConnected); sut.Connect(input); Assert.True(sut.IsConnected); } c60c47c
71. [Fact] public void ScribeApiConnector_Connect_WithValidProperties_ShouldSetIsConnectedToTrue() { var httpHandler = new StubbedResponseHandler( new HttpResponseMessage( HttpStatusCode.OK)); var sut = new ScribeApiConnector(httpHandler); var input = new Dictionary<string, string> { ["Environment"] = "https://sbendpoint.scribesoft.com", ["Username"] = "foo", ["Password"] = Encryptor.Encrypt_AesManaged( value: "foobar", key: "3103dcf5-6d7c-4b56-8297-f9e449b57576") }; Assert.False(sut.IsConnected); sut.Connect(input); Assert.True(sut.IsConnected); } c60c47c
72. [Fact] public void ScribeApiConnector_Connect_WithValidProperties_ShouldSetIsConnectedToTrue() { var httpHandler = new StubbedResponseHandler( new HttpResponseMessage( HttpStatusCode.OK)); var sut = new ScribeApiConnector(httpHandler); var input = new Dictionary<string, string> { ["Environment"] = "https://sbendpoint.scribesoft.com", ["Username"] = "foo", ["Password"] = Encryptor.Encrypt_AesManaged( value: "foobar", key: "3103dcf5-6d7c-4b56-8297-f9e449b57576") }; Assert.False(sut.IsConnected); sut.Connect(input); Assert.True(sut.IsConnected); } c60c47c
73. [Fact] public void ScribeApiConnector_Connect_WithValidProperties_ShouldSetIsConnectedToTrue() { var httpHandler = new StubbedResponseHandler( new HttpResponseMessage( HttpStatusCode.OK)); var sut = new ScribeApiConnector(httpHandler); var input = new Dictionary<string, string> { ["Environment"] = "https://sbendpoint.scribesoft.com", ["Username"] = "foo", ["Password"] = Encryptor.Encrypt_AesManaged( value: "foobar", key: "3103dcf5-6d7c-4b56-8297-f9e449b57576") }; Assert.False(sut.IsConnected); sut.Connect(input); Assert.True(sut.IsConnected); } c60c47c
74. [Fact] public void ScribeApiConnector_Connect_WithValidProperties_ShouldSetIsConnectedToTrue() { var httpHandler = new StubbedResponseHandler( new HttpResponseMessage( HttpStatusCode.OK)); var sut = new ScribeApiConnector(httpHandler); var input = new Dictionary<string, string> { ["Environment"] = "https://sbendpoint.scribesoft.com", ["Username"] = "foo", ["Password"] = Encryptor.Encrypt_AesManaged( value: "foobar", key: "3103dcf5-6d7c-4b56-8297-f9e449b57576") }; Assert.False(sut.IsConnected); sut.Connect(input); Assert.True(sut.IsConnected); } c60c47c
76. public void Disconnect() { _httpClient?.Dispose(); IsConnected = false; } cd38df8 Disconnect • Close Connection • Clean-up the resources • ~IDisposable.Dispose() • IsConnected = false
77. Hello World
78. Metadata • Kinds of metadata • Strategies for Metadata Provider • AOP • Challenges { fullName: "Cat", supportedActions: ["sleep", "eat", "meow", "repeat"] }
79. Scribe CDK Scribe UI Relational DB ObjectDefinition Entity Table
80. Scribe CDK Scribe UI Relational DB PropertyDefinition Field Column
81. Scribe CDK Scribe UI Relational DB RelationshipDefinition Relationship Relationship
82. Scribe CDK Scribe UI Relational DB ActionDefinition Command Block DQL + DML = CRUD ⁷ CRUD ⁷
83. [ScribeConnector(/* ... */)] public sealed class ScribeApiConnector : IConnector { public Guid ConnectorTypeId { get; } public bool IsConnected { get; } public string PreConnect(IDictionary<string, string> properties) h> ""; public void Connect(IDictionary<string, string> properties) { } public void Disconnect() { } public IMetadataProvider GetMetadataProvider() h> null; public IEnumerable<DataEntity> ExecuteQuery(Query query) h> new DataEntity[0]; public OperationResult ExecuteOperation(OperationInput input) h> null; public MethodResult ExecuteMethod(MethodInput input) h> null; } 4351667
84. public interface IMetadataProvider : IDisposable { IEnumerable<IActionDefinition> RetrieveActionDefinitions(); IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false) h> new IObjectDefinition[0]; IObjectDefinition RetrieveObjectDefinition( string objectName, bool shouldGetProperties = false, bool shouldGetRelations = false); IEnumerable<IMethodDefinition> RetrieveMethodDefinitions( bool shouldGetParameters = false); IMethodDefinition RetrieveMethodDefinition( string objectName, bool shouldGetParameters = false); void ResetMetadata() { } } 4351667
85. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = nameof(KnownActions.Query), Name = nameof(KnownActions.Query), Description = nameof(KnownActions.Query), KnownActionType = KnownActions.Query, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 ActionDefinition • Simplest possible implementation of IActionDefinition • CDK provides it
86. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = "Query", Name = "Query", Description = "Query", KnownActionType = KnownActions.Query, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 Names • FullName both used in code and shown to the user • Don't know why we need
 Name and Description :)
87. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = nameof(KnownActions.Query), Name = nameof(KnownActions.Query), Description = nameof(KnownActions.Query), KnownActionType = KnownActions.Query, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 KnownActionType • Affects UI: appearance of action tabs
88. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = nameof(KnownActions.Query), Name = nameof(KnownActions.Query), Description = nameof(KnownActions.Query), KnownActionType = KnownActions.Query, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 Names • Can be any string • For standard actions you can use names of KnownActions enum
89. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = nameof(KnownActions.Query), Name = nameof(KnownActions.Query), Description = nameof(KnownActions.Query), KnownActionType = KnownActions.Query, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 KnownActionType • Affects UI: appearance of action tabs
90. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = nameof(KnownActions.Query), Name = nameof(KnownActions.Query), Description = nameof(KnownActions.Query), KnownActionType = KnownActions.Create, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 KnownActionType • Affects UI: appearance of action tabs
91. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = nameof(KnownActions.Query), Name = nameof(KnownActions.Query), Description = nameof(KnownActions.Query), KnownActionType = KnownActions.Update, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 KnownActionType • Affects UI: appearance of action tabs
92. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = nameof(KnownActions.Query), Name = nameof(KnownActions.Query), Description = nameof(KnownActions.Query), KnownActionType = KnownActions.Delete, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 KnownActionType • Affects UI: appearance of action tabs
93. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = "Command", Name = "Command", Description = "Command", KnownActionType = KnownActions.Create, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 Special Actions • Non-CRUD actions can be handled like this • Examples: send email, run background job, etc.
94. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() h> new[] { new ActionDefinition { FullName = nameof(KnownActions.Query), Name = nameof(KnownActions.Query), Description = nameof(KnownActions.Query), KnownActionType = KnownActions.Query, SupportsBulk = false, SupportsConstraints = true, SupportsInput = false, SupportsRelations = false, SupportsSequences = true, SupportsLookupConditions = true, SupportsMultipleRecordOperations = false } } 4351667 Other Toggles • Turn on/off some features like bulk operations
95. * Actually, you'll see the actions in the UI only after providing the Object Definitions
96. Who're these guys?
97. Autogenerated blocks • Scribe can autogenerate blocks • Query -> Fetch & Lookup • Update + Insert -> Update/ Insert
98. public interface IMetadataProvider : IDisposable { IEnumerable<IActionDefinition> RetrieveActionDefinitions(); IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false); IObjectDefinition RetrieveObjectDefinition( string objectName, bool shouldGetProperties = false, bool shouldGetRelations = false); IEnumerable<IMethodDefinition> RetrieveMethodDefinitions( bool shouldGetParameters = false); IMethodDefinition RetrieveMethodDefinition( string objectName, bool shouldGetParameters = false); void ResetMetadata(); } fd0755d Singular VS Plural • Singular: get the ObjectDefinition by FullName • Plural: get all the ObjectDefinitions
99. public IObjectDefinition RetrieveObjectDefinition( string objectName, bool shouldGetProperties = false, bool shouldGetRelations = false) { return RetrieveObjectDefinitions(shouldGetProperties, shouldGetRelations) .Single(x h> x.FullName uv objectName); } e0d706a Semantics • Singular: get the ObjectDefinition by FullName
100. public IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false) h> new [] { new ObjectDefinition { FullName = "Organization", Name = "Organization", Description = "Represents the entity that is using TIBCO Scribe ® Online", Hidden = false, SupportedActionFullNames = new List<string> { nameof(KnownActions.Query), nameof(KnownActions.Create), nameof(KnownActions.Update), nameof(KnownActions.Delete), "SpecialCommand" }, PropertyDefinitions = new List<IPropertyDefinition> {/* ... */}, RelationshipDefinitions = new List<IRelationshipDefinition>{/* ... */} } }; fd0755d Names • FullName should be unique • To handle here: • escaping of symbols • fully-qualified names
101. public IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false) h> new [] { new ObjectDefinition { FullName = "Organization", Name = "Organization", Description = "Represents the entity that is using TIBCO Scribe ® Online", Hidden = false, SupportedActionFullNames = new List<string> { nameof(KnownActions.Query), nameof(KnownActions.Create), nameof(KnownActions.Update), nameof(KnownActions.Delete), "SpecialCommand" }, PropertyDefinitions = new List<IPropertyDefinition> {/* ... */}, RelationshipDefinitions = new List<IRelationshipDefinition>{/* ... */} } }; fd0755d Hidden • You can hide your ObjectDefinition
102. public IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false) h> new [] { new ObjectDefinition { FullName = "Organization", Name = "Organization", Description = "Represents the entity that is using TIBCO Scribe ® Online", Hidden = false, SupportedActionFullNames = new List<string> { nameof(KnownActions.Query), nameof(KnownActions.Create), nameof(KnownActions.Update), nameof(KnownActions.Delete), "SpecialCommand" }, PropertyDefinitions = new List<IPropertyDefinition> {/* ... */}, RelationshipDefinitions = new List<IRelationshipDefinition>{/* ... */} } }; fd0755d Supported Actions • Full Names of supported Actions • Should be in sync with RetrieveActionDefinitions
103. public IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false) h> new [] { new ObjectDefinition { FullName = "Organization", Name = "Organization", Description = "Represents the entity that is using TIBCO Scribe ® Online", Hidden = false, SupportedActionFullNames = new List<string> { nameof(KnownActions.Query), nameof(KnownActions.Create), nameof(KnownActions.Update), nameof(KnownActions.Delete), "SpecialCommand" }, PropertyDefinitions = new List<IPropertyDefinition> {/* ... */}, RelationshipDefinitions = new List<IRelationshipDefinition>{/* ... */} } }; fd0755d Parameters • Whether or not to provide Properties or Relationships • Can be ignored if it's cheap
 to provide them
104. public IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false) h> new [] { new ObjectDefinition { FullName = "Organization", Name = "Organization", Description = "Represents the entity that is using TIBCO Scribe ® Online", Hidden = false, SupportedActionFullNames = new List<string> { nameof(KnownActions.Query), nameof(KnownActions.Create), nameof(KnownActions.Update), nameof(KnownActions.Delete), "SpecialCommand" }, PropertyDefinitions = new List<IPropertyDefinition> {/* ... */}, RelationshipDefinitions = new List<IRelationshipDefinition>{/* ... */} } }; fd0755d
105. new PropertyDefinition { Description = "Organization Name", FullName = "Name", Name = "Name", IsPrimaryKey = false, Nullable = true, NumericPrecision = 0, NumericScale = 0, PresentationType = columnType uv typeof(DateTime) ? nameof(PresentationType.DateTime) : null, PropertyType = typeof(string).FullName, Size = 0, UsedInActionInput = true, UsedInActionOutput = true, UsedInLookupCondition = true, UsedInQueryConstraint = true, UsedInQuerySelect = true, UsedInQuerySequence = true }; 0432551
106. public interface IMetadataProvider : IDisposable { IEnumerable<IActionDefinition> RetrieveActionDefinitions(); IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false); IObjectDefinition RetrieveObjectDefinition( string objectName, bool shouldGetProperties = false, bool shouldGetRelations = false); IEnumerable<IMethodDefinition> RetrieveMethodDefinitions( bool shouldGetParameters = false); IMethodDefinition RetrieveMethodDefinition( string objectName, bool shouldGetParameters = false); void ResetMetadata(); } c3b66bf MethodDefinitions • These guys are for Replication Services
107. public interface IMetadataProvider : IDisposable { IEnumerable<IActionDefinition> RetrieveActionDefinitions(); IEnumerable<IObjectDefinition> RetrieveObjectDefinitions( bool shouldGetProperties = false, bool shouldGetRelations = false); IObjectDefinition RetrieveObjectDefinition( string objectName, bool shouldGetProperties = false, bool shouldGetRelations = false); IEnumerable<IMethodDefinition> RetrieveMethodDefinitions( bool shouldGetParameters = false); IMethodDefinition RetrieveMethodDefinition( string objectName, bool shouldGetParameters = false); void ResetMetadata(); } 4ba7a96, 0c28526 ResetMetadata & Dispose • ResetMetadata: clean cache • Dispose: clean resources
108. Possible Strategies • Hardcoding
109. Possible Strategies • Hardcoding
110. Possible Strategies • Hardcoding • Convert native schema (DBs with schema, Swagger, WSDL) • Annotate Domain objects / DTOs with Attributes • Keep metadata in external File (XML, JSON, etc.)
111. [AttributeUsage( validOn: AttributeTargets.Class | AttributeTargets.Struct | AttributeTargets.Interface, AllowMultiple = false, Inherited = false)] public class ObjectDefinitionAttribute : Attribute, IObjectDefinition { public string FullName { get; set; } = ""; public string Name { get; set; } = ""; public string Description { get; set; } = ""; public bool Hidden { get; set; } public List<string> SupportedActionFullNames { get; set; } public List<IPropertyDefinition> PropertyDefinitions { get; set; } public List<IRelationshipDefinition> RelationshipDefinitions { get; set; } } 889ed17 Attributes • Each attribute implements metadata interface like IObjectDefinition • Reasonable default values
112. [ObjectDefinition] [ActionDefinition(KnownActions.Query)] [ActionDefinition(KnownActions.Create)] [ActionDefinition(KnownActions.CreateWith)] public class Organization { [PropertyDefinition( UsedInActionOutput = true, UsedInActionInput = false, RequiredInActionInput = false, UsedInQueryConstraint = true, UsedInLookupCondition = false, IsPrimaryKey = true)] public int? Id { get; set; } } 2683c04 Usage of Attributes • Annotate your Domain Objects or DTOs with attributes
113. public sealed class AttributeBasedMetadataProvider : IMetadataProvider { public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { return EntityAssembly .GetTypes() .Where(t h> t.GetCustomAttributes<ObjectDefinitionAttribute>() .Any(o h> !o.Hidden)) .SelectMany(t h> t.GetCustomAttributes<ActionDefinitionAttribute>()) .Distinct(this.actionDefinitionComparer); } /* Other Methods */ } 0466101 MetadataProvider • Scans attribute usages with Reflection • We can use Class, Property,
 Field names as FullNames
114. [ObjectDefinition] [ActionDefinition(KnownActions.Query)] [ActionDefinition(KnownActions.Create)] [ActionDefinition(KnownActions.CreateWith)] public class Organization { [PropertyDefinition( UsedInActionOutput = true, UsedInActionInput = false, RequiredInActionInput = false, UsedInQueryConstraint = true, UsedInLookupCondition = false, IsPrimaryKey = true)] public int? Id { get; set; } } new ObjectDefinition { FullName = "Organization", Name = "Organization", SupportedActionFullNames = new List<string> { "Query", "Create" }, PropertyDefinitions = new List<IPropertyDefinition> { new PropertyDefinition { FullName = "Id", Name = "Id", UsedInActionOutput = true, UsedInActionInput = true, RequiredInActionInput = false, UsedInQueryConstraint = true, UsedInLookupCondition = true, IsPrimaryKey = true } } }; 0466101
115. By The Way • It's a good idea to cache your metadata • And measure performance of your provider • What about logging?
116. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { return EntityAssembly .GetTypes() .Where(t h> t.GetCustomAttributes<ObjectDefinitionAttribute>() .Any(o h> !o.Hidden)) .SelectMany(t h> t.GetCustomAttributes<SupportedActionAttribute>()) .Distinct(this.actionDefinitionComparer); } 0466101
117. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { if (_cache.IsEmpty) { _cache.ActionDefinitions = EntityAssembly .GetTypes() .Where(t h> t.GetCustomAttributes<ObjectDefinitionAttribute>() .Any(o h> !o.Hidden)) .SelectMany(t h> t.GetCustomAttributes<SupportedActionAttribute>()) .Distinct(this.actionDefinitionComparer); } return _cache.ActionDefinitions; }
118. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { try { if (_cache.IsEmpty) { _cache.ActionDefinitions = EntityAssembly .GetTypes() .Where(t h> t.GetCustomAttributes<ObjectDefinitionAttribute>() .Any(o h> !o.Hidden)) .SelectMany(t h> t.GetCustomAttributes<SupportedActionAttribute>()) .Distinct(this.actionDefinitionComparer); } return _cache.ActionDefinitions; } catch (Exception ex) { ScribeLogger.Instance.WriteException("Panic!", ex); } }
119. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { using (new LogMethodExecution("MyConnectorName", nameof(RetrieveActionDefinitions))) { try { if (_cache.IsEmpty) { _cache.ActionDefinitions = EntityAssembly .GetTypes() .Where(t h> t.GetCustomAttributes<ObjectDefinitionAttribute>() .Any(o h> !o.Hidden)) .SelectMany(t h> t.GetCustomAttributes<SupportedActionAttribute>()) .Distinct(this.actionDefinitionComparer); } return _cache.ActionDefinitions; } catch (Exception ex) { ScribeLogger.Instance.WriteException("Panic!", ex); } } }
120. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { using (new LogMethodExecution("MyConnectorName", nameof(RetrieveActionDefinitions))) { try { if (_cache.IsEmpty) { return EntityAssembly .GetTypes() .Where(t h> t.GetCustomAttributes<ObjectDefinitionAttribute>() .Any(o h> !o.Hidden)) .SelectMany(t h> t.GetCustomAttributes<SupportedActionAttribute>()) .Distinct(this.actionDefinitionComparer); } else { return _cache.ActionDefinitions; } } catch (Exception ex) { ScribeLogger.Instance.WriteException("Panic!", ex); } } } And we have 8 other methods…
121. public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { using (new LogMethodExecution("MyConnectorName", nameof(RetrieveActionDefinitions))) { try { if (_cache.IsEmpty) { return EntityAssembly .GetTypes() .Where(t h> t.GetCustomAttributes<ObjectDefinitionAttribute>() .Any(o h> !o.Hidden)) .SelectMany(t h> t.GetCustomAttributes<SupportedActionAttribute>()) .Distinct(this.actionDefinitionComparer); } else { return _cache.ActionDefinitions; } } catch (Exception ex) { ScribeLogger.Instance.WriteException("Panic!", ex); } } } And we have 8 other methods…
122. public class LoggingMetadataProvider : IMetadataProvider { public LoggingMetadataProvider(IMetadataProvider decoratedProvider) { DecoratedProvider = decoratedProvider; } public IMetadataProvider DecoratedProvider { get; } public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { try { return DecoratedProvider.RetrieveActionDefinitions(); } catch (Exception ex) { ScribeLogger.Instance.WriteException(
 this, ex); throw; } } } ff7bd09 Decorator • Implements some interface • Accepts instance of the same interface in constructor
123. public class LoggingMetadataProvider : IMetadataProvider { public LoggingMetadataProvider(IMetadataProvider decoratedProvider) { DecoratedProvider = decoratedProvider; } public IMetadataProvider DecoratedProvider { get; } public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { try { return DecoratedProvider.RetrieveActionDefinitions(); } catch (Exception ex) { ScribeLogger.Instance.WriteException(
 this, ex); throw; } } } ff7bd09 Decorator • Wraps decorated instance with new functionality • Delegates main work for decorated instance
124. public class LoggingMetadataProvider : IMetadataProvider { public LoggingMetadataProvider(IMetadataProvider decoratedProvider) { DecoratedProvider = decoratedProvider; } public IMetadataProvider DecoratedProvider { get; } public IEnumerable<IActionDefinition> RetrieveActionDefinitions() { try { return DecoratedProvider.RetrieveActionDefinitions(); } catch (Exception ex) { ScribeLogger.Instance.WriteException(
 this, ex); throw; } } } ff7bd09
125. [ScribeConnector(/* ... */)] public sealed class ScribeApiConnector : IConnector { public IMetadataProvider GetMetadataProvider() h> new PerformanceMeasuringMetadataProvider( new LoggingMetadataProvider( new CachingMetadataProvider( new AttributeBasedMetadataProvider( Assembly.GetExecutingAssembly(), t h> t.Namespace uv typeof(Organization).Namespace)), ConnectorName); } ff78291 Decorator • The beauty of Composition • Aspect-Oriented Programming
126. [ScribeConnector(/* ... */)] public sealed class ScribeApiConnector : IConnector { public IMetadataProvider GetMetadataProvider() h> new PerformanceMeasuringMetadataProvider( new LoggingMetadataProvider( new CachingMetadataProvider( new AttributeBasedMetadataProvider( Assembly.GetExecutingAssembly(), t h> t.Namespace uv typeof(Organization).Namespace)), ConnectorName); } ff78291 Decorator • The beauty of Composition • Aspect-Oriented Programming
127. Challenges • MySQL • TreatTinyAsBoolean • AllowZeroDateTime, ConvertZeroDateTime • Schema-less Databases: MongoDB • Heuristic: discover the schema based on the content
130. Queries • Filters • Laziness • Pagination • Native Queries • Net Change SELECT Id, ParentId, Name FROM Organization WHERE Status = "OK"
 AND ParentId = "123" LIMIT 25;
131. [ScribeConnector(/* ... */)] public sealed class ScribeApiConnector : IConnector { public Guid ConnectorTypeId { get; } public bool IsConnected { get; } public string PreConnect(IDictionary<string, string> properties) h> ""; public void Connect(IDictionary<string, string> properties) { } public void Disconnect() { } public IMetadataProvider GetMetadataProvider() h> null; public IEnumerable<DataEntity> ExecuteQuery(Query query) h> new DataEntity[0]; public OperationResult ExecuteOperation(OperationInput input) h> null; public MethodResult ExecuteMethod(MethodInput input) h> null; } 5959ab2
132. public class Query { public QueryEntity RootEntity { get; set; } public Expression Constraints { get; set; } public int RowLimit { get; set; } } string RootEntity.ObjectDefinitionFullName SELECT Id, ParentId, Name FROM Organization WHERE Status = "OK"
 AND ParentId = "123" ORDER BY Id ASC LIMIT 25;
133. public class Query { public QueryEntity RootEntity { get; set; } public Expression Constraints { get; set; } public int RowLimit { get; set; } } SELECT Id, ParentId, Name FROM Organization WHERE Status = "OK"
 AND ParentId = "123" ORDER BY Id ASC LIMIT 25; List<string> RootEntity.PropertyList
134. public class Query { public QueryEntity RootEntity { get; set; } public Expression Constraints { get; set; } public int RowLimit { get; set; } } SELECT Id, ParentId, Name FROM Organization WHERE Status = "OK"
 AND ParentId = "123" ORDER BY Id ASC LIMIT 25; List<Sequence> RootEntity.SequenceList
135. public class Query { public QueryEntity RootEntity { get; set; } public Expression Constraints { get; set; } public int RowLimit { get; set; } } SELECT Id, ParentId, Name FROM Organization WHERE Status = "OK"
 AND ParentId = "123" ORDER BY Id ASC LIMIT 25;
136. public class Query { public QueryEntity RootEntity { get; set; } public Expression Constraints { get; set; } public int RowLimit { get; set; } } SELECT Id, ParentId, Name FROM Organization WHERE Status = "OK"
 AND ParentId = "123" ORDER BY Id ASC LIMIT 25; Traverse the Expression Tree
137. public class Query { public QueryEntity RootEntity { get; set; } public Expression Constraints { get; set; } public int RowLimit { get; set; } } Queries • Map to external system request: • SQL, HTTP, SOAP, etc. SELECT Id, ParentId, Name FROM Organization WHERE Status = "OK"
 AND ParentId = "123" ORDER BY Id ASC LIMIT 25;
138. Coffee… Filters!
139. Expression Types • Enum contains a lot more values that you'll not see in the wild
140. Comparison Expressions • Leaf elements • LeftValue is a Property FullName • RightValue is a Constant
141. Logical Expressions • Xor and Or you'll not see in the wild • Logical Expressions increase
 the depth of the tree
144. 6ab4a5f Visitor • Entry-point method which accepts Expression of base type • Instead of void Visitor can return accumulated value public void Visit(Expression expression) { switch (expression.ExpressionType) { case ExpressionType.Comparison: Visit((ComparisonExpression) expression)); break; case ExpressionType.Logical: Visit((LogicalExpression) expression)); break; default: throw new InvalidOperationException(); } }
145. 6ab4a5f Visitor • Dispatch Expressions by ExpressionType and call appropriate Visit method overload public void Visit(Expression expression) { switch (expression.ExpressionType) { case ExpressionType.Comparison: Visit((ComparisonExpression) expression)); break; case ExpressionType.Logical: Visit((LogicalExpression) expression)); break; default: throw new InvalidOperationException(); } }
146. public void Visit(Expression expression) { switch (expression.ExpressionType) { case ExpressionType.Comparison: Visit((ComparisonExpression) expression)); break; case ExpressionType.Logical: Visit((LogicalExpression) expression)); break; default: throw new InvalidOperationException(); } } 6ab4a5f Visitor • Recursively visit Left and Right parts of LogicalExpression private void Visit(LogicalExpression expression) { Visit(expression.LeftExpression); Visit(expression.RightExpression); switch (expression.Operator) { case LogicalOperator.And: break; case LogicalOperator.Or: break; default: throw new InvalidOperationException(); } }
147. public void Visit(Expression expression) { switch (expression.ExpressionType) { case ExpressionType.Comparison: Visit((ComparisonExpression) expression)); break; case ExpressionType.Logical: Visit((LogicalExpression) expression)); break; default: throw new InvalidOperationException(); } } 6ab4a5f Visitor • ComparisonExpression is the Base Case • Once we reach it, we start
 to unwind recursion stack private void Visit(LogicalExpression expression) { Visit(expression.LeftExpression); Visit(expression.RightExpression); switch (expression.Operator) { case LogicalOperator.And: break; case LogicalOperator.Or: break; default: throw new InvalidOperationException(); } } private void Visit(ComparisonExpression expression) { }
148. private void Visit(LogicalExpression expression) { Visit(expression.LeftExpression); Visit(expression.RightExpression); switch (expression.Operator) { case LogicalOperator.And: // Combine subexpressions with AND break; case LogicalOperator.Or: // Combine subexpressions with OR break; default: throw new InvalidOperationException(); } } public void Visit(Expression expression) { switch (expression.ExpressionType) { case ExpressionType.Comparison: Visit((ComparisonExpression) expression)); break; case ExpressionType.Logical: Visit((LogicalExpression) expression)); break; default: throw new InvalidOperationException(); } } 6ab4a5f Visitor • Convert Comparison and Logical Expressions to
 target system expression • SQL, HTTP request, etc. private void Visit(ComparisonExpression expression) { // Convert ComparisonExpression to target system expression }
150. Laziness & Pagination • Query potentially can return A LOT of results • Always use pagination • HTTP APIs: tricky combination of yield and async • ADO.NET: DataAdapter supports pagination • Lazily return result from each page • i.e. request next page only after previous page was read
151. Laziness & Pagination • Query potentially can return A LOT of results • Always use pagination • HTTP APIs: tricky combination of yield and async • ADO.NET: DataAdapter supports pagination • Lazily return result from each page • i.e. request next page only after previous page was read
152. Net Change • Process most recent records based on some Property values • Available for Objects with at least one DateTime Property • To deactivate use string type • For developers it's a filter
153. Native Queries • KnownActionType.NativeQuery • string Query.NativeQueryText bool Query.IsNativeQuery • Any native query language • You should be able to
 infer the metadata by query
154. // Raw sketch of query execution logic public IEnumerable<DataEntity> ExecuteQuery(Query query) { var request = _queryTranslator.Translate(query); var response = _client.SendRequest(request); foreach (var page in response.Pages) foreach (var row in page.Rows) { // Using special ObjectDefinition name for native queries var entity = new DataEntity( query.IsNativeQuery ? nameof(KnownActionType.NativeQuery) : query.ParentEntity.ObjectDefinitionFullName); // Filling EntityProperties which is just a Dictionary<string, object> // • Key: PropertyDefinition.FullName // • Value: value of the property/field _dataEntityConverter.Convert(entity, row); yield return entity; } }
155. Operations • Operation Results • Bulk • Error Types INSERT INTO Organization (Id, Name) VALUES (1, "Aquiva Labs"), (2, "ACME, Inc.");
156. [ScribeConnector(/* ... */)] public sealed class ScribeApiConnector : IConnector { public Guid ConnectorTypeId { get; } public bool IsConnected { get; } public string PreConnect(IDictionary<string, string> properties) h> ""; public void Connect(IDictionary<string, string> properties) { } public void Disconnect() { } public IMetadataProvider GetMetadataProvider() h> null; public IEnumerable<DataEntity> ExecuteQuery(Query query) h> new DataEntity[0]; public OperationResult ExecuteOperation(OperationInput input) h> null; public MethodResult ExecuteMethod(MethodInput input) h> null; } 5959ab2
157. public class OperationInput { public string Name { get; set; } public DataEntity[] Input { get; set; } public Expression[] LookupCondition { get; set; } } UPDATE Organization SET Name = "ACME, Inc.", Website = "whatev.er" WHERE Status = "OK"
 AND ParentId = "123"; IActionDefinition.FullName }> Target System Statement
158. public class OperationInput { public string Name { get; set; } public DataEntity[] Input { get; set; } public Expression[] LookupCondition { get; set; } } Input[0].ObjectDefinitionFullName UPDATE Organization SET Name = "ACME, Inc.", Website = "whatev.er" WHERE Status = "OK"
 AND ParentId = "123";
159. public class OperationInput { public string Name { get; set; } public DataEntity[] Input { get; set; } public Expression[] LookupCondition { get; set; } } UPDATE Organization SET Name = "ACME, Inc.", Website = "whatev.er" WHERE Status = "OK"
 AND ParentId = "123"; Input[0].EntityProperties.Key }> Property Name
 Input[0].EntityProperties.Value }> Property Value
160. public class OperationInput { public string Name { get; set; } public DataEntity[] Input { get; set; } public Expression[] LookupCondition { get; set; } } UPDATE Organization SET Name = "ACME, Inc.", Website = "whatev.er" WHERE Status = "OK"
 AND ParentId = "123"; Traverse the Expression Tree
 (exactly the same way as for Queries)
161. public class OperationInput { public string Name { get; set; } public DataEntity[] Input { get; set; } public Expression[] LookupCondition { get; set; } } UPDATE Organization SET Name = "ACME, Inc.", Website = "whatev.er" WHERE Status = "OK"
 AND ParentId = "123"; Operations • Map to external system request: • SQL, HTTP, SOAP, etc.
162. public class OperationInput { public string Name { get; set; } public DataEntity[] Input { get; set; } public Expression[] LookupCondition { get; set; } } [0] [1] [2] [3] [4] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Name = "Zorg" Site = "Fred" Name = "Thud" Site = "Noob" Name = "Shme" Site = "Spqr" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Parallel Arrays • Arrays are of the same size
163. public class OperationInput { public string Name { get; set; } public DataEntity[] Input { get; set; } public Expression[] LookupCondition { get; set; } } [0] [1] [2] [3] [4] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Name = "Zorg" Site = "Fred" Name = "Thud" Site = "Noob" Name = "Shme" Site = "Spqr" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Parallel Arrays • Arrays are of the same size • IActionDefinition.SupportsBulk • false }> Size = 1
164. public class OperationInput { public string Name { get; set; } public DataEntity[] Input { get; set; } public Expression[] LookupCondition { get; set; } } [0] [1] [2] [3] [4] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Name = "Zorg" Site = "Fred" Name = "Thud" Site = "Noob" Name = "Shme" Site = "Spqr" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Parallel Arrays • Arrays are of the same size • IActionDefinition.SupportsBulk • false }> Size = 1 • true }> Size ~• 1
165. Example: Bulk Insert in PostgreSQL INSERT INTO Organization (Id, Name) VALUES (1, "Aquiva Labs"), (2, "ACME, Inc.");
166. Example: Bulk Create in Jira API POST https://aquiva.attlasian.net/rest/api/v3/issue/bulk Accept: application/json Content-Type: application/json { "issueUpdates": [ { "fields": {} }, { "fields": {} }, ] }
167. What should I do with the OperationInput? 1. Translate 2. Execute 3. Return result [0] [1] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA"
168. [0] [1] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Success true false Objects
 Affected 0 or 1 0 Output Id = "123"
 Name = "Foo" Site = "Bar" null ErrorInfo null Message
 StackTrace Operation Result • Size of OperationResult should be the same as size of OperationInput • Parallel arrays again OperationInput OperationResult
169. [0] [1] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Success true false Objects
 Affected 0 or 1 0 Output Id = "123"
 Name = "Foo" Site = "Bar" null ErrorInfo null Message
 StackTrace Success • Whether or not the operation was a success
170. [0] [1] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Success true false Objects
 Affected 1 0 Output Id = "123"
 Name = "Foo" Site = "Bar" null ErrorInfo null Message
 StackTrace ObjectAffected • Number of objects affected by the operation
171. [0] [1] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Success true false Objects
 Affected 0 or 1 0 Output Id = "123"
 Name = "Foo" Site = "Bar" null ErrorInfo null Message
 StackTrace Output • If success, put in the Output all the properties from the Input • If success, put there auto- generated properties like ID
172. [0] [1] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Success true false Objects
 Affected 0 or 1 0 Output Id = "123"
 Name = "Foo" Site = "Bar" null ErrorInfo null Message
 StackTrace ErrorInfo • Put the detail about the error: • Description }> Message • Detail }> StackTrace • Number }> HRESULT
173. [0] [1] Input Name = "Foo" Site = "Bar" Name = "Baz" Site = "Qux" Lookup
 Condition Status = "OK"
 AND State = "MA" Status = "OK"
 AND State = "MA" Success true false Objects
 Affected 0 or 1 0 Output Id = "123"
 Name = "Foo" Site = "Bar" null ErrorInfo null Message
 StackTrace ErrorInfo • To have a correct Execution History you should correctly fill OperationResult
174. Error Types • Fatal }> retry then throw • connectivity errors • Row }> ErrorInfo • data errors
175. // Raw sketch of operation execution logic public OperationResult ExecuteOperation(OperationInput input) { var objectDefinitionName = input.Input[0].ObjectDefinitionFullName; switch (input.Name) { case "Create" when objectDefinitionName uv "Organization": return CreateOrganization(input); // Other Operations dispatched default: throw new InvalidExecuteOperationException("Operation is not supported") } }
176. // Raw sketch of non-bulk version of operation public OperationResult CreateOrganization(OperationInput input) { var result = new OperationResult(size: 1); try { var statement = _operationTranslator.Translate(input, 0); var nativeResult = _client.Execute(statement); result.ObjectsAffected[0] = 1; result.Output[0] = nativeResult?.ToDataEntity(); result.Success[0] = true; } catch (Exception ex) when (!(ex is FatalErrorException)) { result.Success[0] = false; result.ErrorInfo[0] = new ErrorResult { Description = ex.Message, Detail = ex.StackTrace }; } return result; }
177. What Else? • And that's only for the Integration Services…
178. Resources • Accompanying Source Code for this session • https://github.com/valmaev/scribe-connector-demo • Developer Success Portal • https://dev.scribesoft.com/en/sdk/cdk/basics/ developing_connectors.htm • Scribe Platform API Source Code • https://gitlab.com/ScribeSoftware/sample-scribe- platform-api-connector • Scribe Platform API Reference Architecture • https://dev.scribesoft.com/en/ref_app/api_conn/ scribe_api_connector.htm
179. Thank you! valmaev@aquivalabs.com @vladimir_almaev