Building Reliable Systems from unreliable components

Patterns for dealing with fragility

Transcript

1. Building reliable systems from unreliable components Arnon Rotem-Gal-Oz
2. Arnon Rotem-Gal-Oz
3. What’s in a 9
4. 0.99 reliability We have a nice little legacy business component
5. 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 And we move it to SOA
6. Failsafe hardware Status Technologies FT Server
7. Or try to detect failure , handle it and minimize its effect on the business service © Rosendahl
8. BEFORE WE BEGIN
9. SOA Service describes End Point Exposes Messages Sends/Receives Contracts Binds to Service Consumer implements Policy governed by Sends/Receives Adheres to Component Relation Key Understands Serves
10. SOA is derived from other styles Pipes and Filters Client Server Distributed Agents Layered System Stateless Comm. SOA
11. SOA vs. REST Pipes and Filters Client Server Uniform Interface Virtual Machine Distributed Agents Layered System Replicated Repository Code On Demand Stateless Comm. Cacheable REST SOA
13. 3G Video Calls MMS Dedicated Client Mobile Integration Applications Acquisition Interactions branding System Ad Management Interactions Reference Data Links Resources Monitoring Targeted Advertizing Campaign Mgmt. Usage Datmart Reporting Billing Data mining & Statistics Reports Link Managment Publishing tools integration Interaction Designer Web Front-end Data Interfaces 3rd parties
14. Load balancer Web Server (IIS/Apache App Server App Server App Server App Server App Server App Server App Server Web Server (IIS/Apache MMS Gateway 3G Gateway 3G Gateway Load balancer Firewall Firewall BI & Reporting DB Links Datamart usage DB References Links Registeration Sync. Server DMZ Operational Backend Advertizing clients Firewall Web Server (IIS/Apache NMS Paper Editor Smart phones DMZ Camera Phones Admin Console
15. CHALLENGE – SERVICE AVAILABILITY
16. What’s the effect of a failure - Server E1 line = 30 concurrent video calls Call Flow Service
17. request reaction Edge Service Instance Dispatcher Distribute End point Service Business logic Service Instance
18. What’s the effect of a failure - Channel E1 line = 30 concurrent video calls Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel Call Flow Channel
19. Service Instance with NLB Service Instance NLB Driver Cluster Host NIC Driver TCP/IP Windows Kernel NIC NLB Driver Cluster Host NIC Driver TCP/IP Windows Kernel NIC Virtual IP : 1.1.1.1 Real IP : 1.1.1.2 Real IP : 1.1.1.3 Service Instance Edge Windows Host NIC Driver TCP/IP Windows Kernel NIC Real IP : 1.1.1.4
20. Virtual Endpoint
21. Request/Reply Service EndPoint Synchronous processing 1. Request 2. 3. Reply Service Consumer
22. Things look Cool & Simple ™ Session negations Image extraction identification Translation to links Render resutls 3G Call
23. IVP (RV) 3G GW (RV) 3G VAS (Cestel) WS Resource Manager SIP Listner RTP Image Extractor Alg. Engine Dispatcher WebConnector 3G Builder (Cestel) WebRenderer Turn out Complicated & Ugly ™
24. Relation Key SOA Component Pattern Component Concern/attribute Edge pipeline Perform Task EndPoint Service Request Reaction EndPoint pipeline Perform Task EndPoint pipeline Perform Task EndPoint Queue Request 2 Request 1 Parallel Pipelines
25. Inversion of Communications
26. Consumer view var sendMmsEvent = new SendMmsEvent() { FromNumber = simpleMessageDetails.DialedNumber, Subject = mmsContents.Subject, ToNumber = simpleMessageDetails.Sender, ImageExtension = mmsContents.ImageExtension, ImageAsByteArray = mmsContents.Image, TextAsByteArray = mmsContents.Text }; eventBroker.RaiseEvent(sendMmsEvent); http://www.flickr.com/photos/crimson_wolf/2851737125/sizes/l/
27. Service view public interface ImPostOffice : ImContract, IHandleSendCoupon, IHandleSendSms,IHandleStatus,IHandleAdminStatus,IHandleWapLink, IHandleSendMms { } [ServiceContract] public interface IHandleSendMms { [OperationContract] int SendMms(SendMmsEvent eventOccured); } [ServiceContract] [DataContract] public class SendMmsEvent : ImEvent { /// <summary> /// end user's number. should be in international format: +[country-code]number. Example: +491737692260 /// </summary> [DataMember] public string ToNumber { get; set; } /// <summary> /// service's number, usually a short-code. Example: 84343 /// </summary> [DataMember] public string FromNumber { get; set; }
28. Edge translates external structures to internal ones public int SendMms(SendMmsEvent eventOccured) { var eventContext = eventOccured.ToString(); if (log.IsDebugEnabled) log.Debug("inside 'SendMms', event context = [" + eventContext + "]"); var fromNumber = eventOccured.FromNumber; var sender = mmsSenderFactory.Get(fromNumber); if (null == sender) { if (log.IsWarnEnabled) log.Warn("cannot get mms sender derived from '" + (fromNumber ?? "null") + "'"); return 0; } IMmsSubmitResponse response; try { var mmsMessageDetails = new MmsMessageDetails(eventOccured.ToNumber, eventOccured.TextAsByteArray, eventOccured.ImageAsByteArray, eventOccured.ImageExtension, eventOccured.Subject); response = sender.Submit(mmsMessageDetails); } catch (Exception ex) { log.Error("cannot send mms message, context = [" + eventContext + "]", ex); return 0; } if (log.IsInfoEnabled) { var responseMessage = (null == response) ? "null" : response.ToString(); log.Info("sent mms with event context = [" + eventContext + "], response = [" + responseMessage + "]"); }
29. Sagas tie instances together for conversations
30. SIP Image Extractor Identification Call Recovery
31. Alternative : Orchestration request Workflow Engine Workflow instance Manage Process route request Host Workflows Schedule Orchestration platform Service Service reaction Auxiliary tools Coordinator Protocol Offline designer monitor
32. Be Wary of Nano-Services
33. CHALLENGE - MANAGEMENT
34. Edge Watchdog Edge Report EndPoint Service Request Monitor EndPoint Watchdog Agent Monitor Heal Log Reports Monitor Monitor Monitor Blogjecting Watchdog
35. Blogjects concept is about collaborating objects
36. WatchDog Service A Service B WDWatcher
37. SIP Image Extractor Identification Call Recovery WatchDog
39. Status Service Monitor Edge/Service In Commands Metrics collection Policy governance Security monitoring Fault Monitoring Reporting & Dashboarding Control Edge/Service Status Monitor Act Collect Notify Service Monitor
40. 3 2 1 root / Sessions/ Abcde/ Efgh/ Resources/ Dispatchers/ Xyz/ RESTful resource management
41. http://devrig:52141/RM/Sessions/abc/ • ATOMPUB – Session details • URI (ID) • State (start/end/status etc.) • Resources – Knows status – URI for the Resource representation on the RM – URI for the Resource itself
42. Keep the BIT http://www.flickr.com/photos/37643027@N00/2050024263/sizes/o/
43. SIP Image Extractor Identification X Call Recovery WatchDog WatchDog Liveliness Monitor 3G Call
44. CHALLENGE – MULTI-TENNANCY (LIES, I TELL YOU, ALL LIES)
45. Same event different subscribers Call Flow Player (interaction Renderer) Call Flow Bridge to 3rd Party Play Movie Event
46. Routing [ServiceContract] [Participate("3G")] public interface ImPlayer : ImContract, IHandleCallStarted, IHandleC IHandlePlayMovie,IHandleCallAborted { } [ServiceContract] [Participate("3GPartner")] public interface ImXsightsGateWay : ImContract, IHandleCallAborted, IHandlePlayMovie, IHandleReadyForSearch, IHandleSearchStarted, IHandleJoinThirdParty { }
47. Initiator A Initiator B Participant B Capacity : 1 Raise a saga initiating event Now what ?! Participant A Capacity : 2
48. Reservation Pattern
49. Event Broker Service Host Resource Allocator Business Logic Control Edge Service Host Service Host Business Logic Event Broker Service Host Resource Allocator Control Edge Service Instance #1 Service Host Business Logic Event Broker Service Host Resource Allocator Control Edge Service Instance #2 Service Host Business Logic Event Broker Service Host Resource Allocator Control Edge Reservation
50. Good old 2PC to the rescue
51. Takeaways • Things break • Decouple • Fail Fast • Monitor & Detect • Compensate • Throw state unto others
52. Arnon Rotem-Gal-Oz @arnonrgo http://arnon.me http://www.cloudvalue.com