Event-Driven Services

Transcript

1. IMPROVING THE PERFORMANCE OF THE ONELAB PORTAL QUAN ZHOU 12.SEPT.2016
2. INTRODUCTION WHAT IS ONELAB PORTAL?
3. ONELAB SLICE-BASED FRAMEWORK SFA Framework Testbeds Testbeds Testbeds ▸ Defines the minimal set of interfaces and data types that permit a federation of SFA component from different testbeds to communicate and operate.
4. ONELAB SLICE-BASED FRAMEWORK Testbeds Testbeds Testbeds API API API ▸ Defines the minimal set of interfaces and data types that permit a federation of SFA component from different testbeds to communicate and operate. SFA Framework
5. ▸ SFA components: ▸ SFA Data: ONELAB SLICE-BASED FRAMEWORK Testbed
6. ▸ SFA components: Authority ▸ SFA Data: ONELAB SLICE-BASED FRAMEWORK Testbed
7. ▸ SFA components: Authority, Slice ▸ SFA Data: ONELAB SLICE-BASED FRAMEWORK Testbed
8. ▸ SFA components: Authority, Slice, User ▸ SFA Data: ONELAB SLICE-BASED FRAMEWORK Testbed
9. ▸ SFA components: Authority, Slice, User ▸ SFA Data: Rspec, ONELAB SLICE-BASED FRAMEWORK Testbed
10. ▸ SFA components: Authority, Slice, User ▸ SFA Data: Rspec, Credential ONELAB SLICE-BASED FRAMEWORK Testbed
11. ▸ SFA components: Authority, Slice, User ▸ SFA Data: Rspec, Credential, Tickets ONELAB SLICE-BASED FRAMEWORK Testbed
12. ONELAB SLICE-BASED FRAMEWORK ▸ SFA Registry: manages all the records inside a testbed. ▸ SFA Aggregate Manager: manages the experiments and resources. Testbed
13. ONELAB MYSLICE Testbed Testbed Testbed API API API
14. ▸ A web-based open platform which allows users(experimenters) to discover, browser, and reserve the resources among heterogeneous testbeds and conduct experiments. ▸ MySlice, acts as a thin waist layer communicating with the Slice-based Federation(SFA) framework, and hiding the complexity. TEXT WHAT IS ONELAB PORTAL?
15. MYSLICE SFA Framework Testbed Testbed Testbed MySlice ONELAB HTTP
16. ONELAB MYSLICE SFA Framework Testbed Testbed Testbed MySlice Web interface Django jQuery HTTP
17. MYSLICE SFA Framework Testbed Testbed Testbed MySlice ONELAB Web interface Django jQuery HTTP
18. MYSLICE SFA Framework Testbed Testbed Testbed MySlice ONELAB Manifold API Web interface Django jQuery RPC Client HTTP
19. MYSLICE SFA Framework Testbed Testbed Testbed MySlice Django jQuery ONELAB Manifold API Web interface RPC Client HTTP
20. PROBLEMS BROWSING HETEROGENEOUS RESOURCES IS TIME-CONSUMING!
21. 1. BROWSER RESOURCES Testbed Testbed Testbed PROBLEMS GET RESOURCES OneLab
22. Testbed Testbed Testbed PROBLEMS GET RESOURCES RESOURCES FILES OneLab 1. BROWSER RESOURCES
23. Testbed Testbed Testbed PROBLEMS GET RESOURCES RESOURCES FILES GET RESOURCES OneLab 1. BROWSER RESOURCES
24. Testbed Testbed Testbed PROBLEMS GET RESOURCES RESOURCES FILES GET RESOURCES RESOURCES FILES OneLab 1. BROWSER RESOURCES
25. Testbed Testbed Testbed PROBLEMS GET RESOURCES RESOURCES FILES GET RESOURCES RESOURCES FILES GET RESOURCES OneLab 1. BROWSER RESOURCES
26. Testbed Testbed Testbed PROBLEMS GET RESOURCES RESOURCES FILES GET RESOURCES RESOURCES FILES GET RESOURCES RESOURCES FILES OneLab 1. BROWSER RESOURCES
27. PROBLEMS EACH USER REQUEST IS BLOCKING & SYNCHROUS!
28. Testbed PROBLEMS RESERVE SOME RESOURCE OneLab 2. SYNCHRONOUS & BLOCKING User A
29. Testbed PROBLEMS RESERVE SOME RESOURCE OneLab 2. SYNCHRONOUS & BLOCKING User A
30. Testbed PROBLEMS RESERVE SOME RESOURCE OneLab 2. SYNCHRONOUS & BLOCKING User A
31. Testbed Testbed PROBLEMS RESERVE SOME RESOURCE OneLab 2. SYNCHRONOUS & BLOCKING User A User B UPDATE THE SLICE
32. Testbed Testbed PROBLEMS RESERVE SOME RESOURCE OneLab 2. SYNCHRONOUS & BLOCKING User A User B UPDATE THE SLICE
33. Testbed Testbed PROBLEMS RESERVE SOME RESOURCE OneLab 2. SYNCHRONOUS & BLOCKING User A User B UPDATE THE SLICE RESPONSE
34. Testbed Testbed PROBLEMS RESERVE SOME RESOURCE OneLab 2. SYNCHRONOUS & BLOCKING User A User B UPDATE THE SLICE RESPONSE
35. PROBLEMS NOT FAULT-TORERLANT & NO FEEDBACK!
36. PROBLEMS 3. FAULTS AND ERRORS Request Status: START Web Business Logic XML-RPC Communication User A
37. PROBLEMS 3. FAULTS AND ERRORS Request Status: PENDING Web RESERVE RESOURCE Business Logic XML-RPC Communication User A
38. PROBLEMS 3. FAULTS AND ERRORS Request Status: ERROR Web RESERVE RESOURCE Business Logic XML-RPC Communication ERROR User A OUR WEB SERVER FAILS!
39. PROBLEMS 3. FAULTS AND ERRORS Request Status: START Web Business Logic XML-RPC Communication User A
40. PROBLEMS 3. FAULTS AND ERRORS Request Status: PENDING Web RESERVE RESOURCE Manifold Query Business Logic XML-RPC Communication User A
41. PROBLEMS 3. FAULTS AND ERRORS User A Request Status: ERROR Web RESERVE RESOURCE Manifold Query Business Logic XML-RPC Communication ERROR ERROR MANIFOLD FAILS!
42. PROBLEMS 3. FAULTS AND ERRORS Request Status: START Web Business Logic XML-RPC Communication User A
43. Testbed PROBLEMS 3. FAULTS AND ERRORS Request Status: PENDING Web RESERVE RESOURCE Manifold Query API Call Business Logic XML-RPC Communication User A
44. Testbed PROBLEMS 3. FAULTS AND ERRORS User A ERROR Request Status: ERROR Web RESERVE RESOURCE Manifold Query API Call Business Logic XML-RPC Communication ERROR ERROR TESTBED GOES DOWN OR FAILS!
45. MYSLICE SFA Framework Testbed Testbed Testbed MySlice Django jQuery MYSLICE V2 Manifold API Web interface RPC Client HTTP
46. MYSLICE V2 MySliceV2 MYSLICE V2 HTTP SFA Framework Testbed Testbed Testbed Tornado REST API + Services & Workers MySlicelib API
47. FASTER & NON-BLOCKING API MYSLICE V2 MYSLICELIB API
48. MYSLICELIB API MYSLICE V2 SFA Client SFA Server XML-RPC Communication
49. MYSLICELIB API MYSLICE V2 SFA Client SFA Server Registry Client XML-RPC Communication Onelab Server Endpoint(url=“https://portal.onelab.eu:6080”,type="Reg") Generalisation
50. MYSLICELIB API MYSLICE V2 SFA Client SFA Server XML-RPC Communication
51. MYSLICELIB API MYSLICE V2 SFA Client SFA Server AM Client XML-RPC Communication Generalisation TestBeds Server Endpoint(url=“https://griffin.ipv6.lip6.fr:8001/RPC2”,type="AM")
52. MYSLICELIB API MYSLICE V2 s.endpoints = [ Endpoint(url=“https://portal.onelab.eu:6080”,type=“Reg"), Endpoint(url=“https://fuseco.fokus.fraunhofer.de/api/sfa/am/v3”,type=“AM”) , Endpoint(url=“https://194.199.16.164:12346”,type=“AM”,name="IoTLab" ) , Endpoint(url=“https://griffin.ipv6.lip6.fr:8001/RPC2”,type=“AM”,name="NITOS Paris") , ]
53. MYSLICELIB API MYSLICE V2 MySlicelib API Registry Client AM Client AM Client AM Client Onelab Server Testbed Testbed Testbed XML-RPC Communication Actions MYSLICE V2
54. MYSLICELIB API MYSLICE V2 MySlicelib API Registry Client AM Client AM Client AM Client Onelab Server Testbed Testbed Testbed XML-RPC Communication Actions MYSLICE V2 GetResources() Worker
55. MYSLICELIB API MYSLICE V2 MySlicelib API Registry Client AM Client AM Client AM Client Onelab Server Testbed Testbed Testbed XML-RPC Communication Actions MYSLICE V2 GetResources() GetResources() Worker
56. MYSLICELIB API MYSLICE V2 MySlicelib API Registry Client AM Client AM Client AM Client Onelab Server Testbed Testbed Testbed XML-RPC Communication Actions MYSLICE V2 GetResources() GetResources() GetResources() GetResources() GetResources() Worker
57. MYSLICELIB API MYSLICE V2 MySlicelib API Registry Client AM Client AM Client AM Client Onelab Server Testbed Testbed Testbed XML-RPC Communication Actions MYSLICE V2 GetResources() GetResources() GetResources() GetResources() Rspec Rspec Worker
58. MYSLICELIB API MYSLICE V2 MySlicelib API Registry Client AM Client AM Client AM Client Onelab Server Testbed Testbed Testbed XML-RPC Communication Actions MYSLICE V2 GetResources() GetResources() GetResources() GetResources() Rspec Rspec Rspec Worker
59. LOCAL CACHE & EVENT-DRIVEN SERVICES MYSLICE V2 RETHINKDB
60. MYSLICE V2 RETHINKDB ALL THE RECORDS Sync Worker Onelab Server Testbed Testbed Testbed MySlicelib API Sync()
61. MYSLICE V2 RETHINKDB ALL THE RECORDS Onelab Server Testbed Testbed Testbed Sync() MySlicelib API Sync Worker JSON RSpec
62. MYSLICE V2 RETHINKDB ALL THE RECORDS Onelab Server Testbed Testbed Testbed Sync() MySlicelib API Sync Worker JSON RSpec
63. MYSLICE V2 RETHINKDB ALL THE RECORDS GET Onelab Server Testbed Testbed Testbed Sync() Sync Worker MySlicelib API JSON RSpec
64. MYSLICE V2 RETHINKDB EVENT EVENT SOURCE EVENT CHANNEL EVENT SINKS Web Requests Changefeeds Queues & Threads User Services & Workers Authority Services & Workers Slices Services & Workers
65. MYSLICE V2 RETHINKDB EVENT EVENT SOURCE EVENT CHANNEL EVENT SINKS Web Requests Changefeeds Queues & Threads User Services & Workers Authority Services & Workers Slices Services & Workers EVENT id: <uuid> status: action: CREATE logs: <None> object: <User> NEW
66. MYSLICE V2 RETHINKDB EVENT EVENT SOURCE EVENT CHANNEL EVENT SINKS Changefeeds Queues & Threads User Services & Workers Authority Services & Workers Slices Services & Workers EVENT id: <uuid> status: action: CREATE logs: <None> object: <User> Web Requests NEW
67. MYSLICE V2 RETHINKDB EVENT EVENT SOURCE EVENT CHANNEL EVENT SINKS Changefeeds Queues & Threads User Services & Workers Authority Services & Workers Slices Services & Workers EVENT id: <uuid> status: action: CREATE logs: <None> object: <User> Web Requests PENDING
68. MYSLICE V2 RETHINKDB EVENT EVENT SOURCE EVENT CHANNEL EVENT SINKS Changefeeds Queues & Threads User Services & Workers Authority Services & Workers Slices Services & Workers EVENT id: <uuid> status: action: CREATE logs: <None> object: <User> UPDATE Web Requests SUCCESS
69. MYSLICE V2 RETHINKDB EVENT EVENT SOURCE EVENT CHANNEL EVENT SINKS Changefeeds Queues & Threads User Services & Workers Authority Services & Workers Slices Services & Workers EVENT id: <uuid> status: action: CREATE logs: <A set of logs> object: <User> Web Requests ERROR
70. EASY TO SCALE UP & HANDLE CONCURRENCY EASILY MYSLICE V2 RETHINKDB
71. MYSLICE V2 MYSLICE WEB FAST RESPONSE & REAL-TIME CONTENTS
72. Backend MYSLICE V2 HTTP Tornado REST API + Frontend ReactJS ALT Flux + MYSLICE WEB Event GET
73. Backend MYSLICE V2 HTTP Tornado REST API + Frontend ReactJS ALT Flux + MYSLICE WEB Event GET
74. Backend MYSLICE V2 HTTP Tornado REST API + Frontend ReactJS ALT Flux + MYSLICE WEB Event GET GET /api/v1/resources Restful API
75. Backend MYSLICE V2 HTTP Tornado REST API + Frontend ReactJS ALT Flux + MYSLICE WEB Event GET GET /api/v1/resources RESTful API JSON
76. MYSLICE V2 EVALUATIONS AVERAGE RESPONSE TIME of API REDUCES from 12s to 8s
77. EVALUATIONS MYSLICE V2
78. MYSLICE V2 EVALUATIONS MYSLICEv2 PERFORMS WELL UNDER HIGH CONCURRENCY!
79. EVALUATIONS MYSLICE V2 ONE USER FORTY USER SPAWNED EACH 2 SECS
80. MYSLICE V2 CONCLUSION & FUTURE WORK ▸ Through the design of new architecture and early test, we can expect new portal will performs better. ▸ Future work should consider thread-safety and asynchronous implementations both on client and server side.
81. QUESTIONS? THANKS