Evaluation of Web Processing Service Frameworks

In this presentation, I illustrate, and discuss initial results from a quantitative analysis of the performance of WPS servers. To do so, two test scenarios were used to measure response time, response size, throughput, and failure rate of five WPS servers including 52North, Deegree, GeoServer, PyWPS, and Zoo. I also assess each WPS server in terms of qualitative metrics such as software architecture, perceived ease of use, flexibility of deployment, and quality of documentation. A case study addressing accessibility assessment is used to evaluate the relative advantages and disadvantages of each implementation, and point to challenges experienced while working with these WPS servers.

Transcript

1. EVALUATION OF WPS FRAMEWORKS! Ebrahim Poorazizi! Andrew Hunter! ! Geomatics Eng., University of Calgary! gisciencegroup.ucalgary.ca!
2. OUTLINE! §  Introduction ! §  Web Processing Service! §  WPS Servers! §  Methodology! §  Experimental Results! §  Lessons Learned! §  Discussion and Conclusions! 2!
3. INTRODUCTION! 3! Created by tagxedo.com!
4. INTRODUCTION! 4! How FAST is it?! Is it EASY-TO-USE?! Images courtesy: © funcom.com, iconbug.com, tagxedo.com!
5. INTRODUCTION! 5! Spatial Web Services Evaluations:! §  Web Map Service {Horák, J. et al. (2009), Kalbere, P. (2010), FOSS4G Shootout}! §  Sensor Observation Service {Tamayo, A. et al. (2011), Poorazizi, M.E. et al. (2012)}! §  Web Processing Service {Scholten, M. et al. (2006), Michaelis, C. and Ames, D. (2009), FOSS4G Shootout}! What have been done so far:! §  Focused on influential performance issues, the WPS protocol and its specification, and compliance and interoperability testing!
6. INTRODUCTION! 6! Images courtesy: © iconfinder.com! What needs to be done:! §  Evaluating WPS functionality and performance! Why it should be done:! §  To identify the strengths and weaknesses of each system! §  To improve WPS servers to meet both application user and developer requirements!
7. WEB PROCESSING SERVICE! 7! GET POST SOAP GetCapabilities DescribeProcess Execute  A web service interface to standardize the way that (spatial) algorithms are made available on the Internet!
8. WPS SERVERS! 8! ! 52°North! Deegree! GeoServer! PyWPS! Zoo! Organization! 52°North GmbH! OSGeo! OSGeo! Intevation GmbH! OSGeo! Development Platform! Java! Java! Java! Python! C/C++! License! GNU GPL v. 2! LPGL! GNU GPL v. 2! GNU GPL v. 2! MIT/X-11 style! Supported Libraries! JTS! GeoTools! SEXTANTE! R! GRASS! ArcGIS! SEXTANTE! JTS! GeoTools! GRASS! GDAL! R! GRASS! GEOS! GDAL! N a t i v e l y S u p p o r t e d Development Languages! Java! Java! Java! Python! C/C++! Fortran! Java! Python! PHP! Perl! JavaScript! Service ! Servlet! Servlet! Servlet! CGI! CGI! DCP Request! GET/POST/SOAP! GET/POST/SOAP! GET/POST! GET/POST/SOAP! GET/POST!
9. METHODOLOGY – CASE STUDY! 9! Images courtesy: © walkonomics.com! WalkYourPlace Transit Model à http://webmapping.ucalgary.ca/WPSClient/ ! §  Determine accessible walkable area (walkshed) using pedestrian and transit infrastructure! §  Assess walkshed using a number of quality of life and access indicators such as local crime, or shopping and recreation facilities within that area!
10. METHODOLOGY – ARCHITECTURE ! 10! Design! Implement! Release! Test! §  A bottom-up approach! §  PostGIS & Python! §  Wrapping & exposing as WPS! §  Evaluating from quantitative & qualitative views!
11. METHODOLOGY – ARCHITECTURE ! 11!
12. METHODOLOGY – ARCHITECTURE ! 12!
13. METHODOLOGY – ARCHITECTURE ! 13!
14. StartPoint=51.057,-114.066! StartTime=17:43:00! WalkingTimePeriod=15! WalkingSpeed=1.38! BusWaitingTime=10! BusRideTime=5! DistanceDecayFunction=False! METHODOLOGY – ARCHITECTURE ! 14! Walkshed WPS! WPS Input! WPS Output!
15. METHODOLOGY – ARCHITECTURE ! 15!
16. METHODOLOGY – ARCHITECTURE ! 16! Transit WPS! WPS Input! WPS Output! StartTime=17:43:00! WalkingTimePeriod=15! WalkingSpeed=1.38! BusWaitingTime=10! BusRideTime=5!
17. METHODOLOGY – ARCHITECTURE ! 17!
18. METHODOLOGY – ARCHITECTURE ! 18! Union WPS! WPS Input! WPS Output!
19. METHODOLOGY – ARCHITECTURE ! 19!
20. METHODOLOGY – ARCHITECTURE ! 20! POI WPS! WPS Input! WPS Output!
21. METHODOLOGY – ARCHITECTURE ! 21!
22. METHODOLOGY – ARCHITECTURE ! 22! Crime WPS! WPS Input! WPS Output!
23. METHODOLOGY – ARCHITECTURE ! 23!
24. METHODOLOGY – ARCHITECTURE ! 24! Aggregation WPS! WPS Input! WPS Output! StartPoint=51.057,-114.066! WalkingTimePeriod=15! DistanceDecayFunction=False!
25. SCENARIO A! ü  To assess how fast each server responds! ü  45 Execute requests, random WPS input parameters:! §  Walking Start Point! §  Walking Start Time: 5 a.m. – 12 p.m. ! §  Walking Time Period (WTP): 5 min – 20 min! §  Walking Speed: 3 km/h – 6 km/h! §  Bus Waiting Time (BWT): 0 – WTP! §  Bus Ride Time: 0 – (WTP – BWT)! §  Distance Decay Function: True/False! ! METHODOLOGY – TEST SCENARIO! 25!
26. SCENARIO B! ü  To assess the effect of increased load on each server! §  Number of concurrent requests: 2n , n = {0,…,7}! §  30 replications! §  Random WPS input parameters, like SCENARIO A! METHODOLOGY – TEST SCENARIO! 26!
27. METHODOLOGY – TEST ENVIRONMENT! §  Each WPS server was installed on a separate Virtual Machine (VM)! §  The same machine used to run the VMs and the tests! §  JMeter was used to perform the tests! 27! Hardware! Dell OptiPlex 990 (Host)! VMware (VM)! CPU! Intel Core i5 3.1 GHz! 4 Cores of 8! RAM! 8 GB! 4 GB! HDD! 500 GB! 40 GB! OS! Windows 7 Professional (64-bit)! Ubuntu 12.04 LTS (64-bit)!
28. SCENARIO A! §  Response Time: time taken for a service call to return all response bytes! §  Response Size: the quantity of data exchanged between client and server! ! EXPERIMENTAL RESULTS! 28! WPS Server! Response Time (sec.)! Response Size (kB)! 52°North! 2.784 ± 1.269! 2.448 ± 0.267! Deegree! 2.499 ± 1.259! 2.505 ± 0.267! GeoServer! 2.753 ± 1.255! 2.301 ± 0.267! PyWPS! 3.995 ± 1.661! 2.686 ± 0.269! Zoo! 2.999 ± 1.313! 2.479 ± 0.269!
29. EXPERIMENTAL RESULTS! 29! F(4,220)=1.071, p=0.372 ! F(4,220)=0.739, p=0.566 !
30. SCENARIO B! §  Response Time! §  Server Failure! §  Server Throughput! EXPERIMENTAL RESULTS! 30!
31. EXPERIMENTAL RESULTS! 31! Response Time!
32. EXPERIMENTAL RESULTS! 32! Server Failure Rate!
33. EXPERIMENTAL RESULTS! 33! Server Throughput!
34. LESSONS LEARNED! 34! 52°North! Deegree! GeoServer! PyWPS! Zoo! Installation*! Easy! Easy! Easy! Difficult! Difficult! Create new processes and Configuration* ! Easy! Medium! Easy! Difficult! Easy! Native Development Languages! One ! One ! One ! One ! Many! Quality of Documentation**! Great! Good! Great! Good! Good! Community Support! •  Many Avenues! •  Active Support! •  Many Avenues! •  Active Support! •  Many Avenues! •  Active Support! •  Few Avenues! •  Little Support! •  Many Avenues! •  Active Support! * Ranking ranges: Easy; Medium; Difficult! ** Ranking ranges: Weak; Good; Great!
35. SCENARIO A! §  Deegree returns the response package most rapidly! §  But, based on a One Way ANOVA test, there is no significant difference in response time between the WPS servers tested nor data volume returned! SCENARIO B! §  Deegree and GeoServer performed similarly; Deegree tending to perform better under high loads! §  52°North had difficulty when processing more than 16 concurrent requests, but performed more effectively under low loads! § From a qualitative perspective, compared to other WPS servers, 52°North and GeoServer seem to be the best choices! DISCUSSION! 35!
36. CONCLUSIONS! 36! §  When selecting an appropriate WPS server, it is important to consider both quantitative and qualitative metrics! §  The importance of each metric can be weighted based on different application requirements! §  A recommendation: start with an evaluation process with some basic questions:! ü  Who is the user of the system, end-user or application developer?! ü  How complex are the geoprocessing services?! ü  How should the system function, synchronous or asynchronous?! ü  What is the architecture used to design the processing workflow?! ü  What is the expected number of users?!
37. THANK YOU! ANY QUESTION?! Ebrahim Poorazizi! mepooraz@ucalgary.ca! ebrahim.planyourplace.ca! Andrew Hunter! ahunter@ucalgary.ca! gisciencegroup.ucalgary.ca!