EVALUATION OF WPS FRAMEWORKS! Ebrahim Poorazizi! Andrew Hunter! ! Geomatics Eng., University of Calgary! gisciencegroup.ucalgary.ca! 

OUTLINE! §  Introduction ! §  Web Processing Service! §  WPS Servers! §  Methodology! §  Experimental Results! §  Lessons Learned! §  Discussion and Conclusions! 2! 

INTRODUCTION! 3! Created by tagxedo.com! 

INTRODUCTION! 4! How FAST is it?! Is it EASY-TO-USE?! Images courtesy: © funcom.com, iconbug.com, tagxedo.com! 

INTRODUCTION! 5! Spatial Web Services Evaluations:! §  Web Map Service {Horák, J. et al. (2009), Kalbere, P. (2010), FOSS4G Shootout}! §  Sensor Observation Service {Tamayo, A. et al. (2011), Poorazizi, M.E. et al. (2012)}! §  Web Processing Service {Scholten, M. et al. (2006), Michaelis, C. and Ames, D. (2009), FOSS4G Shootout}! What have been done so far:! §  Focused on influential performance issues, the WPS protocol and its specification, and compliance and interoperability testing! 

INTRODUCTION! 6! Images courtesy: © iconfinder.com! What needs to be done:! §  Evaluating WPS functionality and performance! Why it should be done:! §  To identify the strengths and weaknesses of each system! §  To improve WPS servers to meet both application user and developer requirements! 

WEB PROCESSING SERVICE! 7! GET POST SOAP GetCapabilities DescribeProcess Execute  A web service interface to standardize the way that (spatial) algorithms are made available on the Internet! 

WPS SERVERS! 8! ! 52°North! Deegree! GeoServer! PyWPS! Zoo! Organization! 52°North GmbH! OSGeo! OSGeo! Intevation GmbH! OSGeo! Development Platform! Java! Java! Java! Python! C/C++! License! GNU GPL v. 2! LPGL! GNU GPL v. 2! GNU GPL v. 2! MIT/X-11 style! Supported Libraries! JTS! GeoTools! SEXTANTE! R! GRASS! ArcGIS! SEXTANTE! JTS! GeoTools! GRASS! GDAL! R! GRASS! GEOS! GDAL! N a t i v e l y S u p p o r t e d Development Languages! Java! Java! Java! Python! C/C++! Fortran! Java! Python! PHP! Perl! JavaScript! Service ! Servlet! Servlet! Servlet! CGI! CGI! DCP Request! GET/POST/SOAP! GET/POST/SOAP! GET/POST! GET/POST/SOAP! GET/POST! 

METHODOLOGY – CASE STUDY! 9! Images courtesy: © walkonomics.com! WalkYourPlace Transit Model à http://webmapping.ucalgary.ca/WPSClient/ ! §  Determine accessible walkable area (walkshed) using pedestrian and transit infrastructure! §  Assess walkshed using a number of quality of life and access indicators such as local crime, or shopping and recreation facilities within that area! 

METHODOLOGY – ARCHITECTURE ! 10! Design! Implement! Release! Test! §  A bottom-up approach! §  PostGIS & Python! §  Wrapping & exposing as WPS! §  Evaluating from quantitative & qualitative views! 

METHODOLOGY – ARCHITECTURE ! 11! 

METHODOLOGY – ARCHITECTURE ! 12! 

METHODOLOGY – ARCHITECTURE ! 13! 

StartPoint=51.057,-114.066! StartTime=17:43:00! WalkingTimePeriod=15! WalkingSpeed=1.38! BusWaitingTime=10! BusRideTime=5! DistanceDecayFunction=False! METHODOLOGY – ARCHITECTURE ! 14! Walkshed WPS! WPS Input! WPS Output! 

METHODOLOGY – ARCHITECTURE ! 15! 

METHODOLOGY – ARCHITECTURE ! 16! Transit WPS! WPS Input! WPS Output! StartTime=17:43:00! WalkingTimePeriod=15! WalkingSpeed=1.38! BusWaitingTime=10! BusRideTime=5! 

METHODOLOGY – ARCHITECTURE ! 17! 

METHODOLOGY – ARCHITECTURE ! 18! Union WPS! WPS Input! WPS Output! 

METHODOLOGY – ARCHITECTURE ! 19! 

METHODOLOGY – ARCHITECTURE ! 20! POI WPS! WPS Input! WPS Output! 

METHODOLOGY – ARCHITECTURE ! 21! 

METHODOLOGY – ARCHITECTURE ! 22! Crime WPS! WPS Input! WPS Output! 

METHODOLOGY – ARCHITECTURE ! 23! 

METHODOLOGY – ARCHITECTURE ! 24! Aggregation WPS! WPS Input! WPS Output! StartPoint=51.057,-114.066! WalkingTimePeriod=15! DistanceDecayFunction=False! 

SCENARIO A! ü  To assess how fast each server responds! ü  45 Execute requests, random WPS input parameters:! §  Walking Start Point! §  Walking Start Time: 5 a.m. – 12 p.m. ! §  Walking Time Period (WTP): 5 min – 20 min! §  Walking Speed: 3 km/h – 6 km/h! §  Bus Waiting Time (BWT): 0 – WTP! §  Bus Ride Time: 0 – (WTP – BWT)! §  Distance Decay Function: True/False! ! METHODOLOGY – TEST SCENARIO! 25! 

SCENARIO B! ü  To assess the effect of increased load on each server! §  Number of concurrent requests: 2n , n = {0,…,7}! §  30 replications! §  Random WPS input parameters, like SCENARIO A! METHODOLOGY – TEST SCENARIO! 26! 

METHODOLOGY – TEST ENVIRONMENT! §  Each WPS server was installed on a separate Virtual Machine (VM)! §  The same machine used to run the VMs and the tests! §  JMeter was used to perform the tests! 27! Hardware! Dell OptiPlex 990 (Host)! VMware (VM)! CPU! Intel Core i5 3.1 GHz! 4 Cores of 8! RAM! 8 GB! 4 GB! HDD! 500 GB! 40 GB! OS! Windows 7 Professional (64-bit)! Ubuntu 12.04 LTS (64-bit)! 

SCENARIO A! §  Response Time: time taken for a service call to return all response bytes! §  Response Size: the quantity of data exchanged between client and server! ! EXPERIMENTAL RESULTS! 28! WPS Server! Response Time (sec.)! Response Size (kB)! 52°North! 2.784 ± 1.269! 2.448 ± 0.267! Deegree! 2.499 ± 1.259! 2.505 ± 0.267! GeoServer! 2.753 ± 1.255! 2.301 ± 0.267! PyWPS! 3.995 ± 1.661! 2.686 ± 0.269! Zoo! 2.999 ± 1.313! 2.479 ± 0.269! 

EXPERIMENTAL RESULTS! 29! F(4,220)=1.071, p=0.372 ! F(4,220)=0.739, p=0.566 ! 

SCENARIO B! §  Response Time! §  Server Failure! §  Server Throughput! EXPERIMENTAL RESULTS! 30! 

EXPERIMENTAL RESULTS! 31! Response Time! 

EXPERIMENTAL RESULTS! 32! Server Failure Rate! 

EXPERIMENTAL RESULTS! 33! Server Throughput! 

LESSONS LEARNED! 34! 52°North! Deegree! GeoServer! PyWPS! Zoo! Installation*! Easy! Easy! Easy! Difficult! Difficult! Create new processes and Configuration* ! Easy! Medium! Easy! Difficult! Easy! Native Development Languages! One ! One ! One ! One ! Many! Quality of Documentation**! Great! Good! Great! Good! Good! Community Support! •  Many Avenues! •  Active Support! •  Many Avenues! •  Active Support! •  Many Avenues! •  Active Support! •  Few Avenues! •  Little Support! •  Many Avenues! •  Active Support! * Ranking ranges: Easy; Medium; Difficult! ** Ranking ranges: Weak; Good; Great! 

SCENARIO A! §  Deegree returns the response package most rapidly! §  But, based on a One Way ANOVA test, there is no significant difference in response time between the WPS servers tested nor data volume returned! SCENARIO B! §  Deegree and GeoServer performed similarly; Deegree tending to perform better under high loads! §  52°North had difficulty when processing more than 16 concurrent requests, but performed more effectively under low loads! § From a qualitative perspective, compared to other WPS servers, 52°North and GeoServer seem to be the best choices! DISCUSSION! 35! 

CONCLUSIONS! 36! §  When selecting an appropriate WPS server, it is important to consider both quantitative and qualitative metrics! §  The importance of each metric can be weighted based on different application requirements! §  A recommendation: start with an evaluation process with some basic questions:! ü  Who is the user of the system, end-user or application developer?! ü  How complex are the geoprocessing services?! ü  How should the system function, synchronous or asynchronous?! ü  What is the architecture used to design the processing workflow?! ü  What is the expected number of users?! 

THANK YOU! ANY QUESTION?! Ebrahim Poorazizi! [email protected]! ebrahim.planyourplace.ca! Andrew Hunter! [email protected]! gisciencegroup.ucalgary.ca!