Earlier Work Large body of research in serverless benchmarking However:
Usually microbenchmarks
Often benchmark a single provider
Focus on function response time J. Scheuner and P. Leitner, “Function-as-a-service performance evaluation: A multivocal literature review,” Journal of Systems and Software (JSS), vol. 170, 2020. V. Yussupov, U. Breitenbücher, F. Leymann, and M. Wurster, “A systematic mapping study on engineering function-as-a-service platforms and tools,” in Proceedings of the 12th IEEE/ACM International Conference on Utility and Cloud Computing (UCC). ACM, 2019, pp. 229–240.
User API Gateway Response Time Asynchronous Trigger User Function1: Persist Image Bucket1: Images Function2: Generate Thumbnail Bucket2: Thumbnails API Gateway Response Time Synchronous Invocation End-to-end Latency
Fairness We identify 12 principles for fairly comparing cloud platforms Some central ones:
• Reuse implementations, fi x versions (e.g., OS, runtime) • Use the same workloads
• Strive for geographically close regions
• De fi ne a clear mapping of services (e.g., S3 -> Blob Storage) • Map resource types by cost, not by name • Avoid speci fi c premium features, especially if they are only available on a subset of providers
Example Provider Mapping storage is the most popular external service used by serverless applications [12]. The storage can trigger subscribed functions, e.g., when a new item is created or an existing item is modified. Table I TRIGGER TYPES AND SERVICE MAPPINGS FOR AWS AND AZURE Trigger AWS Service Azure Service HTTP API Gateway API Management Queue SQS Queue Storage Storage S3 Blob Storage Database DynamoDB⇤ CosmosDB Event SNS⇤ Event Grid Stream Kinesis⇤ Event Hubs Message EventBridge⇤ Service Bus Topic Timer CloudWatch Events⇤ Timer ⇤ Not implemented We implement these three important triggers for the two leading cloud providers AWS and Azure [12] and an additional five triggers for Azure. Database triggers react to events in a database such as insertion, deletion, or update. Event traces b the sam Section experie E. Imp We i SDK) t ing our (Figure instrum Insights be auto using P The op custom because dencies needed J. Scheuner, M. Bertilsson, O. Grönqvist, H. Tao, H. Lagergren, JP. Steghöfer, and P. Leitner, "TriggerBench: A Performance Benchmark for Serverless Function Triggers," 2022 IEEE International Conference on Cloud Engineering (IC2E), 2022, pp. 96-103, doi: 10.1109/IC2E55432.2022.00018.
ServiBench Tracing Service Application Package + Deployment Script Serverless Application Cloud Provider Benchmark Orchestrator Workload Profile Deploy Invoke Partial Traces Retrieve Correlated Traces Analyze J. Scheuner, S. Eismann, S. Talluri, E. van Eyk, C. Abad, P. Leitner, and A. Iosup, “Let’s trace it: Fine-grained serverless benchmarking using synchronous and asynchronous orchestrated applications,” doi:10.48550/ARXIV.2205.07696, 2022.
References and Further Reading Benchmarking function platforms: J. Scheuner, S. Eismann, S. Talluri, E. van Eyk, C. Abad, P. Leitner, and A. Iosup, “Let’s trace it: Fine-grained serverless benchmarking using synchronous and asynchronous orchestrated applications,” doi:10.48550/ARXIV.2205.07696, 2022. Used tooling: https://github.com/ServiBench/ReplicationPackage/tree/main/servi-bench Survey of other research on function benchmarking: J. Scheuner and P. Leitner, “Function-as-a-service performance evaluation: A multivocal literature review,” Journal of Systems and Software (JSS), vol. 170, 2020. SPEC RG Cloud https://research.spec.org/working-groups/rg-cloud/
xleitix
sheep_san_white
yuhattor
datarian
sansantech
soudai
ahrkrak
kazukihayase
nagix
devops_vtj
tk3fftk
nazonohito51
jeffersonlam
jnunemaker
orderedlist
addyosmani
eitanlees
chrislema
yeseniaperezcruz
aleyda
tenderlove
tammielis
jonyablonski
cromwellryan