Node.jsでのAWSサーバレスアプリプログラミングを簡単にする技術の研究紹介 (An Introduction of a Technology for Simplifying Serverless Application Programming in AWS with Node.js)

Node.jsでのAWSサーバレスアプリプログラミングを
簡単にする技術の研究紹介
An Introduction of a Technology for Simplifying Serverless Application Programming
in AWS with Node.js
Kosaku Kimura, FUJITSU LABORATORIES LTD.
[email protected]
Copyright 2019 FUJITSU LABORATORIES LTD.
0
JAWS DAYS 2019
2019年02月23日 @ TOC五反田メッセ
This talk is based on the paper entitled “A JavaScript Transpiler for Escaping from Complicated Usage of Cloud Services and APIs.” in APSEC 2018
Full text (preprint): http://bit.do/escapin_paper

View Slide

Speaker’s Profile
Kosaku Kimura（木村功作）
 Researcher at Fujitsu Labs (富士通研究所)
 Working on research in Software Engineering
 Recent Activities
 IEICE（電子情報通信学会），IPSJ（情報処理学会）
• IEICE-KBSE（知能ソフトウェア工学研究会） http://www.selab.is.ritsumei.ac.jp/kbse/
• IEICE-SC（サービスコンピューティング研究会） https://sig-sc.org/
• IPSJ-SE（ソフトウェア工学研究会） http://www.ipsj.or.jp/sig/se/
 Published research papers continually in academic conferences and workshops
• https://www.researchgate.net/profile/Kosaku_Kimura
• https://scholar.google.com/citations?user=JB0e924AAAAJ
• https://dblp.uni-trier.de/pers/hd/k/Kimura:Kosaku
1

View Slide

Agenda
 Background
 Serverless application programming in AWS with Node.js and its difficulties
 Escapin: a JavaScript transpiler for escaping from complicated usage of
cloud services and APIs
 Transpilation Examples
 Evaluation
 Results
 Conclusions
2

View Slide

An Excerpt of Serverless Application on AWS
Public
Web APIs
S3 Bucket
API Gateway Lambda Function
Node.js modules
(request, axios, etc.)
AWS Cloud
IAM
3

View Slide

Lambda Function with Node.js and AWS SDK
import * as AWS from "aws-sdk";
export async function uploadImage(event, context, callback) {
const { name, image } = event.formData;
const buffer = Buffer.from(image, "binary");
try {
await new AWS.S3().putObject({
Bucket: "images",
Key: name,
Body: buffer
}).promise();
callback(null, { message: "Succeeded" });
} catch (err) {
callback(err);
}
}
※For the program uses ES Modules, we need a polyfill to deploy it to Lambda
4

View Slide

Asynchronous Programming in JavaScript
Error-first callback Promise
Promisify
func(args, (err, data) => {
if (err) handleError(err);
else doSomething(data);
});
const p = args => {
return new Promise((resolve, reject) => {
if (err) reject(err);
else resolve(data);
});
});
};
p(args).then(doSomething).catch(handleError);
import { promisify } from "util";
const p = promisify(func);
p(args).then(doSomething).catch(handleError);
5

View Slide

Asynchronous Programming in JavaScript
async / await
generator / yield
import co from "co";
co(function* () {
const data = yield p(args);
doSomething(data);
}).catch(handleError);
(async () => {
try {
const data = await p(args);
doSomething(data);
} catch (err) {
handleError(err);
}
})();
?
6

View Slide

Welcome to async/await hell
“async/await freed us from callback hell, but people have started abusing it
— leading to the birth of async/await hell.” — Aditya Agarwal
(async () => {
const pizzaData = await getPizzaData() // async call
const drinkData = await getDrinkData() // async call
const chosenPizza = choosePizza() // sync call
const chosenDrink = chooseDrink() // sync call
await addPizzaToCart(chosenPizza) // async call
await addDrinkToCart(chosenDrink) // async call
orderItems() // async call
})()
async function selectPizza() {
const pizzaData = await getPizzaData() // async call
const chosenPizza = choosePizza() // sync call
await addPizzaToCart(chosenPizza) // async call
}
async function selectDrink() {
const drinkData = await getDrinkData() // async call
const chosenDrink = chooseDrink() // sync call
await addDrinkToCart(chosenDrink) // async call
}
(async () => {
const pizzaPromise = selectPizza()
const drinkPromise = selectDrink()
await pizzaPromise
await drinkPromise
orderItems() // async call
})()
async function orderItems() {
const items = await getCartItems() // async call
const noOfItems = items.length
for(var i = 0; i < noOfItems; i++) {
await sendRequest(items[i]) // async call
}
}
const noOfItems = items.length
const promises = []
for(var i = 0; i < noOfItems; i++) {
const orderPromise = sendRequest(items[i]) // async call
promises.push(orderPromise) // sync call
}
await Promise.all(promises) // async call
}
const promises = items.map((item) => sendRequest(item))
await Promise.all(promises) // async call
}
OR
7

View Slide

import { resize } from "imagemagick";
import rp from "request-promise-native";
import * as AWS from "aws-sdk";
export async function uploadImage(event, context, callback) {
await new AWS.S3().putObject({
Bucket: "images", Key: name, Body: buffer
}).promise();
resize({
srcData: buffer, format: "jpeg", width: 100
}, (err, stdout, stderr) => {
if (err) {
callback(err);
return;
}
const data = Buffer.from(stdout, "binary");
rp({
url: `https://api.com/v1/thumbnails/${name}`,
method: "put",
formData: { name, data },
headers: { "content-type": "multipart/form-data" }
}).then(() => {
callback(null, { message: "Succeeded" });
}).catch(err => {
throw err;
});
});
}
Serverless program must be complicated and error-prone
even if you believe what you want is simple
api.com
images
API Gateway uploadImage
Node.js modules
AWS Cloud
IAM
8

View Slide

Wouldn’t it be nice if you could write like this?
import api from "https://api.com/v1/spec";
export const images = {};
export function uploadImage(event) {
images[name] = buffer;
try {
const { stdout, stderr } = resize({
});
api.thumbnails[name] = { name, data };
return { message: "Succeeded" };
} catch (err) {
throw err;
}
}
api.com
images
API Gateway uploadImage
AWS Cloud
IAM
Node.js modules
9

View Slide

1. A JavaScript Transpiler for Escaping from Complicated Usage
of Cloud Services and APIs
2. ECMAScript (JavaScript) for Cloud and API Nature
Escapin
10

View Slide

Key Points
 RESTful API calls
just like imported objects
 S3/DynamoDB access
just like exported hash tables
 Lambda function declarations
just like exported functions
 Asynchronous features are
complemented when necessary
(Callback-agnostic programming)
 The above points never extend
JavaScript syntax
try {
});
} catch (err) {
throw err;
}
}
11

View Slide

Key Points
JavaScript syntax
12

View Slide

Key Points
JavaScript syntax
13

View Slide

Key Points
JavaScript syntax
throw err;
}
14

View Slide

Key Points
JavaScript syntax
});
}
15

View Slide

Key Points
JavaScript syntax
try {
});
} catch (err) {
throw err;
}
}
16

View Slide

Type Information
Transpilation Procedure
const items = {};
...
items[id] = obj;
...
...
await new Promise(
(resolve, reject) => {
Storage.put({
name: "items",
key: id,
value: obj
}, (err, res) =>{
else resolve(res);
});
...
Transformation
Rules
platform: a-cloud
where empty object $1 is declared
&& statement matches /$1[$2] = $3;/
when platform == "a-cloud"
do
replace statement with
Storage.put({
name: `${$1}`,
key: $2,
value: $3
});
PIP
PSP for setup
Configuration File
function function type
Storage#put error-first-callback
switch (process.argv[2]) {
case "create":
Storage.createTable({
name: "items", ... });
ret = IAM.createRole(
{ ... });
FaaS.createFunction({
FunctionName: "itemsGET",
Role: ret.Role, ... });
break;
case "update":
...
PSP to be deployed
as a function
Destructuring
Nesting Callbacks
Refinements Asynchronization
 From platform-independent programs (PIPs) to platform-specific programs (PSPs)
having the same functionality but utilize Clouds and APIs in order to satisfy requirements
17

View Slide

Destructuring Nesting Callbacks
Obtain a complete callback-agnostic representation
from a PIP that has legacy nesting callbacks
func(arg, (err, data) => {
if (err) {
handleError(err);
return;
}
doSomething(data);
});
try {
const data = func(arg);
doSomething(data);
} catch (err) {
handleError(err);
}
18

View Slide

Refinements – APIs
Method
Path
GET POST PUT DELETE
/items api.items api.items(body) api.items = body delete api.items
/items/:id api.items[id] api.items[id](body) api.items[id] = body delete api.items[id]
/items/:id
with params
api.items
[id][params]
api.items
[id][params](body)
api.items
[id][params] = body
delete api.items
[id][params]
 Manipulations of objects imported from a URI of an API specification file, which
conforms to OpenAPI Specification, are transformed into corresponding API calls
import api from 'https://path/to/swagger.yaml';
const item = api.items[id];
const item = request({
'uri': `https://api.com/v1/items/${id}`,
'method': 'get',
'contentType': 'application/json',
'headers': {
'authorization': 'Basic ... '
}
});
19

View Slide

Function Lambda
export function foo(bar) {…} createFunction
Refinements – AWS
Object DynamoDB S3
export const obj = {} createTable createBucket
obj[id] getItem getObject
obj[id] = … putItem putObject
delete obj[id] deleteItem deleteObject
Object.keys(obj) scan listObjectV2
 Manipulations of exported objects and functions are transformed into corresponding
API calls for cloud services
20

View Slide

Refinements – AWS
platform: aws
PIP
PSP for setup
Configuration File
PSP to be deployed
export const csv = {};
const keys = Object.keys(csv);
const data = new AWS.DynamoDB().scan({
TableName: 'csv', …
});
const keys = data.Items.map(
item => item.key.S);
import AWS from 'aws-sdk';
switch (process.argv[2]) {
case 'create':
new AWS.DyanmoDB().createTable({
TableName: 'csv', …});
case 'update': …
21

View Slide

Asynchronization
Function Type
DynamoDB#scan error-first-callback
Array#map general-callback
… …
PSP Asynchronized PSP
Type Information
const data = new AWS.DynamoDB().scan({
});
const keys = data.Items.map(
const _data
= await new Promise((resolve, reject) => {
new AWS.DynamoDB().scan({
}, (err, data) => {
else resolve(data);
});
});
const keys = _data.Items.map(
22

View Slide

Transpilation Examples
23

View Slide

Functions with Error-first Callback
if (err) handleError(err);
else doSomething(data);
});
try {
const _data = await new Promise(
else resolve(data);
});
});
doSomething(_data);
} catch (err) {
handleError(err);
}
try {
const data = func(arg);
doSomething(data);
} catch (err) {
handleError(err);
}
OR
24

View Slide

API Calls
import api from 'https://path/to/swagger.yaml';
if (api.items[id].attr === "foo") {
doSomething(api.items[id].attr);
}
const _data = await new Promise(
request({
'uri': `https://api.com/v1/items/${id}`,
'method': 'get',
'contentType': 'application/json',
'headers': {
'authorization': 'Basic ... '
}
}, (err, data) => {
else resolve(data);
});
});
if (_data.attr === "foo") {
doSomething(_data.attr);
}
25

View Slide

Iterations
for (let i = 0; i < n; i++) {
asyncFunc(i);
}
const _p = [];
for (let i = 0; i < n; i++) {
_p.push(asyncFunc(i));
}
await Promise.all(_p);
for (const item of items) {
const data = asyncFunc(item);
doSomething(data);
}
const _promises = [];
for (const item of items) {
_promises.push((async () => {
const data = await asyncFunc(item);
doSomething(data);
})());
}
await Promise.all(_promises);
26

View Slide

Functions with Callback in which async/await cannot be used
func(item => {
const ret = asyncFunc(item);
doSomething(ret);
});
import deasync from "deasync";
func(item => {
let _t, _done = false;
asyncFunc(item).then(data => {
_t = data;
}).catch(err => {
throw err;
}).finally(() => {
_done = true;
});
deasync.loopWhile(() => !_done);
const ret = _t;
doSomething(ret);
});
27

View Slide

A Complicated Case with DynamoDB
export const items = {};
if (Object.keys(items).length > 0) {
doSomething(items[foo]);
} else if (asyncFunc(args)) {
doSomething(bar);
}
const _data = await new Promise((resolve, reject) => {
new AWS.DynamoDB().scan({
TableName: 'items',
ExpressionAttributeNames: { '#ky': 'key', },
ProjectionExpression: '#ky',
}, (err, data) => {
else resolve(data);
});
});
const keys = _data.Items.map(item => item.key.S);
if (keys.length > 0) {
let _data2;
const _data3 = await new Promise((resolve, reject) => {
new AWS.DynamoDB().getItem({
TableName: 'items',
Key: { key: { S: foo, }, },
}, (err, data) => {
else resolve(data);
});
});
if (_data3 === null || _data3.Item === undefined) {
_data2 = undefined;
} else {
_data2
= (_data3.Item.type.S === 'object' || _data3.Item.type.S === 'function')
? JSON.parse(_data3.Item.value.S) : _data3.Item.value.S;
}
doSomething(_data2);
} else {
const _data3 = await asyncFunc(args);
if (_data3) {
doSomething(bar);
}
}
28

View Slide

Evaluation
29

View Slide

Research Questions
RQ1： (Effectiveness)
To what degree does Escapin reduce the amount of
time to develop applications using APIs and cloud
services?
RQ2： (Source code simplicity)
To what degree can engineers write simple source code
by using Escapin?
30

View Slide

Preliminary
Implemented Escapin as services
 using Node.js, Yarn, Babel, Webpack, TypeScript, SwaggerParser
 CLI and documentation also available
Created two API servers on AWS by using Escapin
 Mock server of Uber API implementing price estimation feature only
 Test API for checking whether applications are correctly implemented
Gathered 10 subjects
 researchers and software engineers in Fujitsu.
31

View Slide

Development time = Σ len( )
Experimental Design (1/2)
 Subjects do an application development task using the mock Uber API to confirm:
 for RQ1, development time
 for RQ2, lines of code (LOC), cyclomatic complexity, and cognitive complexity
 Before doing the task, they learn how to use AWS, Node.js, and Escapin enough
to remove the learning time from the evaluation
 Measure development time by requesting Test API in order to do the task at subjects’
own workspaces
start（null）
→ false
submit v1
→ false
suspend（null）
→ false
resume（null）
→ false
submit v2
→ true
submit v3
→ true
t
32

View Slide

Experimental Design (2/2)
 All subjects do the task in two cases in order to compare metrics for each subject
independently
 Case 1: using any existing tools without Escapin
 Case 2: using Escapin
 Separate the ten subjects into two groups that do the task in different order of the cases
 for fair evaluation against the existence of common operations whose artifacts can be reused
in a latter case (e.g., finding correct response parameters of Uber API)
×10
×5 ×5
Group 1 Group 2
Case 2 Case 1
Case 1 Case 2
33

View Slide

Task: Summarizing Estimations of Uber Products
Develop an API GET /rough_estimates conforming to the following API specification:
 The API is deployed on AWS API Gateway and Lambda (Node.js)
 The API has four query parameters:
• start_latitude, start_longitude, end_latitude, and end_longitude
 The API conducts the following steps:
1. Get estimate and product_id of an available car nearby by requesting GET /estimates/time with the query parameters.
2. Rename estimate to time.
3. Get the car information description, capacity and image by requesting GET /estimates/:id with product_id obtained by
step 2.
4. Get price estimation information by requesting GET /estimates/price with the query parameters.
5. From the information obtained by step 4, get estimate and distance that correspond to each product_id obtained by step 2.
6. Exclude entries that estimate is set to "Metered", which means this is a taxi.
7. Rename estimate to price.
8. Format the data as a JSON object and return it.
34

View Slide

Results
35

View Slide

Subject A
bit.ly/escapin-results
415 min 210 min
Case 1 Case 2
36

View Slide

Subject B
200 min 110 min
Case 1 Case 2
37

View Slide

Subject C
81 min 66 min
Case 1 Case 2
38

View Slide

Subject D
175 min 82 min
Case 1 Case 2
39

View Slide

Subject E
127 min 69 min
Case 1 Case 2
40

View Slide

Subject F
226 min 175 min
Case 1 Case 2
41

View Slide

Subject G
309 min 168 min
Case 1 Case 2
42

View Slide

Subject H
167 min 97 min
Case 1 Case 2
43

View Slide

Subject I
329 min 182 min
Case 1 Case 2
44

View Slide

Subject J
299 min 165 min
Case 1 Case 2
45

View Slide

Answer to the Research Questions
 Escapin statistically significantly (α=0.05)
 decreases development time (RQ1)
 simplifies source code (RQ2)
RQ1
Effectiveness
RQ2
Source code simplicity
Development
Time
LOC
Cyclomatic
Complexity
Cognitive
Complexity
p-value 0.00016 0.00003 0.00181 0.01384
Effect Size
(Cohen’s d)
Huge
(11.66398)
Huge
(9.61006)
Huge
(5.04810)
Huge
(2.47486)
Observed
Statistical Power
0.99179 0.99995 0.99994 0.90048
by paired-sample one-tailed t-test
46

View Slide

Threats to Validity
Threats to internal validity
 The number of subjects (n=10) is too small to confirm
whether the all metrics follow a normal distribution
 Correctness of measuring development time by manually requesting Test API
 Overhead of context switching caused by taking breaks
Threats to external validity
 Task is too small and few to remark the generality and applicability
47

View Slide

Conclusions
 Escapin: a JavaScript transpiler for escaping from complicated usage of cloud services and APIs
 Introduce new features without extending JavaScript syntax
• Importing APIs as objects
• Exporting objects and functions as cloud service resources
 Enable callback-agnostic programming
• No need to use asynchronous features in programming
 In the application development task using Uber API, Escapin decreases
development time and simplifies source code compared with existing tools
Future work
 Further evaluation
 more complicated tasks
 a number of subjects enough to do interval estimation of the metrics
 Support cloud services other than AWS
 Provide auxiliary tools: linter, code completion, testing, debugging, etc.
 Transpiling optimizations for satisfying non-functional requirements
48

View Slide

View Slide

Node.jsでのAWSサーバレスアプリプログラミングを簡単にする技術の研究紹介 (An Introduction of a Technology for Simplifying Serverless Application Programming in AWS with Node.js)

Node.jsでのAWSサーバレスアプリプログラミングを簡単にする技術の研究紹介 (An Introduction of a Technology for Simplifying Serverless Application Programming in AWS with Node.js)

Kosaku Kimura

More Decks by Kosaku Kimura

Other Decks in Programming

Featured

Transcript

Node.jsでのAWSサーバレスアプリプログラミングを 簡単にする技術の研究紹介 (An Introduction of a Technology for Simplifying Serverless Application Programming in AWS with Node.js)

Node.jsでのAWSサーバレスアプリプログラミングを 簡単にする技術の研究紹介 (An Introduction of a Technology for Simplifying Serverless Application Programming in AWS with Node.js)

Kosaku Kimura

More Decks by Kosaku Kimura

Other Decks in Programming

Featured

Transcript

Node.jsでのAWSサーバレスアプリプログラミングを簡単にする技術の研究紹介 (An Introduction of a Technology for Simplifying Serverless Application Programming in AWS with Node.js)

Node.jsでのAWSサーバレスアプリプログラミングを簡単にする技術の研究紹介 (An Introduction of a Technology for Simplifying Serverless Application Programming in AWS with Node.js)