Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Node.jsでのAWSサーバレスアプリプログラミングを 簡単にする技術の研究紹介 (An I...

Node.jsでのAWSサーバレスアプリプログラミングを 簡単にする技術の研究紹介 (An Introduction of a Technology for Simplifying Serverless Application Programming in AWS with Node.js)

JAWS DAYS 2019 (2019年02月23日 @ TOC五反田メッセ)での講演資料
https://jawsdays2019.jaws-ug.jp/session/1211/

This talk is based on the paper entitled “A JavaScript Transpiler for Escaping from Complicated Usage of Cloud Services and APIs” in APSEC 2018.
Full text (preprint): https://www.researchgate.net/publication/330533667
DOI: https://doi.org/10.1109/APSEC.2018.00021

Kosaku Kimura

February 23, 2019
Tweet

More Decks by Kosaku Kimura

Other Decks in Programming

Transcript

  1. Node.jsでのAWSサーバレスアプリプログラミングを 簡単にする技術の研究紹介 An Introduction of a Technology for Simplifying Serverless

    Application Programming in AWS with Node.js Kosaku Kimura, FUJITSU LABORATORIES LTD. [email protected] Copyright 2019 FUJITSU LABORATORIES LTD. 0 JAWS DAYS 2019 2019年02月23日 @ TOC五反田メッセ This talk is based on the paper entitled “A JavaScript Transpiler for Escaping from Complicated Usage of Cloud Services and APIs.” in APSEC 2018 Full text (preprint): http://bit.do/escapin_paper
  2. Speaker’s Profile Kosaku Kimura(木村 功作)  Researcher at Fujitsu Labs

    (富士通研究所)  Working on research in Software Engineering  Recent Activities  IEICE(電子情報通信学会),IPSJ(情報処理学会) • IEICE-KBSE(知能ソフトウェア工学研究会) http://www.selab.is.ritsumei.ac.jp/kbse/ • IEICE-SC(サービスコンピューティング研究会) https://sig-sc.org/ • IPSJ-SE(ソフトウェア工学研究会) http://www.ipsj.or.jp/sig/se/  Published research papers continually in academic conferences and workshops • https://www.researchgate.net/profile/Kosaku_Kimura • https://scholar.google.com/citations?user=JB0e924AAAAJ • https://dblp.uni-trier.de/pers/hd/k/Kimura:Kosaku Copyright 2019 FUJITSU LABORATORIES LTD. 1
  3. Agenda  Background  Serverless application programming in AWS with

    Node.js and its difficulties  Escapin: a JavaScript transpiler for escaping from complicated usage of cloud services and APIs  Transpilation Examples  Evaluation  Results  Conclusions Copyright 2019 FUJITSU LABORATORIES LTD. 2
  4. An Excerpt of Serverless Application on AWS Copyright 2019 FUJITSU

    LABORATORIES LTD. Public Web APIs S3 Bucket API Gateway Lambda Function Node.js modules (request, axios, etc.) AWS Cloud IAM 3
  5. Lambda Function with Node.js and AWS SDK Copyright 2019 FUJITSU

    LABORATORIES LTD. import * as AWS from "aws-sdk"; export async function uploadImage(event, context, callback) { const { name, image } = event.formData; const buffer = Buffer.from(image, "binary"); try { await new AWS.S3().putObject({ Bucket: "images", Key: name, Body: buffer }).promise(); callback(null, { message: "Succeeded" }); } catch (err) { callback(err); } } ※For the program uses ES Modules, we need a polyfill to deploy it to Lambda 4
  6. Asynchronous Programming in JavaScript Copyright 2019 FUJITSU LABORATORIES LTD. Error-first

    callback Promise Promisify func(args, (err, data) => { if (err) handleError(err); else doSomething(data); }); const p = args => { return new Promise((resolve, reject) => { func(args, (err, data) => { if (err) reject(err); else resolve(data); }); }); }; p(args).then(doSomething).catch(handleError); import { promisify } from "util"; const p = promisify(func); p(args).then(doSomething).catch(handleError); 5
  7. Asynchronous Programming in JavaScript Copyright 2019 FUJITSU LABORATORIES LTD. async

    / await generator / yield import co from "co"; const p = promisify(func); co(function* () { const data = yield p(args); doSomething(data); }).catch(handleError); const p = promisify(func); (async () => { try { const data = await p(args); doSomething(data); } catch (err) { handleError(err); } })(); ? 6
  8. Welcome to async/await hell Copyright 2019 FUJITSU LABORATORIES LTD. “async/await

    freed us from callback hell, but people have started abusing it — leading to the birth of async/await hell.” — Aditya Agarwal (async () => { const pizzaData = await getPizzaData() // async call const drinkData = await getDrinkData() // async call const chosenPizza = choosePizza() // sync call const chosenDrink = chooseDrink() // sync call await addPizzaToCart(chosenPizza) // async call await addDrinkToCart(chosenDrink) // async call orderItems() // async call })() async function selectPizza() { const pizzaData = await getPizzaData() // async call const chosenPizza = choosePizza() // sync call await addPizzaToCart(chosenPizza) // async call } async function selectDrink() { const drinkData = await getDrinkData() // async call const chosenDrink = chooseDrink() // sync call await addDrinkToCart(chosenDrink) // async call } (async () => { const pizzaPromise = selectPizza() const drinkPromise = selectDrink() await pizzaPromise await drinkPromise orderItems() // async call })() async function orderItems() { const items = await getCartItems() // async call const noOfItems = items.length for(var i = 0; i < noOfItems; i++) { await sendRequest(items[i]) // async call } } async function orderItems() { const items = await getCartItems() // async call const noOfItems = items.length const promises = [] for(var i = 0; i < noOfItems; i++) { const orderPromise = sendRequest(items[i]) // async call promises.push(orderPromise) // sync call } await Promise.all(promises) // async call } async function orderItems() { const items = await getCartItems() // async call const promises = items.map((item) => sendRequest(item)) await Promise.all(promises) // async call } OR 7
  9. import { resize } from "imagemagick"; import rp from "request-promise-native";

    import * as AWS from "aws-sdk"; export async function uploadImage(event, context, callback) { const { name, image } = event.formData; const buffer = Buffer.from(image, "binary"); await new AWS.S3().putObject({ Bucket: "images", Key: name, Body: buffer }).promise(); resize({ srcData: buffer, format: "jpeg", width: 100 }, (err, stdout, stderr) => { if (err) { callback(err); return; } const data = Buffer.from(stdout, "binary"); rp({ url: `https://api.com/v1/thumbnails/${name}`, method: "put", formData: { name, data }, headers: { "content-type": "multipart/form-data" } }).then(() => { callback(null, { message: "Succeeded" }); }).catch(err => { throw err; }); }); } Serverless program must be complicated and error-prone even if you believe what you want is simple Copyright 2019 FUJITSU LABORATORIES LTD. api.com images API Gateway uploadImage Node.js modules AWS Cloud IAM 8
  10. Wouldn’t it be nice if you could write like this?

    Copyright 2019 FUJITSU LABORATORIES LTD. import api from "https://api.com/v1/spec"; import { resize } from "imagemagick"; export const images = {}; export function uploadImage(event) { const { name, image } = event.formData; const buffer = Buffer.from(image, "binary"); images[name] = buffer; try { const { stdout, stderr } = resize({ srcData: buffer, format: "jpeg", width: 100 }); const data = Buffer.from(stdout, "binary"); api.thumbnails[name] = { name, data }; return { message: "Succeeded" }; } catch (err) { throw err; } } api.com images API Gateway uploadImage AWS Cloud IAM Node.js modules 9
  11. 1. A JavaScript Transpiler for Escaping from Complicated Usage of

    Cloud Services and APIs 2. ECMAScript (JavaScript) for Cloud and API Nature Escapin Copyright 2019 FUJITSU LABORATORIES LTD. 10
  12. Key Points  RESTful API calls just like imported objects

     S3/DynamoDB access just like exported hash tables  Lambda function declarations just like exported functions  Asynchronous features are complemented when necessary (Callback-agnostic programming)  The above points never extend JavaScript syntax Copyright 2019 FUJITSU LABORATORIES LTD. import api from "https://api.com/v1/spec"; import { resize } from "imagemagick"; export const images = {}; export function uploadImage(event) { const { name, image } = event.formData; const buffer = Buffer.from(image, "binary"); images[name] = buffer; try { const { stdout, stderr } = resize({ srcData: buffer, format: "jpeg", width: 100 }); const data = Buffer.from(stdout, "binary"); api.thumbnails[name] = { name, data }; return { message: "Succeeded" }; } catch (err) { throw err; } } 11
  13. Key Points  RESTful API calls just like imported objects

     S3/DynamoDB access just like exported hash tables  Lambda function declarations just like exported functions  Asynchronous features are complemented when necessary (Callback-agnostic programming)  The above points never extend JavaScript syntax Copyright 2019 FUJITSU LABORATORIES LTD. import api from "https://api.com/v1/spec"; api.thumbnails[name] = { name, data }; 12
  14. Key Points  RESTful API calls just like imported objects

     S3/DynamoDB access just like exported hash tables  Lambda function declarations just like exported functions  Asynchronous features are complemented when necessary (Callback-agnostic programming)  The above points never extend JavaScript syntax Copyright 2019 FUJITSU LABORATORIES LTD. export const images = {}; images[name] = buffer; 13
  15. Key Points  RESTful API calls just like imported objects

     S3/DynamoDB access just like exported hash tables  Lambda function declarations just like exported functions  Asynchronous features are complemented when necessary (Callback-agnostic programming)  The above points never extend JavaScript syntax Copyright 2019 FUJITSU LABORATORIES LTD. export function uploadImage(event) { return { message: "Succeeded" }; throw err; } 14
  16. Key Points  RESTful API calls just like imported objects

     S3/DynamoDB access just like exported hash tables  Lambda function declarations just like exported functions  Asynchronous features are complemented when necessary (Callback-agnostic programming)  The above points never extend JavaScript syntax Copyright 2019 FUJITSU LABORATORIES LTD. images[name] = buffer; const { stdout, stderr } = resize({ srcData: buffer, format: "jpeg", width: 100 }); api.thumbnails[name] = { name, data }; } 15
  17. Key Points  RESTful API calls just like imported objects

     S3/DynamoDB access just like exported hash tables  Lambda function declarations just like exported functions  Asynchronous features are complemented when necessary (Callback-agnostic programming)  The above points never extend JavaScript syntax Copyright 2019 FUJITSU LABORATORIES LTD. import api from "https://api.com/v1/spec"; import { resize } from "imagemagick"; export const images = {}; export function uploadImage(event) { const { name, image } = event.formData; const buffer = Buffer.from(image, "binary"); images[name] = buffer; try { const { stdout, stderr } = resize({ srcData: buffer, format: "jpeg", width: 100 }); const data = Buffer.from(stdout, "binary"); api.thumbnails[name] = { name, data }; return { message: "Succeeded" }; } catch (err) { throw err; } } 16
  18. Type Information Transpilation Procedure const items = {}; ... items[id]

    = obj; ... ... await new Promise( (resolve, reject) => { Storage.put({ name: "items", key: id, value: obj }, (err, res) =>{ if (err) reject(err); else resolve(res); }); ... Transformation Rules platform: a-cloud where empty object $1 is declared && statement matches /$1[$2] = $3;/ when platform == "a-cloud" do replace statement with Storage.put({ name: `${$1}`, key: $2, value: $3 }); PIP PSP for setup Configuration File function function type Storage#put error-first-callback switch (process.argv[2]) { case "create": Storage.createTable({ name: "items", ... }); ret = IAM.createRole( { ... }); FaaS.createFunction({ FunctionName: "itemsGET", Role: ret.Role, ... }); break; case "update": ... PSP to be deployed as a function Destructuring Nesting Callbacks Refinements Asynchronization Copyright 2019 FUJITSU LABORATORIES LTD.  From platform-independent programs (PIPs) to platform-specific programs (PSPs) having the same functionality but utilize Clouds and APIs in order to satisfy requirements 17
  19. Destructuring Nesting Callbacks Obtain a complete callback-agnostic representation from a

    PIP that has legacy nesting callbacks Copyright 2019 FUJITSU LABORATORIES LTD. func(arg, (err, data) => { if (err) { handleError(err); return; } doSomething(data); }); try { const data = func(arg); doSomething(data); } catch (err) { handleError(err); } 18
  20. Refinements – APIs Method Path GET POST PUT DELETE /items

    api.items api.items(body) api.items = body delete api.items /items/:id api.items[id] api.items[id](body) api.items[id] = body delete api.items[id] /items/:id with params api.items [id][params] api.items [id][params](body) api.items [id][params] = body delete api.items [id][params]  Manipulations of objects imported from a URI of an API specification file, which conforms to OpenAPI Specification, are transformed into corresponding API calls Copyright 2019 FUJITSU LABORATORIES LTD. import api from 'https://path/to/swagger.yaml'; const item = api.items[id]; const item = request({ 'uri': `https://api.com/v1/items/${id}`, 'method': 'get', 'contentType': 'application/json', 'headers': { 'authorization': 'Basic ... ' } }); 19
  21. Function Lambda export function foo(bar) {…} createFunction Refinements – AWS

    Object DynamoDB S3 export const obj = {} createTable createBucket obj[id] getItem getObject obj[id] = … putItem putObject delete obj[id] deleteItem deleteObject Object.keys(obj) scan listObjectV2  Manipulations of exported objects and functions are transformed into corresponding API calls for cloud services Copyright 2019 FUJITSU LABORATORIES LTD. 20
  22. Refinements – AWS platform: aws PIP PSP for setup Configuration

    File PSP to be deployed Copyright 2019 FUJITSU LABORATORIES LTD. export const csv = {}; const keys = Object.keys(csv); const data = new AWS.DynamoDB().scan({ TableName: 'csv', … }); const keys = data.Items.map( item => item.key.S); import AWS from 'aws-sdk'; switch (process.argv[2]) { case 'create': new AWS.DyanmoDB().createTable({ TableName: 'csv', …}); case 'update': … 21
  23. Asynchronization Function Type DynamoDB#scan error-first-callback Array#map general-callback … … PSP

    Asynchronized PSP Type Information Copyright 2019 FUJITSU LABORATORIES LTD. const data = new AWS.DynamoDB().scan({ TableName: 'csv', … }); const keys = data.Items.map( item => item.key.S); const _data = await new Promise((resolve, reject) => { new AWS.DynamoDB().scan({ TableName: 'csv', … }, (err, data) => { if (err) reject(err); else resolve(data); }); }); const keys = _data.Items.map( item => item.key.S); 22
  24. Functions with Error-first Callback Copyright 2019 FUJITSU LABORATORIES LTD. func(args,

    (err, data) => { if (err) handleError(err); else doSomething(data); }); try { const _data = await new Promise( (resolve, reject) => { func(args, (err, data) => { if (err) reject(err); else resolve(data); }); }); doSomething(_data); } catch (err) { handleError(err); } try { const data = func(arg); doSomething(data); } catch (err) { handleError(err); } OR 24
  25. API Calls Copyright 2019 FUJITSU LABORATORIES LTD. import api from

    'https://path/to/swagger.yaml'; if (api.items[id].attr === "foo") { doSomething(api.items[id].attr); } const _data = await new Promise( (resolve, reject) => { request({ 'uri': `https://api.com/v1/items/${id}`, 'method': 'get', 'contentType': 'application/json', 'headers': { 'authorization': 'Basic ... ' } }, (err, data) => { if (err) reject(err); else resolve(data); }); }); if (_data.attr === "foo") { doSomething(_data.attr); } 25
  26. Iterations Copyright 2019 FUJITSU LABORATORIES LTD. for (let i =

    0; i < n; i++) { asyncFunc(i); } const _p = []; for (let i = 0; i < n; i++) { _p.push(asyncFunc(i)); } await Promise.all(_p); for (const item of items) { const data = asyncFunc(item); doSomething(data); } const _promises = []; for (const item of items) { _promises.push((async () => { const data = await asyncFunc(item); doSomething(data); })()); } await Promise.all(_promises); 26
  27. Functions with Callback in which async/await cannot be used Copyright

    2019 FUJITSU LABORATORIES LTD. func(item => { const ret = asyncFunc(item); doSomething(ret); }); import deasync from "deasync"; func(item => { let _t, _done = false; asyncFunc(item).then(data => { _t = data; }).catch(err => { throw err; }).finally(() => { _done = true; }); deasync.loopWhile(() => !_done); const ret = _t; doSomething(ret); }); 27
  28. A Complicated Case with DynamoDB Copyright 2019 FUJITSU LABORATORIES LTD.

    export const items = {}; if (Object.keys(items).length > 0) { doSomething(items[foo]); } else if (asyncFunc(args)) { doSomething(bar); } const _data = await new Promise((resolve, reject) => { new AWS.DynamoDB().scan({ TableName: 'items', ExpressionAttributeNames: { '#ky': 'key', }, ProjectionExpression: '#ky', }, (err, data) => { if (err) reject(err); else resolve(data); }); }); const keys = _data.Items.map(item => item.key.S); if (keys.length > 0) { let _data2; const _data3 = await new Promise((resolve, reject) => { new AWS.DynamoDB().getItem({ TableName: 'items', Key: { key: { S: foo, }, }, }, (err, data) => { if (err) reject(err); else resolve(data); }); }); if (_data3 === null || _data3.Item === undefined) { _data2 = undefined; } else { _data2 = (_data3.Item.type.S === 'object' || _data3.Item.type.S === 'function') ? JSON.parse(_data3.Item.value.S) : _data3.Item.value.S; } doSomething(_data2); } else { const _data3 = await asyncFunc(args); if (_data3) { doSomething(bar); } } 28
  29. Research Questions RQ1: (Effectiveness) To what degree does Escapin reduce

    the amount of time to develop applications using APIs and cloud services? RQ2: (Source code simplicity) To what degree can engineers write simple source code by using Escapin? Copyright 2019 FUJITSU LABORATORIES LTD. 30
  30. Preliminary Implemented Escapin as services  using Node.js, Yarn, Babel,

    Webpack, TypeScript, SwaggerParser  CLI and documentation also available Created two API servers on AWS by using Escapin  Mock server of Uber API implementing price estimation feature only  Test API for checking whether applications are correctly implemented Gathered 10 subjects  researchers and software engineers in Fujitsu. Copyright 2019 FUJITSU LABORATORIES LTD. 31
  31. Development time = Σ len( ) Experimental Design (1/2) 

    Subjects do an application development task using the mock Uber API to confirm:  for RQ1, development time  for RQ2, lines of code (LOC), cyclomatic complexity, and cognitive complexity  Before doing the task, they learn how to use AWS, Node.js, and Escapin enough to remove the learning time from the evaluation  Measure development time by requesting Test API in order to do the task at subjects’ own workspaces start(null) → false submit v1 → false suspend(null) → false resume(null) → false submit v2 → true submit v3 → true t Copyright 2019 FUJITSU LABORATORIES LTD. 32
  32. Experimental Design (2/2)  All subjects do the task in

    two cases in order to compare metrics for each subject independently  Case 1: using any existing tools without Escapin  Case 2: using Escapin  Separate the ten subjects into two groups that do the task in different order of the cases  for fair evaluation against the existence of common operations whose artifacts can be reused in a latter case (e.g., finding correct response parameters of Uber API) ×10 ×5 ×5 Group 1 Group 2 Case 2 Case 1 Case 1 Case 2 Copyright 2019 FUJITSU LABORATORIES LTD. 33
  33. Task: Summarizing Estimations of Uber Products Develop an API GET

    /rough_estimates conforming to the following API specification:  The API is deployed on AWS API Gateway and Lambda (Node.js)  The API has four query parameters: • start_latitude, start_longitude, end_latitude, and end_longitude  The API conducts the following steps: 1. Get estimate and product_id of an available car nearby by requesting GET /estimates/time with the query parameters. 2. Rename estimate to time. 3. Get the car information description, capacity and image by requesting GET /estimates/:id with product_id obtained by step 2. 4. Get price estimation information by requesting GET /estimates/price with the query parameters. 5. From the information obtained by step 4, get estimate and distance that correspond to each product_id obtained by step 2. 6. Exclude entries that estimate is set to "Metered", which means this is a taxi. 7. Rename estimate to price. 8. Format the data as a JSON object and return it. Copyright 2019 FUJITSU LABORATORIES LTD. 34
  34. Subject A bit.ly/escapin-results 415 min 210 min Case 1 Case

    2 Copyright 2019 FUJITSU LABORATORIES LTD. 36
  35. Subject B 200 min 110 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 37
  36. Subject C 81 min 66 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 38
  37. Subject D 175 min 82 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 39
  38. Subject E 127 min 69 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 40
  39. Subject F 226 min 175 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 41
  40. Subject G 309 min 168 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 42
  41. Subject H 167 min 97 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 43
  42. Subject I 329 min 182 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 44
  43. Subject J 299 min 165 min Case 1 Case 2

    bit.ly/escapin-results Copyright 2019 FUJITSU LABORATORIES LTD. 45
  44. Answer to the Research Questions  Escapin statistically significantly (α=0.05)

     decreases development time (RQ1)  simplifies source code (RQ2) RQ1 Effectiveness RQ2 Source code simplicity Development Time LOC Cyclomatic Complexity Cognitive Complexity p-value 0.00016 0.00003 0.00181 0.01384 Effect Size (Cohen’s d) Huge (11.66398) Huge (9.61006) Huge (5.04810) Huge (2.47486) Observed Statistical Power 0.99179 0.99995 0.99994 0.90048 by paired-sample one-tailed t-test Copyright 2019 FUJITSU LABORATORIES LTD. 46
  45. Threats to Validity Threats to internal validity  The number

    of subjects (n=10) is too small to confirm whether the all metrics follow a normal distribution  Correctness of measuring development time by manually requesting Test API  Overhead of context switching caused by taking breaks Threats to external validity  Task is too small and few to remark the generality and applicability Copyright 2019 FUJITSU LABORATORIES LTD. 47
  46. Conclusions  Escapin: a JavaScript transpiler for escaping from complicated

    usage of cloud services and APIs  Introduce new features without extending JavaScript syntax • Importing APIs as objects • Exporting objects and functions as cloud service resources  Enable callback-agnostic programming • No need to use asynchronous features in programming  In the application development task using Uber API, Escapin decreases development time and simplifies source code compared with existing tools Future work  Further evaluation  more complicated tasks  a number of subjects enough to do interval estimation of the metrics  Support cloud services other than AWS  Provide auxiliary tools: linter, code completion, testing, debugging, etc.  Transpiling optimizations for satisfying non-functional requirements Copyright 2019 FUJITSU LABORATORIES LTD. 48