Use Lambda and DynamoDB to build simple dashboard

Transcript

1. Use Lambda and DynamoDB to build simple dashboard @ocowchun
2. What's AWS Lambda and DynamoDB?
3. AWS Lambda 4 Cost by requests ($0.2 for 1 million requests/month) 4 Easy to deploy 4 Easy to manage aws resources(S3, DynamoDB, ...) 4 Support Java, Node.js, Python, C# 4 No Ops(basically)
4. DynamoDB 4 NoSQL 4 Easy to Setup, Operate & Scale 4 Pay only for the throughput and storage you need 4 Write Capacity: $0.0065 for 10 units*/hour 4 Read Capacity: $0.0065 for 50 units*/hour * For small items, one unit of capacity can handle one request per second
5. Ok, so why choose AWS Lambda and DynamoDB
6. I am too lazy to manage my servers
7. Heroku is soooooo expensive for nano project
8. AWS Lambda and DynamoDB is the best solution for nano project
9. Background: I want to know the exchange rate What I need to do: 4 Fetch data from bank website 4 Store data 4 Query data with timestamp 4 Visualize data
10. Fetch data from bank part1 (fetch) cheerio 4 https://github.com/cheeriojs/cheerio 4 Use the goddamm simple jQuery syntax to manipulate DOM in Node.js
11. Fetch data from bank part1 (fetch) function fetchJpyToNtd() { const url = 'https://www.cathaybk.com.tw/cathaybk/exchange/currency-billboard.asp?page=current'; return new Promise(function(resolve, reject) { request(url, function(error, response, body) { if (!error && response.statusCode === 200) { const $ = cheerio.load(body); try { const exchange = $('.icon_currency_jpy').parent().parent(). find('.t_align_right')[1].children[0].data; resolve(exchange); } catch (ex) { reject(ex); } } else { reject(error); } }); }); }
12. Fetch data from bank part2 (cronjob) Using AWS Lambda with Scheduled Events to fetch data hourly 4 http://docs.aws.amazon.com/AmazonCloudWatch/ latest/events/ScheduledEvents.html 4 Minimum precision for schedules is 1 minute
13. Fetch data from bank part2 (cronjob) resource "aws_cloudwatch_event_rule" "cathay_hourly_event" { name = "cathay_hourly_event" schedule_expression = "cron(5 * * * ? *)" } resource "aws_cloudwatch_event_target" "lime_fetchCathaybkJpyToNtd" { target_id = "lime_fetchCathaybkJpyToNtd" rule = "${aws_cloudwatch_event_rule.cathay_hourly_event.name}" arn = "arn:aws:lambda:ap-northeast-1:340065150345:function:lime_fetchCathaybkJpyToNtd" }
14. Store data Data Schema 4 id (Number, hash_key) 4 unix_created_at (Number, range_key) 4 value (Number)
15. Store data export function create(pointParams) { const now = moment(); const unix_created_at = now.valueOf(); const params = { TableName, Item: { id: { S: pointParams.id, }, unix_created_at: { N: unix_created_at.toString(), }, value: { N: pointParams.value, }, }, }; return new Promise(function(resolve, reject) { dynamodb.putItem(params, function(err, data) { if (err) reject(err, err.stack); // an error occurred else { resolve(data); // successful response } }); }); }
16. Query data with timestamp (query) export function last(id, days = 30) { const time = moment().subtract(days, 'days'); const unix_created_at = time.valueOf().toString(); const params = { TableName, ConsistentRead: true, KeyConditionExpression: 'id = :v1 AND unix_created_at > :v2', Limit: days * 24, ScanIndexForward: false, ExpressionAttributeValues: { ':v1': { S: id }, ':v2': { N: unix_created_at }, }, }; return new Promise(function(resolve, reject) { dynamodb.query(params, function(err, data) { if (err) { reject(err, err.stack); // an error occurred } else { if (data.Items.length > 0) { const result = aggItems(data.Items); resolve(result); } else { resolve(null); } } }); }); }
17. Query data with timestamp (API) 4 You can expose AWS Lambda as a HTTP API through API Gateway 4 http://docs.aws.amazon.com/apigateway/latest/ developerguide/api-gateway-create-api-as-simple- proxy-for-lambda.html
18. Query data with timestamp (API) exports.handle = function(e, ctx, cb) { console.log('processing event: %j', e); const requestPath = e.path; const id = 'cathay_jpy_to_ntd'; getLastItems(id).then(function(response) { cb(null, response); }).catch(function(err) { const response = { statusCode: 500, headers: { 'x-lime': 'yoyo', 'Content-Type': 'application/json', }, body: JSON.stringify({ message: 'something went wrong :(' }), }; cb(null, response); }); };
19. Query data with timestamp (API) function getLastItems(id) { return last(id).then(function(items) { const data = items.map(i => ({ value: i.mean })); const result = { postfix: 'JPY TO NTD last 30 days', data }; const body = JSON.stringify(result); const response = { statusCode: 200, headers: { 'x-lime': 'yoyo', 'Content-Type': 'application/json', }, body, }; return response; }); }
20. Visualize data 4 numerics: an iOS app to for all your metrics 4 http://cynapse.com/numerics/
21. Q & A