Upgrade to Pro — share decks privately, control downloads, hide ads and more …

サーバーレス x IoT 〜我々はどういった課題に直面してそれをどのように解決したのか〜

Koji Nakayama
April 06, 2018
1.2k

サーバーレス x IoT 〜我々はどういった課題に直面してそれをどのように解決したのか〜

AWS Deep Night Part.2の登壇資料です。

Koji Nakayama

April 06, 2018
Tweet

Transcript

  1. ᛔ૩奧Օ • Ӿઊ ଛလ • μ϶φϮϊϐϖ ϯϝαϸίϤϷςЄϠφ᮱ • AWSϊϷϲЄτϴЀίЄκϓμϕ •

    ςЄϝЄςαϖεЀυϘί • GitHub: knakayama • 奺䵉 • ηЀϤϹςЄϝ΄晁አ3ଙ • AWS 2ଙ 2
  2. ίЄκϓμώϰ༷ᥝ • ᓕቘᘏአϧЄυ΅CloudFront + S3ͽSPA̵ εЀϖϳЄσ΅ϯϝαϸίϤϷ • ϝϐμεЀϖAPI΅API Gateway +

    Lambda + DynamoDB • Cognito + API Gateway΄θφόϭηЄϊ϶ ασΨڥአͭ͵扯戣/扯ݢ • IoTϔϝαφ΅AWS IoT΁ϔЄόΨPublish̵ Device Shadow奺ኧͽϔϝαφΨ඙֢ • Kinesis Data Streams + LambdaΨڥአͭͼ S3΁ړຉአϔЄόכਂ 6
  3. Θ͜੝ͭͪ奧ՕͯΡ; • Lambda樛හ΄϶Ѐόαϭ΅Python • 䯤౮ᓕቘ΅CloudFormation/AWS SAM • CI/CD΁CircleCIΨڥአͭͼ͚Ρ1 • Lambda樛හ΄Ϻνᓕቘ΅Logentries2

    • ϮϕϷμφ΅Datadogͽ哶憙 2 https://dev.classmethod.jp/devops/managing-logentries-token-with-terraform-and-ssm-parameter-store/ 1 https://dev.classmethod.jp/server-side/serverless/practical-ci-cd-with-aws-sam-circleci-and-localstack/ 7
  4. ᥝͯΡ΁ • AWS IoTӤ΁ਂࣁͯΡ 㫪మጱ΀ ϔϝαφ • ϳЄσ憙ᅩ͡Ο憎Ρ;ͩ΄ϔϝαφ΅㶨΀ΡJSON • ίϤϷ

    - IoTϔϝαφ樌΄᭗מ΅Device ShadowΨլՕͭͼᤈ͜ • Ӟᛱጱ΁IoTϔϝαφ - Device Shadow樌΅MQTT over TLS̵ί ϤϷ - Device Shadow樌΅HTTPS 11
  5. ͘ΡΏͣᇫ䙪Ψᭆמ • ίϤϷ͡Ο͘ΡΏͣᇫ䙪( desired )Ψᭆ מ • ֺ͞Ά櫮რͺͧͼѺ • API:

    UpdateThingShadow { "reported": { "power": "off" }, "desired": { "power": "on" } } 13
  6. Θ͜੝ͭ奞͚ͥ͜͡; • ͘ΡΏͣᇫ䙪( desired );匍ࣁ΄ᇫ䙪( reported )ΨJSON୵ୗͽכਂ̵ͭͳ΢΁ ૧ړ( delta )͢ਂࣁͭ͵䁰ݳ̵IoTϔϝαφ͘͢ΡΏͣᇫ䙪΁ݶ๗ݢᚆ

    • IoTϔϝαφ΅MQTTϕϡϐμΎϮϐψЄυΨPublishͭͼDevice Shadow΁ reported Ψᭆמ • ᭋ΁ϕϡϐμΨSubscribeͯΡͩ;ͽ delta ΨݑͧݐΠ̵ desired ΄ᇫ䙪΁ݶ๗ • Device ShadowΨ඙֢ͯΡϕϡϐμ΅AWS IoTͽԪڹ΁䷥Δ͹ͼ͚Ρ • AWS IoT΅ϮϐψЄυΨᓕቘͯΡ(ϮϐψЄυϣϺЄθЄ)Ͷͧ΀΄ͽ̵ϔϝα φ㯎΁ݶ๗ͯΡ͵Η΄䋚ᤰ͢஠ᥝ 15
  7. ֜᩸ͣ͢͵΄͡ • Device Shadow΁΅ݘ͚ desired ͽӤ䨗ͣͭ΀͚Ξ͜΁Ϻϐμ䱛䯤͢ਂࣁͯ Ρ • Device ShadowΨๅෛͯΡ͵Ή΁αЀμϷϮЀϕͫ΢ΡϝЄυϴЀ΁ΞΡ

    Ϻϐμ • Device Shadow΄ๅෛ͢ݶ䦒΁咲ኞͭ͵͵Η Version conflict ε϶Є͢咲 ኞͭ͵ • ͺΔΠ̵IoTϔϝαφ΁ΞΡ毱ᔺ΀Device Shadow΄ๅෛ΁Ξ͹ͼ̵ίϤϷ ΁ΞΡUpdateThingShadow͢०䤂ͭΚ͚ͯᇫ䙪΁΀͹ͼ͚͵ 19
  8. Ϳ͜ᥴ䷥ͭ͵΄͡ • ϯϝαϸίϤϷͽ಩ൎͭͼͣ͠͵͚ϔЄό;ͳ ͜ͽ΀͚Θ΄ͽPublishͯΡϕϡϐμΨړ櫝ͭ͵ • ಩ൎͭͼͣ͠͵͚ϔЄό: εЀϖϳЄσ΁憎ͱ ͵͚ϔЄό(櫮რ/Ⴥଶ΀Ϳ) • ͳ͜ͽ΀͚ϔЄό:

    㶨΀ΡϔЄόړຉአ΄ϔЄ ό • ͳ͜ͽ΀͚ϔЄό΅Device Shadowአϕϡϐμͽ ΅΀̵ͥ㳨΄ϕϡϐμ΁PublishͯΡΞ͜΁䄜ๅ • ֺ: my/things/<thingName>/state • ͺΔΠ̵ӧᥝ΀Device Shadow΄ๅෛΨಪګͭ͵ 20
  9. ίЄκϓμώϰ • 吖ଉᓕቘϓЄϣϸ΁ϔЄό͢PutItemͫ ΢͵ΟDynamoDB StreamsͽLambda樛 හΨ᩸㵕 • ͳ΄Lambda樛හ͢Step FunctionsΨ᩸㵕 •

    ๋ڡ΁5ړ樌Wait஍̵ϷϫαЀϖڣਧ Lambda樛හΨ᩸㵕ͭ᭗ᎣͯΏͣ͡ڣਧ • ᭗ᎣͯΏͣ䁰ݳ΅εЀϖϳЄσ΁Ϸϫα Ѐϖ • ͩ΢Ψ೰ਧࢧහ媗Πᬬͯ 31
  10. ϸЄϤࢧහ΄ڡ๗戔ਧ ... "States": { "ConfigureRemindCount": { "Type": "Pass", "Result": {

    "count": 5, "index": 0, "step": 1 }, "ResultPath": "$.iterator", "Next": "WaitAMinutes" }, "WaitAMinutes": { "Type": "Wait", "Seconds": 300, "Next": "RemindIterator" }, 32
  11. ೰ਧ䦒樌Wait ... "States": { "ConfigureRemindCount": { "Type": "Pass", "Result": {

    "count": 5, "index": 0, "step": 1 }, "ResultPath": "$.iterator", "Next": "WaitAMinutes" }, "WaitAMinutes": { "Type": "Wait", "Seconds": 300, "Next": "RemindIterator" }, 33
  12. ϷϫαЀϖ΄ڣਧ ... "RemindIterator": { "Type": "Task", "Resource": "<_LAMBDA_ARN_>", "ResultPath": "$.iterator",

    "Next": "IsRemindCountReached" }, "IsRemindCountReached": { "Type": "Choice", "Choices": [ { "Variable": "$.iterator.continue", "BooleanEquals": true, "Next": "Remind" } ], "Default": "Done" }, 34
  13. ϷϫαЀϖڣਧ΄πЄϖ ... def handler(event, context): iterator = event['iterator'] dynamodb =

    boto3.resource('dynamodb').Table(os.environ['TABLE_NAME']) result = dynamodb.get_item(Key={'yourKey': event['yourValue']}) if not result.get('Item'): return {'continue': False} if iterator['index'] <= iterator['count']: continue_ = True else: continue_ = False return { 'index': iterator['index'] + iterator['step'], 'step': iterator['step'], 'count': iteratoro['count'], 'continue': continue_ } 35
  14. ϸЄϤΨ姅姆ͯΡ͡ڣਧ ... "RemindIterator": { "Type": "Task", "Resource": "<_LAMBDA_ARN_>", "ResultPath": "$.iterator",

    "Next": "IsRemindCountReached" }, "IsRemindCountReached": { "Type": "Choice", "Choices": [ { "Variable": "$.iterator.continue", "BooleanEquals": true, "Next": "Remind" } ], "Default": "Done" }, 36
  15. ϓφϕ΁ڥአͭͼ͚ΡϯυϲЄϸ • ϓφϕϢϹЄϭϼЄμ: pytest5 • 䰤伛΄unittestϯυϲЄϸΞΠṛ䱛ᚆ͹Γ͚΄ͽ • AWS IoT΄ϯϐμ: moto4

    • E2E: AWS IoT Device SDK for Python6 • paho7ΞΠು᨝۸ͫ΢ͼ͚ͼֵ͚Κͯ͡͹͵ 7 http://www.eclipse.org/paho/ 6 https://github.com/aws/aws-iot-device-sdk-python 4 https://github.com/spulec/moto 5 https://github.com/pytest-dev/pytest 42
  16. ϳϘϐϕϓφϕ΄䜐ኼ • motoΨڥአͭͼBoto3΄IoT8/IoTDataPlane9Ψϯϐμ۸ • pytest΄ conftest.py ͽfixutreΨਧ嬝ͭͼͥ͠ • ݱ圵ϓφϕξЄφͽfixtureΨ޷Ήڊͭͼϓφϕ๵կΨ戔ਧͯΡ •

    ϓφϕξЄφΨϞ϶ϮЄό۸ͭͼτЀϤϸ΀懿ᬿͽϓφϕθϝ ϹϐυΨṛΗΡ 9 http://boto3.readthedocs.io/en/latest/reference/services/iot-data.html 8 http://boto3.readthedocs.io/en/latest/reference/services/iot.html 44
  17. ϓφϕ䌏᨝΄Ϯϊϐϖ • ᇙਧ΄Thing͢ਂࣁͯΡ͡ڣਧͭͼ͚ΡͶͧ class IoTHandler(object): ... def _does_thing_exist(self, thing_name): response

    = self.iot.describe_thing(thingName=thing_name) if thing_name == response.get('thingName'): return True else: return False 45
  18. ϳϘϐϕϓφϕ΄πЄϖ class TestDoesThingExist(object): @pytest.mark.usefixtures('create_thing') @pytest.mark.parametrize( 'thing_name, expected', [ ('my_thing_01', True),

    ('my_thing_02', False) ]) def test_some_things_exist(self, thing_name, expected): actual = IoTHandler()._does_thing_exist(thing_name) assert actual is expected 46
  19. ϓφϕ䌏᨝΄Ϯϊϐϖ • Boto3΄ഄΡᛩ͚Ψϯϐμͫͱ͵͚ • Δ͵̵Thingͯ͢ͽ΁ਂࣁͭͼ͚Ρᇫ丆Ψ֢Π͵͚ class IoTHandler(object): ... def _does_thing_exist(self,

    thing_name): response = self.iot.describe_thing(thingName=thing_name) if thing_name == response.get('thingName'): return True else: return False 47
  20. ϳϘϐϕϓφϕ΄πЄϖ • ϓφϕͭ͵͚ڹ൉๵կ(Thingͯ͢ͽ΁ਂࣁͭͼ͚Ρ)Ψෆ͞Ρ class TestDoesThingExist(object): @pytest.mark.usefixtures('create_thing') @pytest.mark.parametrize( 'thing_name, expected', [

    ('my_thing_01', True), ('my_thing_02', False) ]) def test_some_things_exist(self, thing_name, expected): actual = IoTHandler()._does_thing_exist(thing_name) assert actual is expected 48
  21. ThingΨԪڹ΁֢౮( conftest.py ) from moto import mock_iot @pytest.fixture(scope='function') def create_thing(request):

    mock = mock_iot() mock.start() iot = boto3.client('iot') iot.create_thing(thingName='my_thing_01') request.addfinalizer(lambda: mock.stop()) 49
  22. ϓφϕ䌏᨝΄Ϯϊϐϖ • Thing͢ਂࣁͯΡ͡ڣਧͯΡړઓΨϓφϕͭ͵͚ class IoTHandler(object): ... def _does_thing_exist(self, thing_name): response

    = self.iot.describe_thing(thingName=thing_name) if thing_name == response.get('thingName'): return True else: return False 50
  23. ϳϘϐϕϓφϕ • ϓφϕͭ͵͚ξЄφΨϞ϶ϮЄό۸ class TestDoesThingExist(object): @pytest.mark.usefixtures('create_thing') @pytest.mark.parametrize( 'thing_name, expected', [

    ('my_thing_01', True), ('my_thing_02', False) ]) def test_some_things_exist(self, thing_name, expected): actual = IoTHandler()._does_thing_exist(thing_name) assert actual is expected 51
  24. ϳϘϐϕϓφϕ΄πЄϖ • 奾ຎΨϓφϕ class TestDoesThingExist(object): @pytest.mark.usefixtures('create_thing') @pytest.mark.parametrize( 'thing_name, expected', [

    ('my_thing_01', True), ('my_thing_02', False) ]) def test_some_things_exist(self, thing_name, expected): actual = IoTHandler()._does_thing_exist(thing_name) assert actual is expected 52
  25. E2Eϓφϕ΄䜐ኼ • ϳϘϐϕϓφϕ(;͚͜͡moto)ͽ΅䌏䖕ͽͣ΀͚塅㾨Ψϓφϕ ͯΡ • AWS IoT΁樛ͭͼ͚͜;AWS IoT Rule Engine΄ഄΡᛩ͚

    • fixtureͽAWS IoT Device SDK for PythonΨ̵ֵ͚ϮϐψЄυΨ Publishͫͱͼͥ͠ • 奾ຎΨώδϐμͭͼϓφϕ 55
  26. ϮϐψЄυ΄Publish @pytest.mark.parametrize( 'publish_message', [ (({ 'state': { 'reported': { 'someIllegalState':

    True } } },)) ], indirect=True) def test_some_illegal_states_occured(publish_message): ... SELECT *, topic(3) as device_id FROM '$aws/things/+/shadow/update' WHERE state.reported.someIllegalState = True 58
  27. conftest.py @pytest.fixture(scope='function') def publish_message(request): for message in request.param: publish2topic(iot_endpoint='_IOT_ENDPOINT_', topic=f'$aws/things/{my_thing}/shadow/update',

    message=message, iot_credentials='_IOT_CREDENTIALS_') time.sleep(10) def publish2topic(iot_endpoint, topic, message, iot_credentials): mqtt = AWSIoTMQTTClient('api-test') mqtt.configureEndpoint(iot_endpoint, 8883) mqtt.configureCredentials(iot_credentials['root_ca'], iot_credentials['private_key'], iot_credentials['certificate']) mqtt.connect() mqtt.publish(topic, json.dumps(message), 1) 59