Scala x AWS Server Side Application Recipe

Scala x AWS Server Side Application Recipe

Transcript

1. Scala × AWS αʔόʔαΠυϨγϐ
2. ࣗݾ঺հ ࿨ా༞հ 1987೥7݄6೔ര஀ ෱Ԭݝ๺۝भࢢখ૔๺۠ খ૔ߴߍ -> ۝भେֶ -> 
 ͍͍ੜ׆ -> Ϋϥεϝιου αʔόαΠυΞϓϦέʔγϣ ϯ։ൃΛओ୲౰
3. ෱Ԭͱ౦ژͷੜ׆ײ ఱਆʹ͍͚͹͍͍ͩͨἧ͏ όεόεΞϯυόε ݪ෇΋ࢹ໺ Ί͠ʢಛʹڕʣ͕͏·͍ ຀ͷ͏ͲΜɺࢿ͞Μ͏ͲΜ Ӻલʹ͍͚͹͍͍ͩͨἧ͏ ి˒ं ΫϩεόΠΫ΋ࢹ໺ ͏·͍Ί͠΋ۚ࣍ୈ ݽಠͷάϧϝͷళʹ͍͚Δ
4. ฏ௩໌ଠࢠͷັྗ ͋Δ౦ژਓʮ෱Ԭͷ໌ଠࢠ ͸ɺͲ͜ͷϝʔΧʔ΋ಉ͡ ຯ͕͢Δʯ
 ↓ ൿ఻ͷλϨͱ10ஈ֊ͷຯ෇ ͚ɾਏ͞ʹΑΓ൒͹ϠέΫ ιͳ͕Βѹ౗తࠩผԽʹ੒ ޭ ฏ௩໌ଠࢠţŪŖžŖ http://www.hira-tsuka.co.jp/hpgen/HPB/entries/4.html
5. ΋͘͡ ֎෦αʔϏεͱͷ͓෇߹͍ 
 - Web API x ElastiCache ϚωʔδϟɺϫʔΧʔͷεϧʔϓοτΛ͍͋͛ͨ
 - Akka Worker x Amazon ECS ΤϯδχΞɺDDoSʹک͑ͳ͍҆Β͔ͳ຾ΓΛ
 - AWS Lambda x AWS WAF
6. Web API × ElastiCache
7. ͱ͋ΔҊ݅ͷ ཁ݅ఆٛձٞʹͯ ؆୯ͳήʔϜͰ ϙΠϯτήοτ εϚʔτϑΥϯΞϓϦ
8. “ϙΠϯτͷ؅ཧ͚ͩͲɺ
 ͏͕ͪ࡞ͬͨϚΠΫϩαʔϏε͕͋ΔΜͩɻ
 ͥͻ࢖ͬͯ͘Εͳ͍͔ͳʁ” – ಉ͡ձࣾͷผ෦ॺͷϙΠϯτ؅ཧγεςϜΛ࡞ͬͨਓ ※ϑΟΫγϣϯͰ͢
9. ϙΠϯτ؅ཧγεςϜͷ ϞόΠϧόοΫΤϯυΛ࡞Δ
10. ϙΠϯτ؅ཧγεςϜͷ ϞόΠϧόοΫΤϯυΛ࡞Δ ͕͜͜ ผαʔϏε
11. Կ͕՝୊͔ 1೔1ճͷϓϨθϯτͰԾ૝௨՟Λ෇༩͢Δ Ծ૝௨՟αʔϏε͸ႈ౳ੑ͕୲อ͞Ε͍ͯͳ͍ ෳ਺୺຤Ͱಉ࣌ʹϓϨθϯτड͚औΓΛ͢Δͱ…
12. ElastiCacheΛ࢖͏ Redis / Memcached ߴύϑΥʔϚϯε ϚωʔδυαʔϏε
13. ElastiCacheΛ࢖͏ Redis / Memcached ߴύϑΥʔϚϯε ϚωʔδυαʔϏε
14. ϚϧναʔόͰͷϩοΫ setnx ͳ͚Ε͹Ωʔ࡞੒ɺ͋Ε͹ࣦഊ ઌউͪͰsetnxͰ͖ͨϦΫΤετ͚͕ͩσΠϦʔϓ ϨθϯτΛड͚औΕΔ
15. ຊ౰ʹϩοΫͰ͖ͯΔͷʁ έʔε ࣮ߦ͞ΕΔ ࣮ߦ͞Εͳ͍ ϩοΫ੒ޭ ɾϢʔβҾ౰ ɾsetnx ɾԾ૝௨՟ Ճࢉ ɾϩοΫղআ ɾͳ͠ ϩοΫࣦഊ ɾϢʔβҾ౰ ɾsetnx ɾԾ૝௨՟ Ճࢉ ɾϩοΫղআ
16. ຊ౰ʹϩοΫͰ͖ͯΔͷʁ έʔε ࣮ߦ͞ΕΔ ࣮ߦ͞Εͳ͍ ϩοΫ੒ޭ ɾϢʔβҾ౰ ɾsetnx ɾԾ૝௨՟ Ճࢉ ɾϩοΫղআ ɾͳ͠ ϩοΫࣦഊ ɾϢʔβҾ౰ ɾsetnx ɾԾ૝௨՟ Ճࢉ ɾϩοΫղআ
17. Unit Test Code @ Specs2 "locked" should { trait Before { self: CommonBefore => mockLockComponent.lock(anyString) returns Left(new IllegalStateException()) } class Context extends CommonContext with Before "throw DailyPresentConflictError" in new Context { val result = service.present(mockUserName) await(result) must throwA[DailyPresentConflictError] } "no update process" in new Context { val result = service.present(mockUserName) await(result) must throwA[Throwable] there was one(mockUserRepository).findByName(anyString) andThen one(mockLockComponent).lock(anyString) andThen no(mockPointRepository).append(anyString, anyInt) andThen no(mockLockComponent).unlock(anyString) } } DailyPresentServiceSpec.scala
18. Unit Test Code @ Specs2 "locked" should { trait Before { self: CommonBefore => mockLockComponent.lock(anyString) returns Left(new IllegalStateException()) } class Context extends CommonContext with Before "throw DailyPresentConflictError" in new Context { val result = service.present(mockUserName) await(result) must throwA[DailyPresentConflictError] } "no update process" in new Context { val result = service.present(mockUserName) await(result) must throwA[Throwable] there was one(mockUserRepository).findByName(anyString) andThen one(mockLockComponent).lock(anyString) andThen no(mockPointRepository).append(anyString, anyInt) andThen no(mockLockComponent).unlock(anyString) } } DailyPresentServiceSpec.scala ϩοΫ࣌Ͱ΋ ࣮ߦ͞ΕΔ͸ͣ
19. Unit Test Code @ Specs2 "locked" should { trait Before { self: CommonBefore => mockLockComponent.lock(anyString) returns Left(new IllegalStateException()) } class Context extends CommonContext with Before "throw DailyPresentConflictError" in new Context { val result = service.present(mockUserName) await(result) must throwA[DailyPresentConflictError] } "no update process" in new Context { val result = service.present(mockUserName) await(result) must throwA[Throwable] there was one(mockUserRepository).findByName(anyString) andThen one(mockLockComponent).lock(anyString) andThen no(mockPointRepository).append(anyString, anyInt) andThen no(mockLockComponent).unlock(anyString) } } DailyPresentServiceSpec.scala ϩοΫ࣌ ࣮ߦ͞Εͳ͍͸ͣ
20. Akka × Amazon ECS
21. “͍΍͊Ϣʔβ૿͖͑ͯͨͶʂͱ͜ΖͰ೔࣌Ͱಈ͍ ͍ͯΔ͋ͷόον͊͞ɺࠓ6͔͔࣌ؒͬͯΔ͡Ό Μɺ1ඵʹҰճ͡Όͳͯ͘0.5ඵʹ1ճʹͯ͠Αɻ ͑ʁσϓϩΠ͕ཁΔͷʁ͡Ό͋σϓϩΠ͠Αʂ” – ͝ػݏͳϚωʔδϟ ※ϑΟΫγϣϯͰ͢
22. όονϫʔΧʔͷঢ়گ 1ඵʹ1ճϙʔϦϯά
23. όονϫʔΧʔͷঢ়گ 1ඵʹ1ճϙʔϦϯά ϙΠϯτ ֫ಘཤྺ
24. όονϫʔΧʔͷঢ়گ 1ඵʹ1ճϙʔϦϯά => 0.5ඵʹ1ճʁ ϙΠϯτ ֫ಘཤྺ
25. σϓϩΠͨ͘͠ͳ͍
26. ൃ૝Λม͑ͯΈΔ ϙʔϦϯάස౓Λ্͛Δ ཁ͢Δʹεϧʔϓοτ͕େ ͖͘ͳΕ͹OK ॲཧ୆਺Λ૿΍͢
27. Կ͕՝୊͔ ೔࣌ͷόονॲཧʹ͔͔Δ͕࣌ؒ૿͖͑ͯͨͨΊɺ ࣌ؒΛ୹ॖ͍ͨ͠ ϙʔϦϯάִؒΛখ͘͢͞Δ͜ͱͰॲཧ࣌ؒͷ୹ ॖ͕ݟࠐΊΔ͕ɺमਖ਼σϓϩΠͨ͘͠ͳ͍ EC2ͷ୆਺Λ૿΍͢͜ͱͰεϧʔϓοτΛେ͖͘ ͠Α͏
28. Կ͕՝୊͔ ೔࣌ͷόονॲཧʹ͔͔Δ͕࣌ؒ૿͖͑ͯͨͨΊɺ ࣌ؒΛ୹ॖ͍ͨ͠ ϙʔϦϯάִؒΛখ͘͢͞Δ͜ͱͰॲཧ࣌ؒͷ୹ ॖ͕ݟࠐΊΔ͕ɺσϓϩΠͨ͘͠ͳ͍ EC2ͷ୆਺Λ૿΍͢͜ͱͰεϧʔϓοτΛେ͖͘ ͠Α͏ EC2ΠϯελϯεΛ૿΍͢ ՝্ֹ͕͕ۚͬͯ͠·͏
29. Amazon ECSΛ࢖͏ Kubernetes ͷ AWS൛ DockerίϯςφʹରԠͯ͠ ͓ΓɺEC2ΠϯελϯεͰ ίϯςφΫϥελΛ૊Ή͜ ͱ͕Ͱ͖Δ ࣮ߦ͢Δίϯςφͷ਺΍ Auto Scaling ͷઃఆ͕Մೳ
30. Amazon ECS Λಋೖ͢Δ
31. Akka Worker ͷ
 Docker Image ࡞੒ lazy val news = project.in(file("./news")) .enablePlugins(PlayScala) .enablePlugins(DockerPlugin) .settings(commonSettings: _*) .settings( name := "ecn-worker-news", version := "0.1.0" ) .settings(libraryDependencies ++= Seq( )) .settings( maintainer in Docker := "Yusuke Wada <wada.yusuke@classmethod.jp>", dockerExposedPorts in Docker := Seq(9000, 9443), dockerRepository := Some("cmwadayusuke") ) .dependsOn(library % “test->test;compile->compile") $ ./activator docker:publish build.sbt shell
32. Akka Worker ͷ Docker Image ࡞੒ lazy val news = project.in(file("./news")) .enablePlugins(PlayScala) .enablePlugins(DockerPlugin) .settings(commonSettings: _*) .settings( name := "ecn-worker-news", version := "0.1.0" ) .settings(libraryDependencies ++= Seq( )) .settings( maintainer in Docker := "Yusuke Wada <wada.yusuke@classmethod.jp>", dockerExposedPorts in Docker := Seq(9000, 9443), dockerRepository := Some("cmwadayusuke") ) .dependsOn(library % “test->test;compile->compile") $ ./activator docker:publish build.sbt shell Docker༻ͷ ৘ใΛఆٛ
33. Akka Worker ͷ Docker Image ࡞੒ lazy val news = project.in(file("./news")) .enablePlugins(PlayScala) .enablePlugins(DockerPlugin) .settings(commonSettings: _*) .settings( name := "ecn-worker-news", version := "0.1.0" ) .settings(libraryDependencies ++= Seq( )) .settings( maintainer in Docker := "Yusuke Wada <wada.yusuke@classmethod.jp>", dockerExposedPorts in Docker := Seq(9000, 9443), dockerRepository := Some("cmwadayusuke") ) .dependsOn(library % “test->test;compile->compile") $ ./activator docker:publish build.sbt shell activator (sbt ϥούʔ)ͷ ίϚϯυΛ࣮ߦ
34. ίϯςφ਺Λ੍ޚ͢Δ αʔϏεͱ͍͏ઃఆ߲໨Ͱ ίϯςφ਺ͷ੍ޚ͕Մೳ Կ୆ͷEC2ΫϥελΛ૊Ή ͔ɺͱ͍͏͜ͱ΋ઃఆՄೳ 1୆ͷEC2্ʹෳ਺ίϯςφ Λ্ཱͪ͛Δ͜ͱͰεϧʔ ϓοτ͕͕͋Δ͸ͣ
35. ίϯςφ਺vsεϧʔϓοτ SQSͷϝοηʔδΛಡ ΜͰDynamoDBʹॻ͘ ϝοηʔδ਺͸100 100ϝοηʔδফԽͷ εϧʔϓοτΛܭଌ ॲཧϝοηʔδ਺ / ඵ 0 1.25 2.5 3.75 5 1ίϯςφ 2 3 4
36. ίϯςφ਺vsεϧʔϓοτ SQSͷϝοηʔδΛಡ ΜͰDynamoDBʹॻ͘ ϝοηʔδ਺͸100 100ϝοηʔδফԽͷ εϧʔϓοτΛܭଌ ॲཧϝοηʔδ਺ / ඵ 0 1.25 2.5 3.75 5 1ίϯςφ 2 3 4 ઢܗʹ εϧʔϓοτΞοϓ
37. ίϯςφ਺vsεϧʔϓοτ SQSͷϝοηʔδΛಡ ΜͰDynamoDBʹॻ͘ ϝοηʔδ਺͸100 100ϝοηʔδফԽͷ εϧʔϓοτΛܭଌ ॲཧϝοηʔδ਺ / ඵ 0 1.25 2.5 3.75 5 1ίϯςφ 2 3 4 EC2Πϯελϯε͸Ұ୆ ՝ֹۚ͸มΘΒͳ͍
38. AWS LAMBDA × AWS WAF
39. “DDoSΛ৯ΒͬͨɻೝূγεςϜ͕མͪͨɻ
 Τϯυ͔ΒʮϩάΠϯͰ͖ͳ͍ʯͱ͍͏໰͍߹Θ ͕͖͍ͤͯΔΒ͍͠ɻ͓Εͨͪ͸΋͏ͩΊͩɻ” – ઈ๬ͷϚωʔδϟ ※ϑΟΫγϣϯͰ͢
40. Կ͕՝୊͔ DDoSΛड͚ͨɻ
 ߈ܸऀ͸User-AgentΛِ૷͍ͯ͠Δ ͻͱ·ͣ Network ACL Ͱ౰֘IPΞυϨεΛःஅ͠ ͕ͨɺ߈ܸऀ͸͙͢ʹΞΫηεݩIPΞυϨεΛม ͑ͯ͘Δ Πλνͬ͜͝Ͱӡ༻ΤϯδχΞ΋ফ໣
41. “ͱΓ͋͑ͣ AWS WAF ΛೖΕ·͠ΐ͏ɻ
 AWS Lambda ͱ૊Έ߹ΘͤΕ͹
 ࣗಈ๷ޚػߏ͕ͭ͘Εͦ͏Ͱ͢ɻ” –ΤϯδχΞ ※ϑΟΫγϣϯͰ͢
42. ALB + EC2 ͱ͍͏ߏ੒
43. AWS WAF ࣗಈઃఆͷ࡞ઓ
44. ᶃ AWS WAF ͷಋೖ
45. ᶃ AWS WAF ͷಋೖ AWS WAF ΛηοτΞοϓɻःஅର৅ͷ
 IP Addresses ͸ͻͱ·ۭͣʹ
 →͜͜Λ AWS Lambda Ͱࣗಈઃఆ AWS WAF Λ CloudFront ʹؔ࿈͚ͮΔ
46. ᶄ fluentdͰΞΫηεϩά
47. ᶄ fluentdͰΞΫηεϩά nginx X-Forwarded-For Ͱ
 ΞΫηεݩIPΛग़ྗ͢ΔΑ͏ઃఆ fluentd nginxͷΞΫηεϩάΛύʔε ύʔεޙͷϩάΛS3΁Ξοϓ
48. tg-agent.conf.j2ʢҰ෦ ʣ <match log.webapi.access> type s3 s3_bucket "webapi-access-log" path "webapi/access/" s3_object_key_format %{path}%{time_slice}/access_%{index}.% {file_extension} buffer_path /var/log/td-agent/s3/webapi_access time_slice_format %Y/%m/%d/%H time_slice_wait 1m utc buffer_chunk_limit 512k format json include_time_key true time_key log_time </match> tg-agent.conf.j2
49. tg-agent.conf.j2ʢҰ෦ ʣ <match log.webapi.access> type s3 s3_bucket "webapi-access-log" path "webapi/access/" s3_object_key_format %{path}%{time_slice}/access_%{index}.% {file_extension} buffer_path /var/log/td-agent/s3/webapi_access time_slice_format %Y/%m/%d/%H time_slice_wait 1m utc buffer_chunk_limit 512k format json include_time_key true time_key log_time </match> tg-agent.conf.j2 1࣌ؒ΋͘͠͸ 512KB͝ͱʹϩάΛ S3΁
50. S3όέοτʹΞοϓ͞ΕΔIPΞυϨεϦετ {"host":"5.24.105.49","log_time":"2017-01-16T00:07:40Z"} {"host":"245.52.215.137","log_time":"2017-01-16T00:07:42Z"} {"host":"9.13.105.49","log_time":"2017-01-16T00:07:49Z"} {"host":"4.44.105.49","log_time":"2017-01-16T00:07:50Z"} {“host":"678.124.40.68","log_time":"2017-01-16T00:08:07Z"} {"host":"5.24.105.49","log_time":"2017-01-16T00:07:40Z"} {"host":"245.52.215.137","log_time":"2017-01-16T00:07:42Z"} {"host":"9.13.105.49","log_time":"2017-01-16T00:07:49Z"} {"host":"4.44.105.49","log_time":"2017-01-16T00:07:50Z"} {“host":"678.124.40.68","log_time":"2017-01-16T00:08:07Z"} access.log
51. ᶅ AWS Lambda ߏங
52. ᶅ AWS Lambda ߏங S3όέοτ΁ͷϩάసૹΛτϦΨʔʹىಈ͢Δ
53. Lambda Function ຊମΛॻ͘ def handle(event, context): # Get the object from the event and show its content type bucket = event['Records'][0]['s3']['bucket']['name'] key = urllib.unquote_plus(event['Records'][0]['s3']['object'] ['key'].encode('utf8')) try: # open file, unzip s3.download_file(bucket, key, '/tmp/file.gz') f = gzip.open('/tmp/file.gz', 'rb') content = f.read() f.close # create black list ip_address_list = to_ip_address_list(content.decode('utf-8')) black_list = to_ip_black_list(ip_address_list) main.py
54. Lambda Function ຊମΛॻ͘ def handle(event, context): # Get the object from the event and show its content type bucket = event['Records'][0]['s3']['bucket']['name'] key = urllib.unquote_plus(event['Records'][0]['s3']['object'] ['key'].encode('utf8')) try: # open file, unzip s3.download_file(bucket, key, '/tmp/file.gz') f = gzip.open('/tmp/file.gz', 'rb') content = f.read() f.close # create black list ip_address_list = to_ip_address_list(content.decode('utf-8')) black_list = to_ip_black_list(ip_address_list) main.py IPΞυϨε͚ͩ நग़
55. Lambda Function ຊମΛॻ͘ def handle(event, context): # Get the object from the event and show its content type bucket = event['Records'][0]['s3']['bucket']['name'] key = urllib.unquote_plus(event['Records'][0]['s3']['object'] ['key'].encode('utf8')) try: # open file, unzip s3.download_file(bucket, key, '/tmp/file.gz') f = gzip.open('/tmp/file.gz', 'rb') content = f.read() f.close # create black list ip_address_list = to_ip_address_list(content.decode('utf-8')) black_list = to_ip_black_list(ip_address_list) main.py ूܭɾ͖͍͠஋ ൑ఆ
56. if len(black_list): # create ip set for waf update_ip_set=map(to_ip_set_for_waf, black_list) # create sns settings sns_settings=to_sns_notification_settings(black_list) # waf setting (side-effect function) waf_block(update_ip_set) # notification (side-effect functions) sns_notification(sns_settings) if slack_notification_enabled: slack_notification(black_list) else: pass else: print(u'nothing to do.') pass return black_list except Exception as e: print(e) raise e
57. if len(black_list): # create ip set for waf update_ip_set=map(to_ip_set_for_waf, black_list) # create sns settings sns_settings=to_sns_notification_settings(black_list) # waf setting (side-effect function) waf_block(update_ip_set) # notification (side-effect functions) sns_notification(sns_settings) if slack_notification_enabled: slack_notification(black_list) else: pass else: print(u'nothing to do.') pass return black_list except Exception as e: print(e) raise e AWS WAF API ίʔϧ
58. if len(black_list): # create ip set for waf update_ip_set=map(to_ip_set_for_waf, black_list) # create sns settings sns_settings=to_sns_notification_settings(black_list) # waf setting (side-effect function) waf_block(update_ip_set) # notification (side-effect functions) sns_notification(sns_settings) if slack_notification_enabled: slack_notification(black_list) else: pass else: print(u'nothing to do.') pass return black_list except Exception as e: print(e) raise e ௨஌
59. ಉҰIP͔ΒͷΞΫηε͕Ұఆͷ͖͍͠஋ʢLambda Function ͷ؀ڥม਺ͰઃఆͰ͖ΔΑ͏ʹͨ͠ʣʹ ୡ͢ΔͱϒϥοΫϦετ൑ఆ͠ɺ AWS WAF ͷϒ ϩοΫIPΛࣗಈͰߋ৽͢Δ
60. λΠϜϥΠϯ 08:00 
 ىচɺΞϥʔτ·ΈΕͷSlack 10:00 
 ग़ࣾɺϚωʔδϟʔ૵ന 12:00 
 ΞʔΩςΫνϟܾఆɺ࣮૷։࢝ 15:00 
 ࣮૷׬ྃɺಈ࡞ݕূ։࢝ 17:00 
 ಈ࡞ݕূ׬ྃɺ
 εςʔδϯά؀ڥʹσϓϩΠ ཌ11:00 ຊ൪ϦϦʔε ཌ18:00 υϠؼ୐
61. ͜Μͳײ͡Ͱ ָ͘͠΍ͬͯ·͢ ࠷৽ͷٕज़Λ࣮Ҋ݅΁ੵۃతʹ౤ೖ͍ͨ͠ʂ AWSɺαʔόʔαΠυʹؔ͢Δ༇շͳΤϯδχΞͱҰॹʹಇ͖ ͍ͨʂ Developers.IOͰϒϩάॻ͖͍ͨʂ εΩϧ σβΠϯ / iOS / Android Java / Scala / Ruby / node.js / Go && AWS ϞόΠϧόοΫΤϯυΤϯδχΞ ― https://classmethod.jp/recruit/jobs/aws-web-app-mobile-engineer/
62. Scalaͱ͍͏ݴޠʹ͍ͭͯ
63. Scalaͱ͍͏ݴޠʹ͍ͭͯ