How to play Scala on dockerized infrastructure

At ScalaMatsuri 2016

Transcript

1. HOW TO SCALA ON DOCKERIZED INFRASTRUCTURE NAOKI AINOYA RECRUIT MARKETING PARTNERS CO, LTD. SCALA MATSURI 2016
2. ▸ Naoki Ainoya ▸ Server-side / Ops / 
 iOS App developer ABOUT ME
3. ABOUT US ▸ Recruit Marketing Partners Co, Ltd ▸ Our business domain is in
 "Life Event"
4. OUR PRODUCTS ▸ English education ▸ ӳ୯ޠαϓϦ (iOS/Android) ▸ ӳޠαϓϦ (iOS/Web)
5. TECHNOLOGY STACK CLIENT-SIDE: IOS(SWIFT)/ANDROID/WEB(ANGULARJS) SERVER-SIDE: SCALA(PLAY FRAMEWORK) INFRASTRUCTURE: DOCKER(AWS/EC2 CONTAINER SERVICE) Client Server Infrastructure
6. BACKEND ARCHITECTURE ▸ RESTful API Servers on EC2 Container Service (ECS) ▸ Servers are properly decoupled by docker containers Client AUTH CS ETC.. PUBLIC API ADMIN Docker Containers Data HTTPS MySQL Elasticache S3 αʔό͸%PDLFSίϯςφʹΑͬͯ෼ׂɾૄ݁߹Խ
7. WHY DO WE USE SCALA ? ▸ Scalable to develop ▸ Easy to adopt DDD style ▸ Easy to refactor 4DBMB͸։ൃΛεέʔϧͤ͞΍͍͢
8. EASY TO ADOPT DDD STYLE ▸ Easy to make DDD-style architecture ▸ Value object is defined as "object" ▸ Entity is defined as "case class" ▸ Separate layers with "protected" PRESENTATION DOMAIN DATA ACCESS Layered architecture ཧ༝ͷҰͭ͸%%%ͱ਌࿨ੑ͕ߴ͍͜ͱ
9. EASY TO ADOPT DDD STYLE ▸ DDD-style makes good habits of Separation of Concerns (SoC) ▸ It keeps codes clean %%%Λҙࣝͨ͠։ൃͰίʔυ͕៉ྷʹอͨΕ΍͘͢ͳͬͨ PRESENTATION DOMAIN DATA ACCESS Layered architecture
10. EASY TO REFACTOR ▸ Functional feature ▸ Static typing ϦϑΝΫλͷ͠΍͢͞΋ॏཁɿؔ਺ܕݴޠɾ੩తܕ෇͚
11. FUNCTIONAL FEATURE HELPS CODE MAINTAINABLITY ▸ Minimize side-effects ▸ Handle errors in functional way ▸ Either \/ monad in Scalaz ▸ ScalazΛ࢖͓͏ #1 – NET BIZ DIV. TECH BLOG 
 http://tech.recruit-mp.co.jp/server-side/post-2540/ Ϟφυ΍ؔ਺ܕͷಛ௃Λ׆༻ͯ͠มߋʹڧ͍ίʔυΛॻ͘
12. I DON'T HAVE TO EMAIL EVERYONE IF I CHANGE FUNCTION NAMES ANYMORE. Developer STATIC TYPING ੩తܕ෇͚͸νʔϜ։ൃͷॿ͚ʹͳΔ
13. SCALABLE TO DEVELOP ▸ Keep development stable against increasing developers 2015/01 1 developer 3 developers 6+ developers 2015/04 Now ݁Ռͱͯ͠ϝϯόʔ͕ٸʹ૿͑ͯ΋҆શʹ։ൃͰ͖ͨ
14. WHY DO WE USE DOCKER ? ▸ Make a infrastructure immutable
 like a Scala code ▸ Easy to keep an application environment clean ▸ Easy to deploy %PDLFSΛ࢖ͬͯΠϯϑϥ΋Πϛϡʔλϒϧͳӡ༻Ͱ
15. EASY TO KEEP AN APPLICATION ENVIRONMENT CLEAN ▸ Easy to know in what environment server is running for devs ▸ Platform ▸ JDK version ▸ Timezone AUTH CS ETC.. PUBLIC API ADMIN Production Environment DOCKER IMAGE Run containers %PDLFSʹΑͬͯ։ൃऀ͕ຊ൪؀ڥͷ༷ࢠΛ೺Ѳ͠΍͘͢ͳΔ
16. EASY TO DEPLOY ▸ Simplify deploy system ▸ We only have to know how to
 docker run <image>
 in production AUTH CS ETC.. PUBLIC API ADMIN Production Environment CODE Build Image &
 Deploy Devs ༰қͳσϓϩΠ͕Մೳ
17. HOW ARE WE OPERATING SCALA/PLAY DEVELOPMENT TEST DEPLOYMENT SCHEDULED JOB DATABASE MANAGEMENT MONITORING
18. DEVELOPMENT ▸ IntelliJ ▸ Great code completion ▸ Easy to debug ▸ Use MySQL container in local ▸ If It's needed
 (for migration test etc…) Docker Container
 in docker-machine
 (Virtualbox) H2 Database Unit test (In-memory DB) Switch
 if needed ։ൃʹ͸JOUFMMJ+Λར༻ɺ%#͸ඞཁʹԠͯ͡੾Γସ͑
19. TEST ▸ Unit test ▸ Specs2 ▸ with using H2Database
 (in-memory DB) ▸ Jenkins CI ▸ Run unit tests ▸ with using MySQL Devs Develop + Write Test Run tests each commits git push webhook +FOLJOT্Ͱͷςετ࣮ߦ͸.Z42-ͱ઀ଓ
20. COMPILATION TIME ▸ It takes much time ! ▸ Don't use sbt clean 
 if it's not needed ▸ Use incremental compile function if possible Clean & Test Test Test Test Clean & Test develop feature-A ίϯύΠϧ஗͘ͳΔͷͰ$MFBO͸ඞཁͳ࣌ʹ͔͠͠ͳ͍
21. DEPLOYMENT develop feature-A master DEPLOY
22. DEPLOYMENT develop feature-A master Build Docker image
 with
 sbt-native-packager ./activator docker:publish TCUOBUJWFQBDLFSͰ%PDLFS
    JNBHFΛϏϧυ
23. DEPLOYMENT develop feature-A master Build Docker image
 with
 sbt-native-packager private registry Push docker image ./activator docker:publish ͦͷ··+FOLJOT্͔ΒQSJWBUF
    SFHJTUSZ΁ΠϝʔδΛQVTI
24. DEPLOYMENT develop feature-A master Build Docker image
 with
 sbt-native-packager private registry Push docker image ECS Upgrade
 services
 via ECS API &$4
    "1*Λୟ͍ͯίϯςφΛσϓϩΠ
25. DEPLOYMENT develop feature-A master Build Docker image
 with
 sbt-native-packager private registry Push docker image ECS Upgrade
 services
 via ECS API pull
26. DEPLOYMENT develop feature-A master Build Docker image
 with
 sbt-native-packager private registry Push docker image ECS Upgrade
 services
 via ECS API pull AUTH CS ETC.. PUBLIC API ADMIN Production Environment
27. DEPLOYMENT develop feature-A master Build Docker image
 with
 sbt-native-packager private registry Push docker image ECS Upgrade
 services
 via ECS API pull AUTH CS ETC.. PUBLIC API ADMIN Production Environment Switch server's role
 by application.conf ԿͷαʔόΛσϓϩΠ͢Δ͔͸DPOGϑΝΠϧͰࢦఆ
28. CONFIGURATION MANAGEMENT ECS AUTH CS ETC.. PUBLIC API ADMIN Production Environment Switch server's role
 by application.conf ▸ Prepare config files each server roles ▸ prod_api.conf/ prod_admin.conf/ prod_auth.conf etc… ▸ Choose config file with option
 JAVA_OPTS=
 -Dconfig.resource=prod_api.conf αʔόͷछྨ͝ͱʹDPOGϑΝΠϧΛ༻ҙ
29. CONFIGURATION MANAGEMENT ▸ application.conf can also switch routing settings play { http { router = route.api.Routes } } prod_api.conf ECS AUTH CS ETC.. PUBLIC API ADMIN Production Environment Switch server's role
 by application.conf DPOGϑΝΠϧͰΤϯυϙΠϯτͷ੾Γସ͑΋ߦ͏
30. CONFIGURATION MANAGEMENT ▸ application.conf can also switch routing settings play { http { router = route.api.Routes } } prod_api.conf # Routes api. # ~~~~ -> / route.partial.api.Routes -> / route.partial.common.Routes route.api.routes
31. CONFIGURATION MANAGEMENT ▸ application.conf can also switch routing settings play { http { router = route.api.Routes } } prod_api.conf # Routes api. # ~~~~ -> / route.partial.api.Routes -> / route.partial.common.Routes route.api.routes # partial Routes api. # ~~~~ POST /api/v1/publish_id controllers.api.Auth.publishId
 … route.partial.api.routes
32. CONFIGURATION MANAGEMENT ▸ application.conf can also switch routing settings play { http { router = route.api.Routes } } prod_api.conf # Routes api. # ~~~~ -> / route.partial.api.Routes -> / route.partial.common.Routes route.api.routes # partial Routes common.
 # ~~~~ GET / controllers.Application.index # Map static resources from the /public folder to the /assets URL path
 
 … route.partial.common.routes NOTE: If endpoint is duplicated, 
 Scala compile fails 3PVUFTϑΝΠϧΛڞ௨Խ͢Δ
33. SCHEDULED JOB ▸ Daily batch job ▸ Daily KPI reporting ▸ Scheduled by Jenkins Executes play application
 by one shot όον࣮ߦʹ͸+FOLJOTΛར༻
34. EXECUTE PLAY APPLICATION BY ONE SHOT ▸ Create application jar file
 ./sbt clean assembly ▸ Execute jar
 Executes play application
 by one shot java -Dconfig.resource=batch.conf 
 -cp ./target/scala-2.11/app-x.x.jar
 tasks.SomeTask ίϯύΠϧࡁΈKBSΛεέδϡʔϧ࣮ߦ͢Δ
35. EXECUTE PLAY APPLICATION BY ONE SHOT Executes play application
 by one shot java -Dconfig.resource=batch.conf 
 -cp ./target/scala-2.11/app-x.x.jar
 tasks.SomeTask package tasks import com.typesafe.scalalogging.LazyLogging object SomeTask extends App with Task with LazyLogging { withApplication( app => { doSomething() // Do something }, ex => ex match { case ex => logger.error(ex.getMessage, ex) println("Something went wrong") } ) } ͍ͤͨ͞όονॲཧΛίϚϯυҾ਺Ͱࢦఆ
36. DATABASE MANAGEMENT ▸ Slick 3.1 ▸ Schema migration with Evolutions ▸ Enable useLocks option if you have Play nodes that may potential run evolutions ▸ Evolutions 
 https://www.playframework.com/documentation/2.4.x/Evolutions VTF-PDLTͰෳ਺୆σϓϩΠ࣌ͷFWPMVUJPOͷ๫ൃΛࢭΊΔ
37. EVOLUTION BEFORE DEPLOYMENT ▸ Even if useLocks option is enabled,
 Evolution should be executed carefully ▸ Execute evolutions safely before deployments with one-shot style as a Jenkins job java -Dconfig.resource=batch.conf 
 -Dplay.evolutions.db.default.autoApply=true
 -Dplay.mode=prod
 -cp ./target/scala-2.11/app-x.x.jar
 tasks.EvolutionsCheckTask One-shot evolution example FWPMVUJPO͸σϓϩΠͷલʹ୯ମͰ࣮ߦ͓ͯ͘͠ͷ͕҆৺
38. EVOLUTION BEFORE DEPLOYMENT ▸ Even if useLocks option is enabled,
 Evolution should be executed carefully ▸ Execute evolutions safely before deployments with one-shot style as a Jenkins job java -Dconfig.resource=batch.conf 
 -Dplay.evolutions.db.default.autoApply=true
 -Dplay.mode=prod
 -cp ./target/scala-2.11/app-x.x.jar
 tasks.EvolutionsCheckTask One-shot evolution example package tasks import com.typesafe.scalalogging.LazyLogging object EvolutionsCheckTask extends App with Task with LazyLogging { withApplication( app => { logger.info("unnecessary") }, ex => ex match { case ex: play.api.db.evolutions.InvalidDatabaseRevision => logger.warn("necessary") case ex => logger.error(ex.getMessage, ex) println("Error") } ) } tasks.EvolutionsCheckTask
39. MONITORING ▸ Datadog watches system metrics ▸ Collect error logs with Errbit ▸ Errbit is OSS clone of Airbrake System metrics Error logs ௨ৗͷ؂ࢹ͸%BUB%PHͰɺΤϥʔϩά؂ࢹ͸&SSCJUΛ࢖༻
40. AIRBRAKE-LOGBACK ▸ Only set config in logback.xml Error logs <configuration> <conversionRule conversionWord="coloredLevel" converterClass="play.api.Logger$ColoredLevel" /> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <encoder> <pattern>%coloredLevel - %logger - %message%n%xException</pattern> </encoder> </appender> <appender name="AIRBRAKE" class="net.anthavio.airbrake.AirbrakeLogbackAppender"> <apiKey>${config.apikey}</apiKey> <env>production - ${config.resource}</env> <notify>ALL</notify> <url>https://${yourhost}/notifier_api/v2/notices</url> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>WARN</level> </filter> </appender> <appender name="ASYNC_AIRBRAKE" class="ch.qos.logback.classic.AsyncAppender"> <appender-ref ref="AIRBRAKE" /> </appender> <logger name="play" level="INFO" /> <logger name="application" level="INFO" /> <root level="INFO"> <appender-ref ref="STDOUT" /> </root> <root level="WARN"> <appender-ref ref="ASYNC_AIRBRAKE" /> </root> </configuration> MPHCBDLͷઃఆ͚ͩͰ༰қʹ࿈ܞͰ͖Δ
41. WHAT WE GET FROM PLAY APPS ON DOCKERIZED INFRASTRUCTURE ▸ Docker maximizes Scala's advantage ▸ Scalable ▸ Architecture ▸ Development %PDLFSʹΑΔΠϯϑϥӡ༻͸4DBMBͷར఺Λ࠷େԽ͢Δ
42. WE'RE HIRING SCALA DEVELOPER! LET'S
43. REFERENCE ▸ Play2/ScalaͰυϝΠϯۦಈઃܭΛར༻ͨ͠େن໛WebΞϓϦ έʔγϣϯͷεΫϥϜ։ൃͷצॴ 
 http://www.slideshare.net/sifue/developers-summit-2014- play2scalaweb ▸ AWS Lambda Ͱ PlayʢScalaʣ ͷόονॲཧΛ࣮ߦ͢Δ 
 http://tech.recruit-mp.co.jp/server-side/post-5473/ ▸ ScalazΛ࢖͓͏ #1 – NET BIZ DIV. TECH BLOG 
 http://tech.recruit-mp.co.jp/server-side/post-2540/