About DMM.com Group DMM.com is one of Japan's leading E-commerce websites. The Outline of DMM's website and the company: · 2.5 billion page view per month. · Managing more than 40 services. · Members exceeding 20 million. · Unit sales of about $ 700 million. · Group sales of about $ 1.4 billion. (Converted by: 1USD=100JPY)
Մ"1*Λ "LLB4USFBNTBOE)551Ͱ࡞Γ͠·ͨ͠ɻ Refurbishment history of DMM’s platform OAuth 2.0 Connect API (Fuel PHP) API Gateway (Fuel PHP) Non-blocking API GW which was built in OAuth2.0 API Scala 2.11 / Akka Streams and HTTP Non-blocking API Gateway (Java7/Vert.x2) Countermeasure for C10K www.dmm.com / www.dmm.co.jp LAMP application Typical problems, large and monolithic state was given by business growth priority. Another microservices Purchase info API, Settlement agent API etc.. (Fuel PHP) Unified frontend shopping cart (FuelPHP/Couchbase) Legacy emoney systems Legacy shopping carts Unified Emoney API (Java7/Vert.x2/In-memory Data Grid) new Member API (Java8/Spring) Old Member API (LAMP) since ‘03〜? ‘14 Q1 ‘15 Q2 ‘16 Q3 ‘17 Q1〜 Today’s agenda.
"1*Λ"LLBͰ౷߹͠·ͨ͠ɻ After Before What we have done Backend API servers API-GW clients (API Gateway) (OAuth2.0 API) (Unified API Gateway and OAuth2.0 API) The purpose of the project ・To increase authorization performance. ・To centralize management of API routing and authorizing (to do in the future).
ॏཁͳγεςϜͰ͢ɻ Responsibility of API Gateway LB LB ・Mobile devices ・3rd party etc. Backend API servers ・HTTPS TLS decoder ・Ephemeral port buffer API Gateway (Today's agenda.) ・Routing to backend APIs ・Authentication with OAuth2.0 … Actual to x,000req/sec during peak time. … Managing of ・member ・billing ・accounting etc.
Technology stack (storages & drivers) ScalikeJDBC (using MySQL) • It's Query driver which has SQL interpolation function that is very easy to use. • Future-based asynchronous interface. Rediscala (using Redis® for cache storage) • Supporting failover using Redis Sentinel® (discovery service for Redis®). • Future-based asynchronous interface. (Temporary solution for taking over from PHP version and running in parallel.) Hashicorp Vault with original driver implements Akka HTTP Client • For the purpose of keeping secret data. (For example DB password, salt for hash function etc.) • Unified akka HTTP Clinet because it’s interface is HTTP REST API. • Future-based asynchronous interface (via Akka Streams’ mat).
"LLB4USFBNTͷόοΫϓϨογϟʔ͕͋Εᘳʂ Why did we adopt Akka Streams? As API Gateway, stability is the most important, and productivity (fulfillment of libraries and drivers etc.) is also important. Stability • Running on JVM 8 which is a runtime of high achievement and reliability. • Using Akka with sufficient operational results for internal messaging. • Having protection of internal messaging flow by Akka streams’ back pressure. Productivity • Various libraries and drivers in several repositories. e.g. Maven, Ivy, etc. • Scala features, type safety and excellent syntax. Akka Streams is a comprehensive powerful solution!
ҲΕͳ͍Α͏্ྲྀ࿈తʹྲྀྔใΛ௨͢ΔΈͷ͜ͱ What is back pressure? In information terminology, in order to prevent the downstream buffer from overflowing in flow control, flow information is notified in a chain manner to the upstream. Part of supported libraries • Akka Streams(Scala) • Scalaz Stream(Scala) • Node.js (JavaScript) • ReactiveX • RxJava • RxScala • RxJS etc. Implementation depends on each language, library, and backend queue system. Publisher (upstream) Subscriber (downstream) For example in Akka streams’ back pressure. ‘Dynamic push / pull mode’ ✉✉✉✉✉ request(5)
)JHIMFWFM"1*ͰϧʔςΟϯάͷ%4-͕ϋʔυίʔυͷͨΊº The use of API Gateway requires reloading of route setting in memory at an arbitrary timing. Implementing this with the Akka HTTP High-level API can not satisfy the requirement, because routing DSL is hard code. Low-level API High-level API Flexibility Yes. Akka HTTP can bind original flow implimented by ourself. No. Hard-coded routing DSL generates flow to bind to Akka HTTP. Difficulty Hard. Requireing of understanding of Akka streams Graph DSL. Easy. Routing DSL is Spray-like simpile and intuitive directive. We chose Akka HTTP Low-level API
ϦϓϨʔεલγεςϜͷઃఆ͕KTPOͷͨΊ্ҐޓͱΕΔɻ HOCON based original routing system routes. before.json (JSON) routes.conf (HOCON) routes. before.conf (HOCON) Copy as is (and rename) include SIGHUP! In a common practice, when the application receives a HUP signal, it reloads route information. HOCON is upward compatible with json. By using this, compatibility can be keeping by including the pre-replacement configuration as it is.
ΞͰ3PVUJOH'MPXྲྀ͠ࠐΈ·͢ɻ Sample code of swapping routes trait ApplicationServer extends ActorSystemAware { def serverName: String // This AtomicReference is updated when SIGHUP is received. protected lazy val routesRef: AtomicReference[Routes] = Routes.atomicRefercnce protected lazy val serverFlow: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest] // It is necessary to the latest route information. .map { req => (req, routesRef.get) } .via(RoutingFlow.flow) // Start the HTTP server. The started server can stop using return value ‘Http.ServerBindig’. protected lazy val serverBinding: Future[Http.ServerBinding] = { Global.serverSetting(serverName) match { case (host, port, Some(connectionContext))=> Http().bindAndHandle(serverFlow, host, port, connectionContext) case (host, port, None) => Http().bindAndHandle(serverFlow, host, port) } } }
Combined Routing flow Routing flow diagram Partition (find route) Endpoint Flow e.g. ・HTTP ・OAuth2.0 and future ・AMQP etc. Http. Incoming Connection IncomingFiltersFlow Inco ming Filter Inco ming Filter ・・・ OutgoingFiltersFlow Outg oing Filter Outg oing Filter ・・・ Partition (continue or abort) Merge AbortFlow Merge Not Found Flow If found If not found If abort If continue
͜ͷਤͰ͘هͨ͠/PU'PVOE'MPXͱ"CPSU'MPXͰ͢ɻ Combined Routing flow Routing flow diagram (short circuit) Partition (find route) Endpoint Flow e.g. ・HTTP ・OAuth2.0 and future ・AMQP etc. Http. Incoming Connection IncomingFiltersFlow Inco ming Filter Inco ming Filter ・・・ OutgoingFiltersFlow Outg oing Filter Outg oing Filter ・・・ Partition (continue or abort) Merge AbortFlow Merge Not Found Flow If found If not found If abort If continue
Combined Routing flow Routing flow diagram (conditional branch) Partition (find route) Endpoint Flow e.g. ・HTTP ・OAuth2.0 and future ・AMQP etc. Http. Incoming Connection IncomingFiltersFlow Inco ming Filter Inco ming Filter ・・・ OutgoingFiltersFlow Outg oing Filter Outg oing Filter ・・・ Partition (continue or abort) Merge AbortFlow Merge Not Found Flow If found If not found If abort If continue
݅ذͳΒ'BOPVUδϟϯΫγϣϯͷ1BSUJUJPOΛ͍·͢ɻ Akka Streams’ GraphDSL can be used for expressing branch of Flow. (And also for balance, broadcast or etc.) In the conditional branching case, • The branching uses the Fan-out junction 'Partition' (1 input n outputs). Give the Partition’s factory method’s argument a branch judgment function ‘IN => Int’. This function is an implementation that returns the index number of downstream according to IN contents. • Finally, the branch integration uses the Fan-in junction 'Merge' (n inputs 1 output). How to express conditional branch of Flow Partition Merge Any Flows … …
ذఆͷؔΛҾʹ͚ͩ͢Ͱ͑·͢ɻ Sample code of Partition and Merge val routingFlowShape = GraphDSL.create() { implicit builder => import GraphDSL.Implicits._ type Input = (Routes, HttpRequest) val input = builder.add(Flow[Input]) val (found, notFound) = (1, 0) val findingRouteP = builder.add(Partition[Input](2, { case (routes, req) => if (routes.isFound(req)) found else notFound })) val findingRouteM = builder.add(Merge[HttpResponse](2)) val foundResponse = builder.add(/* Implementation in following slide! :) */???) val notFoundResponse = builder.add(Flow[Input])map(_ => HttpResponse(status = NotFound))) input ~> findingRouteP.in findingRouteP.out(found) ~> foundResponse ~> findingRouteM.in(found) findingRouteP.out(notFound) ~> notFoundResponse ~> findingRouteM.in(notFound) FlowShape(input.in, findingRouteM.out) } Defining Partition and Merge. Give the factory method’s argument a branch judgment function and junction width.
)551ϔομͷ༩+40/31$㱻3&45มͳͲΛߦ͍·͢ɻ Combined Routing flow Routing flow diagram (fliter plug-in) Partition (find route) Endpoint Flow e.g. ・HTTP ・OAuth2.0 and future ・AMQP etc. Http. Incoming Connection IncomingFiltersFlow Inco ming Filter Inco ming Filter ・・・ OutgoingFiltersFlow Outg oing Filter Outg oing Filter ・・・ Partition (continue or abort) Merge AbortFlow Merge Not Found Flow If found If not found If abort If continue
ϧʔτใʹඥ͍ͮͨϓϥάΠϯઃఆ͔Βಈతʹߏங͠·͢ɻ The plug-in modules are for flexible editing and verifying the input / output contents to the Endpoint. For example, • Addition Internal HTTP headers • Interconversion of REST and non-REST • Verifying authorized with OAuth2.0 etc. The composed result of the plug-in is constructed dynamically by the plug-in setting which links to the route information. For composing and applying filters in order, Flow’s IN and OUT type signatures are the same. IncomingFiltersFlow (OutgoingFiltersFlow is similar to this structure.) Source (single) Verify Autho rized REST ⇔non REST Foo Filter Add Head er Dynamically concatination flows by route info using ‘Flow.flatMapConcat(Out1 => Source[Out2, Mat])’. HttpRequestContext (request, route) About incoming (outgoing) filter plug-in
ιουͷॲཧΛద༻͠·͢ɻ Sample code of IncomingFilter trait // EI type variable is endpoint incoming data. V type variable is voluntary data. trait IncomingFilter[EI <: RequestContext, V] extends ActorSystemAware { type Acc = (HttpResponse ∨ EI, V) // Applied filter result type signature. type IO = (HttpRequestContext, Acc) // IncomingFilter.flow’s In-Out type signature. val flow: Flow[IO, IO, _] } trait AsyncIncomingFilter[EI <: RequestContext, V] extends IncomingFilter { protected def apply(route: Route, orig: HttpRequest, acc: Acc): Future[Acc] lazy val flow: Flow[IO, IO, _] = Flow[IO].flatMapConcat { case (orig, acc) => Source.fromFuture(apply(orig.route, orig.request, acc)).map((orig, _)) } } For the apply method, narrowing down the necessary input / output values and return the result with type Future[Acc]. Therefore, knowledge of Akka Streams is unnecessary for plug-in implementation. Note for: The original HttpRequestContext should not be passed to the apply method’s return value. As a reason, it is routed to the next IncomingFilter through a detour path so as not to destroy original.
݁Ռ͕MFGUͩͬͨ߹ԼྲྀͰ"CPSU'MPXసૹ͍ͯ͠·͢ɻ Sample code of right-bind IncomingFilter trait RightIncomingFilter[EI <: RequestContext, V] extends AsyncIncomingFilter[EI, V] { override final def apply(route: Route, orig: HttpRequest, acc: Acc): Future[Acc] = { acc match { case (∨-(ei), v) => applyRight(route, orig, (ei, v)) case left => Future.successful(left) } } protected def applyRight(route: Route, orig: HttpRequest, acc: (EI, V)): Future[Acc] } Rignt-bind implementation. ・ To short-circuit when the left comes. ・ ∨’s left type is HttpResponse. (It is sent downstream to AbortFlow.)
݁Ռ'VUVSFʹ͘Δ·ΕΔͷͰ໌ࣔతͳϒϩοΫෆཁͰ͢ɻ Sample code of HTTP method editing private def editMethodWithEntityToQuery(destMethod: HttpMethod)(srcReq: HttpRequset): Future[HttpRequest] = val srcUri = srcReq.uri val srcEntity = srcReq.entity Unmarshal(srcEntity).to[FormData].map { formData => val srcQuery = srcUri.query() val fields = formData.fields val queryBuilder = Query.newBuilder // mutable API queryBuilder ++= srcQuery queryBuilder ++= fields val destQuery = queryBuilder.result val destQueryString = destQuery.toString.some.filterNot(_.isEmpty) val destHeaders = srcReq.headers.filterNot(_.name == `Content-Type`.name) srcReq.copy( method = destMethod, headers = destHeaders, entity = HttpEntity.Empty, uri = srcUri.copy(rawQueryString = destQueryString) ) } recover { case e: UnsupportedContentTypeException => logger.warn(s"Availed HTTP entity may be not single-part form data in HttpMethodEditing of entity to que srcReq } } How to parse object in Akka HTTP? Akka HTTP Unmarshaller is useful for parsing HTTP entities. Explicit blocking is unnecessary because parsing results are wrapped in Future. Depending on the type classes, the required implementation of Akka HTTP embedded is resolved at compilation time.
ͪ͜Βҙͷ࣮ΛઃఆͰ͖ΔΑ͏ʹϓϥάΠϯܗࣜͰ͢ɻ Combined Routing flow Routing flow diagram (endpoint plug-in) Partition (find route) Endpoint Flow e.g. ・HTTP ・OAuth2.0 and future ・AMQP etc. Http. Incoming Connection IncomingFiltersFlow Inco ming Filter Inco ming Filter ・・・ OutgoingFiltersFlow Outg oing Filter Outg oing Filter ・・・ Partition (continue or abort) Merge AbortFlow Merge Not Found Flow If found If not found If abort If continue
ࠓճೝՄ"1*&OEQPJOUϓϥάΠϯͱ࣮ͯ͠͠·ͨ͠ɻ As the API Gateway, Endpoint plug-in is a plug-in primarily for external communication. For example, • HTTP / HTTPS client • Authorization API's business logic • AMQP client (to do in the future) • Kafka client (to do in the future) • gRPC client (to do in the future) etc. In this time, OAuth 2.0 API is implemented by this as a external communication. About Endpoint plug-in
༻ɻϗετຖͷϓʔϧ44-ͷઃఆBQQMJDBUJPODPOGɻ Sample code of Akka HTTP client trait HttpClients extends ActorSystemAware { // Type parameter [T] means to keep path-through data in Flow’s I/O. def forUri[T](apiUrl: Uri): Flow[(HttpRequest, T), (Try[HttpResponse], T), HostConnectionPool] = { val host = apiUrl.authority.host.toString val port = apiUrl.authority.port val hostSettings = resolveHostSpecificSettings(host) | defaultConnectionPoolSettings val sslCtxt = Global.clientHttpsContext | Http().defaultClientHttpsContext (apiUrl.scheme, port) match { case ("http" , 0) => Http().cachedHostConnectionPool[T](host, settings = hostSettings) case ("http" , _) => Http().cachedHostConnectionPool[T](host, port, hostSettings) case ("https", 0) => Http().cachedHostConnectionPoolHttps[T](host, connectionContext = sslCtxt , settings = hostSettings) case ("https", _) => Http().cachedHostConnectionPoolHttps[T](host, port, sslCtxt, hostSettings) case _ => throw new NoSuchElementException } // ....... } This time it is an API Gateway server, so we set up endpoints host connection pool. The number of connection pools and the setting of SSL for each host is basically set in application.conf.
Receive HttpRequest at the Flow's parameter. • According to the setting of routing, make as many Partitions as number of routes. • Applied plug-in for each route is different and the length of the previous Flow by Partition is different. Implementation process ϧʔτఆٛͱಉ͚ͩ͡1BSUJUJPOʹΑΔذΛ࡞Γ·͢ɻ'MPX ͷ͞ϧʔτຖʹҟͳΓ·͢ɻ
Depending on the requested path, the index (or root) of the Partition is specified. • Connect partial Graph per sort of plugin to Merge. ϦΫΤετ͞ΕͨύεʹԠͯ͡1BSUJUJPOͷࡧҾ ϧʔτͷࡧҾ ͕ ಛఆ͞Εɺ.FSHFʹܨ͛ΒΕ·͢ɻ Implementation process
As the routing configuration size (Partition size) increased, heap size expanded. • When the route size exceeds 100, heap size rises 1 GB at the time of start. • Performance has been dragged by Full GC, and throughput has also shrunken. ϧʔτͷ 1BSUJUJPOͷαΠζ ͕େ͖͘ͳΔʹͭΕͯɺىಈ࣌ͷ ώʔϓαΠζ͕ΒΈɺύϑΥʔϚϯεͷྼԽ͕ݟΒΕ·ͨ͠ɻ But ...
*O0VUଟྔͷϦΫΤετͷॲཧʹ͍͍ͯͳ͍Α͏Ͱ͢ɻ • Removed Partition in Rouing Flow and Graph got smaller. • The heap size at startup also has been decreased to 100-200M. • Full GC also disappeared and throughput has got improved. • Fan In/Fan Out is not suitable for large number of request processing. After improvement
GMBU.BQ$PODBU͕༗ޮͰͨ͠ɻ • However, depending on the result of routing, plug-ins that are organized by Flow must be selected and connect dynamically per request. • Flow#flatMapConcat was useful for the part where Flow should be dynamically connected. After improvement val incomingPluginsFlow = Flow[(HttpRequest,Route)].flatMapConcat { case (request,route) => // flatMapConcat requires SourceShape. Source.single((request -> route)).via { // Connect all filter(Flow) in this route. route.infilters.map{ infilter => infilter.flow}.foldLeft(initalValue)(_.via(_)) } }
'MPXͰॻ͘ͱͲ͏ͳΔͰ͠ΐ͏͔ • We use Graph DSL in a main Flow. • DSL is very useful for visualizing that whole processing flow. • if this flow written without DSL... By the way... // format: OFF request ~> findingRouteP.in findingRouteP.out(found) ~> incomingPluginsFlow ~> p.in continueOrAbortP.out(continue) ~> endpointAndOutgoingPluginsFlow ~> continueOrAbortM.in(continue) continueOrAbortP.out(abort) ~> abortFlow ~> continueOrAbortM.in(abort) continueOrAbortM.out ~> response ~> findingRouteM.in(found) findingRouteP.out(notFound) ~> notFoundResponse ~> findingRouteM.in(notFound) // format: ON FlowShape(request.in, findingRouteM.out)
ेͳεϧʔϓοτΛಘΔ͜ͱ͕Ͱ͖·ͨ͠ɻ ࣾൺ Performance testing result • Throughput which created by Akka is comparable to previous Gateway(Vert.x 2) . • About 8% of performance was reduced. • We obtained sufficient throughput even with considering additional functions.
͍ΤϯυϙΠϯτɺ͍ΤϯυϙΠϯτΛ༻ҙ͠·ͨ͠ɻ Testing the characteristics of Gateway • We prepared a slow endpoint (emulating processing time 1 second) and a fast endpoint (emulating processing time 50 ms) behind Gateway. Fast Endpoint Slow Endpoint 1500 req/sec constantry
ͷӨڹΛड͚ͯશମͷεϧʔϓοτ͕མͪͳ͍͔ݕূ͠·ͨ͠ɻ • We loaded both endpoints at the same time and tested whether the overall throughput has not decreased because it was affected by the slow endpoint. Testing the characteristics of Gateway Fast Endpoint Slow Endpoint 1500 req/sec constantry
Λҡ͍࣋ͯ͠·ͨ͠ɻ Proxy destination Throughput (req/sec) Minimum Response Time(msec) Average Response Time(msec) Maximum Response Time(msec) Fast Endpoint 1490.066 52 88 608 Slow Endpoint 80.972 1002 25920 600010 Test Result • At the fast endpoint, consistently high throughput was maintained even at high loads.
Ճ͕ͨ͠ɺΤϯυϙΠϯτ͕ఀࢭ͢Δࣄ͋Γ·ͤΜͰͨ͠ɻ Test Result • At the slow endpoint, timeout increased as load increased. • Connecting to the endpoint that has been timed out in 60 seconds. • Slow endpoints were not destroyed even when high load was applied. Proxy destination Throughput (req/sec) Minimum Response Time(msec) Average Response Time(msec) Maximum Response Time(msec) Fast Endpoint 1490.066 52 88 608 Slow Endpoint 80.972 1002 25920 600010
Λ͑ͭͭɺߴ͍ύϑΥʔϚϯεΛಘΔ͜ͱ͕ग़དྷ·ͨ͠ɻ • Akka HTTP (and Akka Streams) performed the expected behavior of minimizing influence on the endpoint and the gateway itself using back-pressure control. • While performing backpressure control, it was possible to obtain sufficient throughput even compared to before transplantation. Summary
)PTU$POOFDUJPO1PPM ʹΑΔϓϩΩγɺ (BUFXBZ ϚΠΫϩαʔϏε ͷಛੑʹద߹ͯ͠͠·͢ʂ • Flow is suitable for constructing a static structure. • The proxy by HostConnectionPool is suitable due to the nature of the Gateway server. • Akka Streams and HTTP are useful for microservice architecture! Summary
mananan
bkuhlmann
jaguar_imo
pinkumohikan
letelete
1wedge_one
ryunakayama
scalar
dunglas
memory1994
pyama86
zakiyama
sstephenson
smashingmag
zenorocha
bermonpainter
maltzj
tanoku
chrislema
mthomps
3n
addyosmani
rocio
