Apache FlinkͰߏங͢Δ ϦΞϧλΠϜετϦʔϜॲཧ ύΠϓϥΠϯ Scala෱Ԭ2017 2017/07/29 @matsumana 

ࣗݾ঺հ • ໊લɿ দ࡚ ֶ • ॴଐɿ LINE Fukuokaגࣜձࣾ • Roleɿ SRE • Twitterɿ @matsumana 

ΞδΣϯμ • Flinkͷ֓ཁ • TwitterͷσʔλΛूܭͯ͠ՄࢹԽ͢ΔσϞ 

Flinkͷ֓ཁ 

• Streaming first • Fault tolerant • Scalable • Performance ಛ௃ 

ϩΰ͸ՄѪ͘ͳ͍ 

جຊతͳॲཧͷྲྀΕ 

Source(input), Sink(output) ͸৭ʑબ୒Մೳ • Twitter (source) • Kafka (source/sink) • RabbitMQ (source/sink) • Apache NiFi (source/sink) • AWS Kinesis (source/sink) 

Source(input), Sink(output) ͸৭ʑબ୒Մೳ • HDFS (sink) • Elasticsearch (sink) • Cassandra (sink) • Redis (sink) • Flume (sink) • ActiveMQ (sink) • Third-Party Projects (e.g. Apache Zeppelin) 

ίʔυྫ 

JavaͰ΋ScalaͰ΋ॻ͚Δ͕ Scalaͷํ͕ ॻ͖΍͍͢͠ಡΈ΍͍͢ 

// Word count in Scala // set up the execution environment val env = ExecutionEnvironment.getExecutionEnvironment // get input data val text = env.fromElements( "To be, or not to be --that is the question:--", "Whether 'tis nobler in the mind to suffer", "The slings and arrows of outrageous fortune", "Or to take arms against a sea of troubles") // count val counts = text .flatMap { _.toLowerCase.split("\\W+") } .map { (_, 1) } .groupBy(0) .sum(1) // emit result and print result counts.print() 

// Word count in Java // set up the execution environment // (লུ) // get input data // (লུ) // count DataStream> counts = text .flatMap((String line, Collector out) -> { String[] tokens = line.toLowerCase().split("\\W+"); Arrays.stream(tokens) .forEach(out::collect); }) .map(s -> new Tuple2<>(s, 1)) .groupBy(0) .sum(1); // emit result and print result // (লུ) 

Flink DataStream API Programming Guide ʹ΋αϯϓϧίʔυ͕͋Γ·͢ https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html 

Flink͸LINEͰ΋ ࢖ΘΕ͍ͯ·͢ 

ڈ೥ͷΧϯϑΝϨϯεͰ঺հ͞Ε͍ͯΔͷͰ ৄ͘͠͸ͦͪΒΛݟͯԼ͍͞ ʢಈըɾεϥΠυ͕ެ։͞Εͯ·͢ʣ • LINE DEVELOPER DAY 2016
 https://engineering.linecorp.com/ja/blog/detail/87 • B-6 ηογϣϯ
 New stream processing platform with Apache Flink 

• ֤http statusίʔυ਺ • http statusίʔυͷׂ߹
 (2xx, 3xx, 4xx, 5xx) • ϨεϙϯελΠϜͷׂ߹ • ϨεϙϯελΠϜͷpercentile
 (avg, min, 50, 90, 95, 98, 99) ͪͳΈʹɺΧϯϑΝϨϯεͰ࿩͕ग़ͯΔ ΞΫηεϩάूܭ͸͜Μͳ΍ͭͰ͢ 

࣍͸ Twitter͔ΒσʔλΛऔ͖ͬͯͯ ՄࢹԽ͢ΔσϞ 

֓ཁਤ 

Thank you :)