AdTech Scala Meetup 9 Spark SQL Thrift JDBC Server

by Shuya Tsukamoto

Slide 1

Slide 1 text

"E5FDI4DBMB.FFUVQୈճ  ʮ4QBSL42-5ISJGU+%#$TFSWFSʯ $"1SP'JU94IVZB5TVLBNPUP

Slide 2

Slide 2 text

w 5PQ  IUUQTTQBSLBQBDIFPSH w %PDT  IUUQTTQBSLBQBDIFPSHEPDTMBUFTU w %PDT ೔ຊޠ ɹ͏Ε͍͠ɾɾɾʂ  IUUQNPHJMFXFCGDDPNTQBSL

Slide 3

Slide 3 text

4QBSL42- w 4QBSLͰ42-͕࢖͑Δ  IUUQNPHJMFXFCGDDPNTQBSLTRM QSPHSBNNJOHHVJEFIUNM 

Slide 4

Slide 4 text

4QBSL42-FYBNQMF val sqlContext = new org.apache.spark.sql.SQLContext(sc) import sqlContext.implicits._ case class Person(name: String, age: Int) val people = sc.textFile("examples/src/main/resources/people.txt") .map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt)).toDF() people.registerTempTable("people") val teenagers = sqlContext .sql("SELECT name, age FROM people WHERE age >= 13 AND age <= 19") teenagers.map( t => "Name: " + t.getAs[String]("name") ).collect().foreach(println)

Slide 5

Slide 5 text

42-PO4DBMBDPEFʜ w 4DBMBίʔυ্ʹ42- w ·͋ྑ͍͚Ͳɺ࢖͍ͮΒ͘ͳ͍ͷʁ w ී௨ͷ%#ࢀরπʔϧ΍ɺී௨ͷ#BUDIϓϩάϥϜ ͔Β4&-&$5͍ͨ͠

Slide 6

Slide 6 text

4QBSL42-5ISJGU+%#$TFSWFS w +%#$઀ଓͰ͖·͢ʂ w %PD  IUUQNPHJMFXFCGDDPNTQBSLTRM QSPHSBNNJOHHVJEFIUNMEJTUSJCVUFETRM FOHJOF w ҙ֎ͱ؆୯ʹࢼͤΔ 

Slide 7

Slide 7 text

+%#$TFSWFSFYBNQMF 4QBSL%- 1SFCVJMUGPS$%)   IUUQTTQBSLBQBDIFPSHEPXOMPBETIUNM UBSYW[GTQBSLYYYUH[ DETQBSLYYY DPOGIJWFTJUFYNMΛ࡞੒  hive.metastore.warehouse.dir /var/spark/hive/warehouse TVEPNLEJSQ  TVEPDIPXO

Slide 8

Slide 8 text

+%#$TFSWFSFYBNQMF w ޙ͸ެࣜEPDʹै͍ͭͭɾɾɾ TCJOTUBSUUISJGUTFSWFSTI CJOCFFMJOF DPOOFDUKECDIJWFMPDBMIPTU  6TFS/BNF͸ద౰ɺ1BTT͸ۭ DSFBUFจ create table users(name string, age int) row format delimited fields terminated by ',' lines terminated by '\n';

Slide 9

Slide 9 text

+%#$TFSWFSFYBNQMF ͖ͬ͞ͷXBSFIPVTFEJS഑Լʹద౰ͳαϯϓϧσʔ λ EBUBDTW Λ഑ஔ      ͜ͷλΠϛϯάͰผͷ৔ॴʹEBUBDTW͕ίϐʔ͞ΕΔ TFMFDUGSPNVTFST foo,20 bar,22

Slide 10

Slide 10 text

࣮ӡ༻ͷ৔߹ʁ w ༧ΊUISJGUTFSWFSΛݐ͓ͯͯ͘ w XBSFIPVTFEJSࣗମ͸)%'4഑Լʹ w XBSFIPVTFEJS഑ԼʹͲΜͲΜϑΝΠϧΛ഑ஔˍࣗ ಈͰಡΈࠐ·ͤΔ

Slide 11

Slide 11 text

ϝϦοτ ૝૾ w +%#$ͰूܭΛ׬݁Ͱ͖ΔͷͰɺී௨ͷ#BUDIϓϩ άϥϜ42-ͷܗࣜͰγεςϜΛ࡞ΕΔ  ʢ4QBSL%SJWFSϓϩάϥϜΛॻ͘ඞཁ͕ແ͍ʣ w +%#$ͳͷͰΞυϗοΫʹ%#ࢀরπʔϧͰΫΤϦ Λ࣮ߦՄೳ  ʢ࢖͍׳ΕͨπʔϧɾՄࢹԽπʔϧΛར༻Մೳʣ

Slide 12

Slide 12 text

ະௐࠪ w XBSFIPVTFEJSҎԼͷಛఆͷσʔλ͚ͩର৅ʹ͢Δɹ Έ͍ͨͳ͜ͱ͕Ͱ͖Δͷ͔ɾɾɾʁ  ʢ೔෇୯ҐͰςʔϒϧ෼͚Δͱ͔͠ͳ͍ͱμϝʁʣ w Ͱɺ+%#$%SJWFS +"3 ͸Ͳ͜ʁ w ࣮ӡ༻ͷ࿩͕ฉ͖͍ͨɻ࣮ྫ͸ͲΕ͘Β͍͋Δʁ