AdTech Scala Meetup 9 Spark SQL Thrift JDBC Server

CyberAgent アドテクスタジオで開催されているScala Meetupで行ったLTの資料です。Thrift JDBC Server についての話。

Transcript

1. "E5FDI
   4DBMB
   .FFUVQ
   ୈճ
 ʮ4QBSL
   42-
   5ISJGU
   +%#$
   TFSWFSʯ 
   $"
   1SP'JU9
   4IVZB
   5TVLBNPUP 
2. w 5PQ
 IUUQTTQBSLBQBDIFPSH
   w %PDT
 IUUQTTQBSLBQBDIFPSHEPDTMBUFTU
   w %PDT ೔ຊޠ ɹ͏Ε͍͠ɾɾɾʂ
 IUUQNPHJMFXFCGDDPNTQBSL 
3. 4QBSL
   42- w 4QBSLͰ42-͕࢖͑Δ
 IUUQNPHJMFXFCGDDPNTQBSLTRM QSPHSBNNJOHHVJEFIUNM
 
4. 4QBSL
   42-
   FYBNQMF val sqlContext = new org.apache.spark.sql.SQLContext(sc) import sqlContext.implicits._ case class Person(name: String, age: Int) val people = sc.textFile("examples/src/main/resources/people.txt") .map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt)).toDF() people.registerTempTable("people") val teenagers = sqlContext .sql("SELECT name, age FROM people WHERE age >= 13 AND age <= 19") teenagers.map( t => "Name: " + t.getAs[String]("name") ).collect().foreach(println) 
5. 42-
   PO
   4DBMB
   DPEF
   ʜ w 4DBMBίʔυ্ʹ42-
   w ·͋ྑ͍͚Ͳɺ࢖͍ͮΒ͘ͳ͍ͷʁ
   w ී௨ͷ%#ࢀরπʔϧ΍ɺී௨ͷ#BUDIϓϩάϥϜ ͔Β4&-&$5͍ͨ͠ 
6. 4QBSL
   42-
   5ISJGU
   +%#$
   TFSWFS w +%#$઀ଓͰ͖·͢ʂ
   w %PD
 IUUQNPHJMFXFCGDDPNTQBSLTRM QSPHSBNNJOHHVJEFIUNMEJTUSJCVUFETRM FOHJOF
   w ҙ֎ͱ؆୯ʹࢼͤΔ
 
7. +%#$
   TFSWFS
   FYBNQMF  4QBSL
   %-
   
   1SFCVJMU
   GPS
   $%) 
 IUUQTTQBSLBQBDIFPSHEPXOMPBETIUNM
    UBS
   YW[G
   TQBSLYYYUH[
    DE
   TQBSLYYY
    DPOGIJWFTJUFYNM
   Λ࡞੒
 <configuration> <property> <name>hive.metastore.warehouse.dir</name> <value>/var/spark/hive/warehouse</value> </property> </configuration> TVEP
   NLEJS
   Q
 TVEP
   DIPXO 
8. +%#$
   TFSWFS
   FYBNQMF w ޙ͸ެࣜEPDʹै͍ͭͭɾɾɾ
    TCJOTUBSUUISJGUTFSWFSTI
    CJOCFFMJOF
    DPOOFDU
   KECDIJWFMPDBMIPTU
 6TFS/BNF͸ద౰ɺ1BTT͸ۭ
    DSFBUFจ create table users(name string, age int) row format delimited fields terminated by ',' lines terminated by '\n'; 
9. +%#$
   TFSWFS
   FYBNQMF  ͖ͬ͞ͷXBSFIPVTF
   EJS഑Լʹద౰ͳαϯϓϧσʔ λ EBUBDTW Λ഑ஔ
 
 
 ͜ͷλΠϛϯάͰผͷ৔ॴʹEBUBDTW͕ίϐʔ͞ΕΔ
   TFMFDU
   
   GSPN
   VTFST foo,20 bar,22 
10. ࣮ӡ༻ͷ৔߹ʁ w ༧ΊUISJGUTFSWFSΛݐ͓ͯͯ͘
    w XBSFIPVTF
    EJSࣗମ͸)%'4഑Լʹ
    w XBSFIPVTF
    EJS഑ԼʹͲΜͲΜϑΝΠϧΛ഑ஔˍࣗ ಈͰಡΈࠐ·ͤΔ 
11. ϝϦοτ ૝૾ w +%#$ͰूܭΛ׬݁Ͱ͖ΔͷͰɺී௨ͷ#BUDIϓϩ άϥϜ 42-ͷܗࣜͰγεςϜΛ࡞ΕΔ
 ʢ4QBSL
    %SJWFSϓϩάϥϜΛॻ͘ඞཁ͕ແ͍ʣ
    w +%#$ͳͷͰΞυϗοΫʹ%#ࢀরπʔϧͰΫΤϦ Λ࣮ߦՄೳ
 ʢ࢖͍׳ΕͨπʔϧɾՄࢹԽπʔϧΛར༻Մೳʣ 
12. ະௐࠪ w XBSFIPVTF
    EJSҎԼͷಛఆͷσʔλ͚ͩର৅ʹ͢Δɹ Έ͍ͨͳ͜ͱ͕Ͱ͖Δͷ͔ɾɾɾʁ
 ʢ೔෇୯ҐͰςʔϒϧ෼͚Δͱ͔͠ͳ͍ͱμϝʁʣ
    w Ͱɺ+%#$
    %SJWFS +"3 ͸Ͳ͜ʁ
    w ࣮ӡ༻ͷ࿩͕ฉ͖͍ͨɻ࣮ྫ͸ͲΕ͘Β͍͋Δʁ