sbt-emr-spark

Transcript

1. sbt-emr-spark Run your Spark on AWS EMR with sbt
2. @pishen (ピシェン)
3. 私たちは台湾から来ました!!
4. と言う Meetup Group の者です 一ヶ月に一回、20 人ぐらい gitter channel があります、Please call us if you come to 台北！
5. pishen / sbt-emr-spark お願いします！
6. どうしてこのツールを作ったの？
8. pip install awscli pip は何？食べ物ですか？
9. aws emr create-cluster --release-label emr-5.2.1 --name my-cluster --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.x large InstanceGroupType=CORE,InstanceCount=2,InstanceType= m3.xlarge --log-uri s3://my-bucket/log-dir/ --service-role EMR_DefaultRole --ec2-attributes SubnetId=subnet-xxxxxxxx,AdditionalMasterSecurityGroups=sg -xxxxxxxx,InstanceProfile=EMR_EC2_DefaultRole,AdditionalSl aveSecurityGroups=sg-xxxxxxxx sbt assembly aws s3 cp target/... s3://... aws emr add-steps ... 長い！面倒くさい！
10. Just use SBT and get things done!
11. resolvers += Resolver.bintrayIvyRepo("pishen", "sbt-plugins") addSbtPlugin("net.pishen" % "sbt-emr-spark" % "0.3.0") project/plugins.sbt
12. name := "sbt-emr-spark-test" scalaVersion := "2.11.8" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.0.2" % "provided" ) sparkAwsRegion := "ap-northeast-1" sparkSubnetId := Some("subnet-xxxxxxxx") sparkAdditionalSecurityGroupIds := Some(Seq("sg-xxxxxxxx")) sparkS3JarFolder := "s3://my-emr-bucket/my-emr-folder/" sparkInstanceCount := 2 build.sbt
13. package mypackage import org.apache.spark._ object Main { def main(args: Array[String]): Unit = { //setup spark val sc = new SparkContext(new SparkConf()) //your algorithm val n = 10000000 val count = sc.parallelize(1 to n).map { i => val x = scala.math.random val y = scala.math.random if (x * x + y * y < 1) 1 else 0 }.reduce(_ + _) println("Pi is roughly " + 4.0 * count / n) } }
14. $ sbt > sparkCreateCluster > sparkSubmitJob
15. pishen / sbt-emr-spark Please see the README for more information.