Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
L FG A
Slide 2
Slide 2 text
• S')1 0(6T • L>A9 XFLAG CDB= !?NRK • GRD /%Q$7 • GRDO:>3GRD;<8H;C-,/ ACFM • P?/5#2(4&"Q 1+/GRDJPR • BIERN/ • @RIC.*6 / • GitHub: ojima-h 2
Slide 3
Slide 3 text
4 DAUKPI !
Slide 4
Slide 4 text
5
Slide 5
Slide 5 text
6 • • 2TB/day 30 → 1000
Slide 6
Slide 6 text
7 • 5 → 100
Slide 7
Slide 7 text
− 8 S3
Slide 8
Slide 8 text
− 9 S3
Slide 9
Slide 9 text
− 10 Redshift
Slide 10
Slide 10 text
− 11
Slide 11
Slide 11 text
12 Data Lake Architecture
Slide 12
Slide 12 text
Data Lake " • -4,&$#!-4,+.' • -4,&%"%,(13*+)40&% ! (Schema on Read) • Data Lake -4,& DWH 24/$ $% 13
Slide 13
Slide 13 text
Data Lake 14 Hive Metastore
Slide 14
Slide 14 text
Hive Metastore 15
Slide 15
Slide 15 text
Hive " • Hadoop%(47-:.69!; • SQL ,*7&$S3 # HDFS !1:/ #1:/ & • ORC !3')83+:502& 16
Slide 16
Slide 16 text
Hive Metastore • S3/HDFS * "-SQL /1,&(.&0 (.&%)! • ,&(.& • * "- • * "-*#.+') • (.&%$.+ • 17
Slide 17
Slide 17 text
Hive Metastore • EMR ! Hive Metastore ! • • EMR 30 18
Slide 18
Slide 18 text
Hive Metastore • Hive Metastore MySQL • Hive Metastore (HCatalog) server • EMR 5 19
Slide 19
Slide 19 text
Hive Metastore S3 20
Slide 20
Slide 20 text
Hive Metastore • ' • '"% • 'ORC • '!&' ' !'#$$ 21
Slide 21
Slide 21 text
Hive Metastore • Hive Metastore S3 " S3" !" 22
Slide 22
Slide 22 text
Hive Metastore * • "+$%- :>:>(*+ • 8C6*/,# • 3C;4' Hive DB / • Hive ).!% S3&*8C6/ • Hive &.(8C6)-*@C@/ 23 3C;4 D=A49B<019?C2BBE 8C6579 8C6 Hive Database Table Partition S3 s3://BUCKET/warehouse/SERVICE.db/ s3://BUCKET/warehouse/SERVICE.db/TABLE/ s3://BUCKET/warehouse/SERVICE.db/TABLE/y=YYYY/m=MM/d=DD/
Slide 23
Slide 23 text
Hive Metastore • %)"&'&'%)" • &$# ! ( 24
Slide 24
Slide 24 text
Hive Metastore 1. Hive Metastore 25
Slide 25
Slide 25 text
Hive Metastore 1. Hive Metastore 2. 26
Slide 26
Slide 26 text
Hive Metastore 1. Hive Metastore 2. 3. Hive Metastore 27
Slide 27
Slide 27 text
Hive Metastore 1. Hive Metastore 2. 3. Hive Metastore 4. 28
Slide 28
Slide 28 text
Hive Metastore ! 1. ),(!$ Hive Metastore # 2. ),($'*, 3. Hive Metastore ! $ 4. ),($ &%+ $ "),($ 29
Slide 29
Slide 29 text
Hive Metastore 30
Slide 30
Slide 30 text
Hive Metastore • Hive Redshift "%!$%# • Redshift COPY "%! csv+gzip • Hive "%! ORC • Redshift csv+gzip Hive ORC ⇒ Redshift Spectrum 31
Slide 31
Slide 31 text
Redshift Spectrum • Redshift S3(#$+ &%*" • ',)+ Hive Metastore ! Hive ',)+" 32 CREATE EXTERNAL SCHEMA schema_name FROM HIVE METASTORE DATABASE 'database_name’ URI 'hive_metastore_uri’;
Slide 32
Slide 32 text
Hive Metastore • Redshift Hive 33 INSERT INTO ‘Redshift ’ SELECT … FROM ‘Hive ’ WHERE y=YYYY AND m=MM AND d=DD;
Slide 33
Slide 33 text
Hive Metastore • Redshift Spectrum Hive Metastore • Spark SQL • Presto • Athena • Flink 34
Slide 34
Slide 34 text
Hive Metastore Hive Metastore S3 Hive, Redshift Spectrum , Spark 35
Slide 35
Slide 35 text
36
Slide 36
Slide 36 text
($) • Hive Metastore '25103-$251.4/4& • Hive Metastore ,$" Data Lake , !$# 251&*251&%+$#! Hive Metastore , +$# Data Lake , "$#(!6 37
Slide 37
Slide 37 text
No content