Upgrade to Pro — share decks privately, control downloads, hide ads and more …

クラウドDWHとしても進化を続けるPivotal Greenplumご紹介 / Pivotal...

クラウドDWHとしても進化を続けるPivotal Greenplumご紹介 / Pivotal Greenplum for Cloud DWH

Avatar for Masayuki Matsushita

Masayuki Matsushita

September 25, 2019

Other Decks in Technology

Transcript

  1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. 2019Ý9Đ25ą

    PivotalKp^zĘàŽņ ē ĥ‚ Ct<WDWH .Ɣ°6ŝ3 Pivotal Greenplumŗ‡
  2. Agenda • %-# Ø ¥-$Ąº#Pivotal Greenplum ĝů Ø Greenplum$Ɣ° -

    Pivotal Greenplum 6 • Ct<WDWH $ Pivotal Greenplum Ø ;zbtƟ•Î"ÒŬĺÇ Ø Ct<WDWHĢŦ Ø Ŝ·4›Ɣ£ĔÄŁ • + -
  3. 3 © Copyright 2019 Pivotal. All rights reserved. Pivotal Greenplum

    ö¥«]‹ • U|P£ĔÄŁºPivotal Data Suite(CPUF9$t;NzL‘Œ) • ƕĽſũ6ƊĮ3Qq|YzDwLêí • ל{ƕĽFLV6¨Į3„©ĜŹĭub8wzL9|BTCQpŭ¾ • dt;f|V(ŒíĺÇ){^cuRCCt<WK8s$¦Ľ.µŦ • MPPÂDB#03ƆƢƐ$¤¡ĹëŦ • U|Px|W¡Ĺ9zx|W+6Н#¤ÒŪ • ĠİüƋ$žÕ"ƢƐ°ĢŦ(U|PÀŠAtmLV9kvQwfv^|T:Jsz etc..) • èů"Ċ#ďƖ"JLTmòƂ6Òĸ3LE|v9<V9|BTCQp • H|][|WƎ¬#02ëŦ.uY9#º~ • Ct<WŒí°FoU:T:H|] žÕ"ÒŬĺÇ#Öé ôâë ƢƐë FLV^b?|kzL
  4. 4 © Copyright 2019 Pivotal. All rights reserved. Pivotal Greenplum

    ¸ô¿ÑÀÏê MPP (Massively Parallel Processing) Ǽ¸ôÔÕÐÇòÁ¸ô¿ÑÀÏê ºòÍÄ×ÀÓ ÚÉ ... ... åÉÍô ÆôÚ x 2 À½íßìòÖòÁõ Ò¹ÉÛÐÏ ËÁèòÓ ÆôÚ x 2  À½í«0{õ ÒôÍ«Ps SQL +ˆÌôÉ ðôÒ¹òÁ• ÉÓíôæòÁn SQL¶}N • ËÁèòÓÆôÚ«£®«J‡ ©0{ßìò¶ : ÛìïîÒôÍ Ýðô½òÈò› ÙôÔÆôÚ8z¶ J-Yb gNetÌÝÓ»¼¸ ºòÍÄ×ÀÓª±´ ËÁèòӍ«^f© Òô̈́ ÛìïîðôÔª±´ ’†ðôÒ¹òÁ åÉÍôÆôڕºòÍÄ×ÀÓÚɕËÁèòÓÆôڕ¢ ¦•žµ²¶p`¡´£®«p`ɺÐϚ²S:Ÿµ¦˜­¡–
  5. 5 © Copyright 2019 Pivotal. All rights reserved. CPU nou

    Ò¹ÉÀI/O¶C ¦`¶’† U:LC CPU nou U:LC CPU nou U:LC CPU nou U:LC CPU nou U:LC CPU¶ ˜œµ©˜ CPU¶ ˜œµ´ CPU¶ ˜œµ´ CPU¶ ˜œµ´ CPU¶ ˜œµ´ Ò¹ÉÀI/O›ãÓî×ÐÀ¨©³•Ø ôÔ§¬`«,‰5ªd›—´ `¡´ž¨§•I/O›CŸµ• HWLM«8z¶ ˜´ž¨›§œ´ 7M&RDB b ^ b ^ C`&DB b ^ b ^ b ^
  6. 6 © Copyright 2019 Pivotal. All rights reserved. ÇÉÑçS:#( 1/2)

    256GB RAM 1.8TB 10Krpm SAS HDD 6L RAID5 (4ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 256GB RAM 256GB RAM 256GB RAM 256GB RAM 256GB RAM 1.8TB 10Krpm SAS HDD 6L RAID5 (4ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) åÉÍôÆôÚ ÉÍòÚº åÉÍôÆôÚ ×ÐÓñôÀɺÐÏ#1 (10Gbps x 52äôÓ) ×ÐÓñôÀɺÐÏ#2 (10Gbps x 52äôÓ) ËÁèòÓÆôÚ#1 ËÁèòÓÆôÚ#2 ËÁèòÓÆôÚ#3 ËÁèòÓÆôÚ#4 Intel E5-2680v3 2CPU,24Core Intel E5-2680v3 2CPU,24Core Intel E5-2680v3 2CPU,24Core Intel E5-2680v3 2CPU,24Core Intel E5-2680v3 2CPU,24Core Intel E5-2680v3 2CPU,24Core 1.8TB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 1.8TB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 1.8TB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 1.8TB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 1.8TB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 1.8TB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 1.8TB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 1.8TB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) mLAG
  7. 7 © Copyright 2019 Pivotal. All rights reserved. ÇÉÑç`S:#( 2/2)

    64GB RAM 300GB 10Krpm SAS HDD 6L RAID5 (4ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 64GB RAM 64GB RAM 64GB RAM 64GB RAM 64GB RAM 300GB 10Krpm SAS HDD 6L RAID5 (4ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) åÉÍôÆôÚ ÉÍòÚº åÉÍôÆôÚ ×ÐÓñôÀɺÐÏ#1 (10Gbps x 52äôÓ) ×ÐÓñôÀɺÐÏ#2 (10Gbps x 52äôÓ) ËÁèòÓÆôÚ#1 ËÁèòÓÆôÚ#2 ËÁèòÓÆôÚ#3 ËÁèòÓÆôÚ#4 Intel E5-2660 2CPU,16Core Intel E5-2660 2CPU,16Core Intel E5-2660 2CPU,16Core Intel E5-2660 2CPU,16Core Intel E5-2660 2CPU,16Core Intel E5-2660 2CPU,16Core 300GB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 300GB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 300GB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 300GB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 300GB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 300GB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 300GB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) 300GB 10Krpm SAS HDD 12L RAID5 (10ÒôÍ + 1Ûíѹ + 1âÐÓÉá¸) åÉÍô ºòÉÍòÉ ÉÍòÚº åÉÍô ºòÉÍòÉ åÉÍôÆôÚ HA ×ÐÓñôÀ Bonding ßìºåíËÁèòÓ#1 æìôËÁèòÓ#1 ßìºåíËÁèòÓ#9 æìôËÁèòÓ#9 ßìºåíËÁèòÓ#17 æìôËÁèòÓ#17 5 æìôËÁèòÓ#25 ßìºåíËÁèòÓ#25 mLAG
  8. Greenplum$Ɣ° v1 v4 v5 v6 2003 - 2009 2010 2015-2018

    2013 2019 U|P<=9\<L BI ÒôÍ *@ VQ.y Z4.y ÁìÝ ÒôÍ Æº½òÉ %`l Ñ¿ÉÓ Ct< W
  9. Cover w/ Image DWH6ŸÑŢ • PBLE|v • !.Ŋš: Ct<W@zdwlLŒíĺÇ •

    SLA$ƢlRJszCuT:Avy|Cx|W • Ƣ¸ĊÒŪë • Ã-ƌ+4£Ĕ{U|PH;>zL • OSS >zP|dt;M $Ŝ· €ľ#1@|dzO|LDWH
  10. Greenplum 6 #%Ë$PostgresĢŦ6k|K v8.4 – 2314 commits v9.0 – 1859

    commits v9.1 – 2035 commits v9.2 – 1945 commits v9.3 – 1603 commits v9.4 – 1964 commits TOTAL: 11,720 Commits Merged Code Quality via Open Source Optimized for Big Data in Greenplum “Customers frequently called out the open-source alignment with PostgreSQL as a strong and cost- effective positive” -- Gartner MQ 2019
  11. Greenplum 6 $OLTPëŦ ďÌ70˜$ëŦº~ • ďƍ$Ë$£ĔŒJLTm%£ĔŒ OLTPŒ$ Ĭ·¡Ĺ6èů 43 •

    ¤$OLTPŀ"þ’6Ÿµ • 24,448 TPS for Update transactions in GP6 • 46,570 TPS for Single Row Insert in GP6 • 140,000 TPS for Select Only Query in GP6 • Ëğ"y|Cx|WŏĹ#uO|LDv|d6ĩĽ Real world analytical database and data warehouse use cases require a mixed workload of long and short queries as well as updates and deletes
  12. Greenplum 6 Replicated Tables create table table_replicated (a int ,

    b text) distributed replicated; insert into table_replicated select id, 'val ' || id from generate_series (1,10000) id; select pg_relation_size('table_replicated'); pg_relation_size ------------------ 917504 create table table_non_replicated (a int , b text) distributed randomly; insert into table_non_replicated select id, 'val ' || id from generate_series (1,10000) id; select pg_relation_size('table_non_replicated'); pg_relation_size ------------------ 458752 With Non Replicated table With Replicated Tables Size is multiplied by the number of primaries select gp_segment_id, count(*) from table_replicated group by 1; ERROR: column "gp_segment_id" does not exist LINE 1: select gp_segment_id, count(*) from ... ^ select gp_segment_id, count(*) from table_non_replicated group by 1; gp_segment_id | count ---------------+------- 0 | 5011 1 | 4989 The field gp_segment_id doesn't exist in replicated tables
  13. Greenplum 6 Replicated Tables Query Plan explain select count(*) from

    table_fact f inner join table_replicated d on f.a = d.a; QUERY PLAN ---------------------------------------------------------------------------------------------------- Aggregate (cost=0.00..874.73 rows=1 width=8) -> Gather Motion 2:1 (slice1; segments: 2) (cost=0.00..874.73 rows=1 width=8) -> Aggregate (cost=0.00..874.73 rows=1 width=8) -> Hash Join (cost=0.00..874.73 rows=50000 width=1) Hash Cond: (table_fact.a = table_replicated.a) -> Seq Scan on table_fact (cost=0.00..432.15 rows=50000 width=4) -> Hash (cost=431.23..431.23 rows=10000 width=4) -> Seq Scan on table_replicated (cost=0.00..431.23 rows=10000 width=4) Optimizer: PQO version 3.29.0 explain select count(*) from table_fact f inner join table_non_replicated d on f.a = d.a; QUERY PLAN --------------------------------------------------------------------------------------------------------------------- Aggregate (cost=0.00..874.31 rows=1 width=8) -> Gather Motion 2:1 (slice3; segments: 2) (cost=0.00..874.31 rows=1 width=8) -> Aggregate (cost=0.00..874.31 rows=1 width=8) -> Hash Join (cost=0.00..874.31 rows=50000 width=1) Hash Cond: (table_fact.a = table_non_replicated.a) -> Redistribute Motion 2:2 (slice1; segments: 2) (cost=0.00..433.15 rows=50000 width=4) Hash Key: table_fact.a -> Seq Scan on table_fact (cost=0.00..432.15 rows=50000 width=4) -> Hash (cost=431.22..431.22 rows=5000 width=4) -> Redistribute Motion 2:2 (slice2; segments: 2) (cost=0.00..431.22 rows=5000 width=4) Hash Key: table_non_replicated.a -> Seq Scan on table_non_replicated (cost=0.00..431.12 rows=5000 width=4) Optimizer: PQO version 3.29.0 With Non Replicated table 1 slice vs 3 slices No redistribution With Replicated table
  14. U|PÀŠ$ôâH/WuO|L$ĪƁ6Őœ • zStandardÀŠ • Facebook OSS°ă9vGuMm • ÀŠ{Ŵ ƢƐ •

    02Ø"U|P#ÀŠ • CREATE TABLEĂ$WITH´#öÑ WITH (compresstype=zstd)
  15. UnloggedT|cv: ƢƐ"}ĊŀU|P$ƢƐ¡Ĺ • WAL6č ¢"UnloggedT|cv#U|P6č ¢ƏÜT |cv‹~#ƢƐ¡Ĺ • }ĊT|cv %¸%":

    ĸŪNRJsz{VtzICJsz ؃.WxRd4" • }Ũŀ"DB $ëƃ%¸ŵù 3 U|P%ħŝń ú4" CtRJq ƅ Æ·#%Í543 . 4" create unlogged table table_unlogged (a int , b text) distributed randomly;
  16. !.£ĔĺÇ6ŊšµŦ Private Cloud Bare-Metal Public Cloud Greenplum Building Blocks •

    The most performant way to run Greenplum on premise • Pivotal Blueprint for Dell reference hardware configs • Superior price/performance; no expensive proprietary hardware • Certified and supported by Pivotal Run Greenplum in Any Environment Greenplum for Kubernetes Other Kubernetes (on VMs or not) Google Container Engine Enterprise & Essentials(OSS K8s) • @|dzO|Lf|L • ;zbtƟ•Î: µûë100%ObV<=9Ouq|Jsz • ¸£ĔdtRVb?|m$-L;RQzDFLVďØ°
  17. Ct<W$¦IJ6ĩĽ • @zUkzWUdx; ◦ ^cuRCCt<W~ģ€Š£Ĕ{AIÄŁ$-#Ħơ$ij–ęë Ŧ6ø”3Pivotal Ť GreenplumďƖ°JLTm ◦ NBquT:(ȶ°)6ŤîfLVdtCT:L

    • Ct<WŏĹ ◦ ŧ­nzTXzL ◦ ŧ­@dJszFzj|ZzV ◦ ŧ­ObV<=99RdDw|W ◦ H|]H;W$LB|kct<I C>uS|v • Ct<WZ;T:c ◦ Nvb_|uzD ◦ DR$-$AZFa|$U:LCLXRdJsRV ◦ JzDvkLP|hLVHAĞï
  18. Ct<W$¦IJ6ĩĽ @zUkzWUdx; • ā£1ĊƚUdx;Ѓ • ź–{Ƅœ$-$ƝÈ6³2Ɯ • `KZLrYRV6åó [|Wœ4Ď (Nvb_|uzD)

    • !$[|WƝÓ.ŧ­œ4Ď ŧ ­çĆ • 5£ʼnß$НçĆ ◦ @zdwlL$çĆ%౐  3 • LPz]; NDnzVhLV6@z t;zuA]u • u]tzL©#pgBouncer6™Ĥ DBLXRdJsRV • gpsnap/gpcronsnap - LEKq |v’ïuLV¨ƜuL V9 • iuq|m6ƢƐ]RC9Rd 3IaaS$LXRdJsRV • CtLP‘]RC9Rd ⣠• ƓģLXRdJsRV]RC9R d6³æ3-$ŧ­ŷÑ • LXRdJsRV%¤ÒŪ4 3-ƢƐ Azure Resource Group Deployment AWS CloudFormation GCP Deployment Manager V M V M V M V M V M X Data Volume Snapshot Restore
  19. !.£ĔĺÇ6ŊšµŦ Run Greenplum in Any Environment Greenplum for Kubernetes Other

    Kubernetes (on VMs or not) Google Container Engine Enterprise & Essentials(OSS K8s)
  20. • µûë Ķōë$3ŏĹ$-$ FzTX'$ř,ƌ, • ƢµĽë ä«ë$-#K8s# 03FzTXŏĹ • ŧ­°$-#“Ľ43K8s@g

    w|P • bvLPRCHj|V3kv QCt<W day-2@gw|Jsz$ -$PKS Container Operator Bringing Cloud Databases On-Premises
  21. ŕćő±"Udx; • }$GreenplumNDnzV (Postgres);zLPzL / Pod / VM(vMotion) • ā£GreenplumCtLP$Ud

    x; • }ƀë$3Ÿĸ µŦ • „©ZRVy|CŷÑ • „©ÑŢ • ŧ­^;dt;z$}Ɨ U dx; µŦ K8s worker 1 K8s worker n PKS/K8s cluster pod pod K8s worker VMs: 8 to 32 GB
  22. Ѓå#ŶŎuO|L$Ŵÿ • Greenplum;zLPzL$¨Ɯ ◦ LT|V/U|P$—õ ◦ .%(6ńě • Greenplum;zLPzL6¨Ɯ3 #0

    3^RQƖĽ/9RdDw|W ◦ ăŰ$;n|K ¸ħŝ°LVw|K 6“Podš6ő±#ƅ­ • VM ™Ĥ3 K8sCtLPžăŰ$ GreenplumNDnzV$Pod ’ï43 ◦ Pod%¸ħŝ°LVw|K#Ÿ‰Ù3 Persistent Volume 1 . . n K8s worker 1 K8s worker n PKS/K8s cluster pod pod
  23. Pivotal¯® …6ł÷ŃöÑ$şß{Śß 12kmđį$ATM24Ċƚ‹ž#200Wv6 á ¢¹© Peter.%Pavan#3†Ĵ$Žė drop function if exists

    get_people(text,text,integer,integer,float,float); CREATE FUNCTION get_people(text,text,integer,integer,float,float) RETURNS integer AS $$ declare linkchk integer; v1 record; v2 record; begin execute 'truncate table results;'; for v1 in select distinct a.id,a.firstname,a.lastname,amount,tran_date,c.lat,c.lng,address,a.description,d.score from people a,transactions b,location c, (SELECT w.id, q.score FROM people w, gptext.search(TABLE(SELECT 1 SCATTER BY 1), 'gpadmin.public.people' , 'Pivotal', null) q WHERE (q.id::integer) = w.id order by 2 desc) d where soundex(firstname)=soundex($1) and a.id=b.id and amount > $3 and (extract(epoch from tran_date) - extract(epoch from now()))/3600 < $4 and st_distance_sphere(st_makepoint($5, $6),st_makepoint(c.lng, c.lat))/1000.0 <= 2.0 and b.locid=c.locid and a.id=d.id loop for v2 in select distinct a.id,a.firstname,a.lastname,amount,tran_date,c.lat,c.lng,address,a.description,d.score from people a,transactions b,location c, (SELECT w.id, q.score FROM people w, gptext.search(TABLE(SELECT 1 SCATTER BY 1), 'gpadmin.public.people' , 'Pivotal', null) q WHERE (q.id::integer) = w.id order by 2 desc) d where soundex(firstname)=soundex($2) and a.id=b.id and amount > $3 and (extract(epoch from tran_date) - extract(epoch from now()))/3600 < $4 and st_distance_sphere(st_makepoint($5, $6),st_makepoint(c.lng, c.lat))/1000.0 <= 2.0 and b.locid=c.locid and a.id=d.id loop execute 'DROP TABLE IF EXISTS out, out_summary;'; execute 'SELECT madlib.graph_bfs(''people'',''id'',''links'',NULL,'||v1.id||',''out'');' ; select 1 into linkchk from out where dist=1 and id=v2.id; if linkchk is not null then insert into results values (v1.id,v1.firstname,v1.lastname,v1.amount,v1.tran_date,v1.lat,v1.lng,v1.address,v1.description,v1.score); insert into results values (v2.id,v2.firstname,v2.lastname,v2.amount,v2.tran_date,v2.lat,v2.lng,v2.address,v2.description,v2.score); end if; end loop; end loop; return 0; end $$ LANGUAGE plpgsql; -- person1 , person 2, amount, duration in hours, longtitude, latitude (in question) select get_people('Pavan','Peter',200,24,103.912680, 1.309432) ; Greenplum POSTGIS functions st_distance_sphere() and st_makepoint() calculate distance between ATM location and reference lat ,long < 2 KM GPText.search() function is used to know if both people work at ‘Pivotal’ Greenplum and Apache MADlib BFS search to know if there are direct or indirect links between people Greenplum Fuzzy String Match function Soundex() to know if people name sounds like ‘Pavan’ or ‘Peter’ Greenplum Time functions to calculate difference in amount withdrawn time < 24 hours Amount > $200 “Pivotal§” Ñ¿ÉÓN - GPText “"›Peter¯ ¬ Pavanª ¦˜´” IGRt - Fuzzy String Match “™˜¶h?i¤¦˜¦” ÁìÝN - Apache MADlib “=/«w5óu5š² 2kmK\«ATM” %`l - PostGIS “24H ” F/HŽD “200 Ôî”
  24. Ļļë$º~: 3,000+ vs 34 LOAD customer data from HDFS and

    put to HIVE DESCRIPTION Column needs to be indexed SEARCH IN Column & WRITE Result to HDFS WRITE CODE : Pulling Data Into Spark Data Frame WRITE CODE : CHECK Soundex WRITE CODE : MATCH SOLR Result WRITE CODE : GRAPH LINK Analysis WRITE CODE : POSTGI S Distance Calculation WRITE CODE : GRAPH LINK Analysi s WRITE CODE : WRITE RESULTS TO HIVE TABLE “Investigate a crime suspect whose name sounds like ‘Pavan’, who knows Peter directly, who withdraw Peter’s $500 at an ATM located 2km from Changi yesterday.” Using a Hadoop Ecosystem: 10 steps, 3000+ Lines of code across 4 different systems 1 2 3 4 5 6 7 8 9 10 Using Greenplum: 1 step, 1 query – 34 Lines of Code One query – using built-in functions: Soundex (sounds like), NLP (work at same company), Machine Learning MADlib (know directly), Time (yesterday), PostGIS (within 2km)
  25. Pivotal Greenplum$£ĔŒOSS • ÌŰġU|PNRV 6¤£Ā{;zU |Pf|LĢĚÏţ • 506Ɔ 3@|dz O|L$ĢĚÏţ{

    ŜŶ{Dtb{āĹ ŜŶƛā • Pivotal Greenplum $Ɔ¤£ĀÒŪ6 Òĸ • ApacheVRdwf vdxK=CV# 2017Ý7ĐĈę: http://madlib.apach e.org Apache MADlib • @|dzO|L$ ĂĜŖ{TBLV£ ĔJLTm • rY]|HvC>u dxNRHb8N RVĜŖĜŖśĕ $ãŽūŅ • http://lucene.apache .org/solr/ Apache Solr • ÁĹ{ŌƚìÅJL Tm6Òĸ3- $PostgreSQL$ôâ OSSoKq|v • ÁĹU|P  IJŞ£juGz 41$Ůā$ř ,·56ñ ¢Ē3 • ÁĹŌƚìÅ$ıŎ 6Ū   ƠŋƇƞ¼¿$ Ƙ"!6ŶŎ 3 • http://postgis.net/ PostGIS • @|dzO|L$£ Ĕ{ŜŶŴĔº$ dxDtlzDŵŻ • ŜŶ 16ĢĚÏ ţ{U|Pk;Yz D"!ĖƉ"U| P£Ĕ űŲ° • U|PH;>zLS |v RŵŻ Þċ²3 • https://www.r- project.org/ R • ĨĽdxDtlzD ŵŻ • F|W Jzdv ñ/ŷŶ4 3 • Ëā$¦ĽµŦ"^ RE|K{t;ct u ĢĚÏţ$Òƈ 6Ôĉ#3 • U|PH;>zL $ų½ë Ƣ • https://www.python. org/ Python
  26. LE|tcvIn-DB£Ĕ ĢĚÏţ • Open source https://github.com/apache/madlib • Downloads and docs

    http://madlib.apache.org/ • Wiki https://cwiki.apache.org/confluence/display/MADLIB/ Apache MADlib: `RDU|P{ĢĚÏţ6SQL @|dzO|L ApacheVRd wfvdxK= CV PostgreSQL & GreenplumĽ U|PH;>zT:LV $-$ã«"ĢĚÏţ DtbŜŶŴĔ
  27. Functions Data Types and Transformations Array and Matrix Operations Matrix

    Factorization • Low Rank • Singular Value Decomposition (SVD) Norms and Distance Functions Sparse Vectors Encoding Categorical Variables Path Functions Pivot Sessionize Stemming May 2018 Graph All Pairs Shortest Path (APSP) Breadth-First Search Hyperlink-Induced Topic Search (HITS) Average Path Length Closeness Centrality Graph Diameter In-Out Degree PageRank and Personalized PageRank Single Source Shortest Path (SSSP) Weakly Connected Components Model Selection Cross Validation Prediction Metrics Train-Test Split Statistics Descriptive Statistics • Cardinality Estimators • Correlation and Covariance • Summary Inferential Statistics • Hypothesis Tests Probability Functions Supervised Learning Neural Networks Support Vector Machines (SVM) Conditional Random Field (CRF) Regression Models • Clustered Variance • Cox-Proportional Hazards Regression • Elastic Net Regularization • Generalized Linear Models • Linear Regression • Logistic Regression • Marginal Effects • Multinomial Regression • Naïve Bayes • Ordinal Regression • Robust Variance Tree Methods • Decision Tree • Random Forest Time Series Analysis • ARIMA Unsupervised Learning Association Rules (Apriori) Clustering (k-Means) Principal Component Analysis (PCA) Topic Modelling (Latent Dirichlet Allocation) Utility Functions Columns to Vector Conjugate Gradient Linear Solvers • Dense Linear Systems • Sparse Linear Systems Mini-Batching PMML Export Term Frequency for Text Vector to Columns Nearest Neighbors • k-Nearest Neighbors Sampling Balanced Random Stratified
  28. īÚÏţ6»7GreenplumŜ·£ĔÄŁ Standby Master … Master Host SQL Interconnect Segment Host

    Node1 Segment Host Node2 Segment Host Node3 Segment Host Node N GPU N … GPU 1 GPU N … GPU 1 GPU N … GPU 1 … GPU N … GPU 1 In-Database Functions Machine learning & statistics & math & graph & utilities Massively Parallel Processing Best of both worlds: GPU- focused and CPU-focused data science workloads • Unified platform for full range of data science workloads • Higher productivity due to no data movement • Persistent data storage and management integrated with core machine learning & API compute engine Supporting the full spectrum of data science workloads: Data preparation, feature generation, machine learning, geospatial, deep learning, etc
  29. Data Types Structured Data Unstructured Data Geographic Data Real Time

    Data Natural Language Data Time Series Data Event Data Network Data Linked Data
  30. ğ"ƣß$U|P6C>u3 • ƕĽů#ÄLV w|K • āŇ©#’14U|P 6þ’ + Ƥ •

    āą©$U|P 1wj |V’ï + Ƥ • Ýб9|A;c 4U|P6Ĝė +  Ƥ In-Memory Database RDBMS data Data Lake HOT DATA WARM DATA COLD DATA 41
  31. 44 © Copyright 2019 Pivotal. All rights reserved. -Š«%`ól9((GIS)ÒôͶ’†N`z©ÜÐÁÒôÍYb'g¶Sq 3

    : $mjkŒeX9(…jkVS(NICT)T ßðÝ·ºî • %`ól9(N(ÜÐÁÒôÍYb)ÇÉÑç >b`c • Pivotal Greenplum›<¥’5NéÈëô1Ÿ • PostGIS: %`9(ÇÉÑç¶0_¡´£®«;6OSS éÈëôî • Apache MADlib: ÒôÍNb«v}N°VQ.y« ¸îÅíÊ綐®£ºòÒôÍàôÉOSSìºÞìíx • ÉÂôÓz©ßìÐÓݾôç 3O • -Š«%`ól9(ÒôͶ’†ª`z • Ãí샑2oBA • a)ÒôͨÌôÇêîÜÐÁÒôÍ«UEfNª±´ {BA • åîÏÉÂôîóåîÏéôÎî©-Wa)|[ÒôÍ« ![
  32. HadoopU|Pw;C Ɔ¤£Ā9|BTCQp ÛÞíÐÀÀì»ÔóÒôÍïºÀ NrìºÞìí ßðÁìç~€ GPText ¤U|Px|W ƢƐU|Pƒý £ĔU|P$ BpRJzD

    ;znou{U|PDuRW ¤U|Px|W 0&ÊƗūĢŦ Pivotal Greenplum 6“4 1$U|P£Ĕ¡ĹÄŁ ºòÒôÍàôÉN Cold Hot Warm Data Temperature PIVOTAL GEMFIRE PIVOTAL GREENPLUM (Data Warehouse)
  33. £Ĕ 9duE|Jsz Z;T:c ;zP|b=|L PIVOTAL GREENPLUM kvQĞƑ°U|P U|PO|L{ U|Pƒý Structured

    Data JDBC, OBBC SQL ANSI SQL r|I ĖƉ"ÒŬĺÇ x|Av LVw|K ˆ RDBMS Spark GemFire Ct<W@ cK=CV LVw|K HDFS JSON, Apache AVRO, Apache Parquet and XML Teradata SQL ˆDB SQL Apache MADlib ĢĚÏţ/ŜŶ/Dtb Python. R, Java, Perl, C dxDtm Apache SOLR TBLV PostGIS ÁĹ{Ōƚ Custom Apps BI / Reporting Machine Learning AI @zdwlL ģ€ŠU|P dtRVb?|m Pivotal Greenplum Kafka ETL Spring Cloud Data Flow Ɔ¤£Ā (MPP) PostgreSQL A|Zv gP];VL E|v$x| U:zD C>u@dT :k;I (GPORCA) y|Cx|W ŏĹ Ëğ" U|PLV9 Command Center SQL…ùë (Hyper-Q) IT ƙĿ `KZL U|PH;> zT:LV ^duRC Ct<W dt;f|V Ct<W kZ|KW Ct<W
  34. + - • Pivotal Greenplum 3$ĵƘ Ø ôâëƢƐëFLV^b?|kzL • 9|BTCQp

    Ø H|]ƚ£Ā&H|]ž£Ā Ø PostgreSQLf|LDWH • ăuu|LGreenplum 6 ĢŦŗ‡ • Ct<WDWH $ Pivotal Greenplum • Ŝ·›Ɣ£ĔÄŁ • TW: @greenplummy • connpass: https://pivotal-japan.connpass.com/