Riak and Riak CS @ Open Source Conference 名古屋 2013

Riak は高い可用性と運用の簡単さにフォーカスした分散キーバリューストア。Riak CS は Riak の利点を引き継ぎ、S3 互換の Rest API を備えるクラウドストレージ。これらふたつの利点と特徴を説明します。

Transcript

1. Basho Technologies 2013-06-22 ΦʔϓϯιʔεΧϯϑΝϨϯε 2013 @ ໊ݹ԰
2. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. & ෼ࢄ KVS Ϋϥ΢υ ετϨʔδ Basho δϟύϯגࣜձࣾ Senior Software Engineer ࣰݪ
   ढ़Ұ
   
3. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. •  Ϋϥ΢υετϨʔδ
   Riak CS
   •  ෼ࢄ
   ΩʔόϦϡʔετΞ
   Riak
   •  ॲཧܥ
   Erlang/OTP
   •  OS
   Linux/Solaris/*BSD
   Riak & Riak CS : ελοΫ
   
   
4. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. •  Ϋϥ΢υετϨʔδ
   Riak CS
   •  ෼ࢄ
   ΩʔόϦϡʔετΞ
   Riak
   •  ॲཧܥ
   Erlang/OTP
   •  OS
   Linux/Solaris/*BSD
   Φʔϓϯιʔε!
   
   Erlang Public License (Mozilla Public License ೿ੜ)
   
5. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. •  Ϋϥ΢υετϨʔδ
   Riak CS
   •  ෼ࢄ
   ΩʔόϦϡʔετΞ
   Riak
   •  ॲཧܥ
   Erlang/OTP
   •  OS
   Linux/Solaris/*BSD
   Φʔϓϯιʔε!
   
   Erlang Public License (Mozilla Public License ೿ੜ)
   Apache License 2.0 code, wiki, issues on github.com
6. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED.
7. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Riak ͸
   ෼ࢄ
   KVS
8. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. CAPఆཧ
9. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ʮͲΜͳނোͰ΋ σʔλ͸ৗʹ੔߹͍ͯ͠·͢͠ γεςϜ͕ ࢭ·Δ͜ͱ͸͋Γ·ͤΜʯ ͱ͍͏γεςϜ͸ଘࡏ͠·ͤΜ = CAPఆཧ
10. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • ӡ༻ͷ༰қ͞
    • ߴՄ༻ੑ
    • ਫฏ֦ுੑ
    ઃܭϙϦγʔ
    
11. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Ωʔίϯηϓτ1 ϊʔυ͸͢΂ͯର౳ɺಉ౳ •  ෳ਺ϊʔυͰΫϥελΛߏ੒   •  ͢΂ͷϊʔυ͸ର౳Ͱɺ   Ϛελʔ΍୯Ұো֐఺͸ͳ͍   •  ͢΂ͯͷϊʔυ͸ಉ౳Ͱɺ   ϦεΤετΛࡹ͖ɺσʔλΛ อ࣋͢Δ   node   node   node   node   node  
12. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Ωʔίϯηϓτ2 Ωʔ
    / όϦϡʔ
    / όέοτ •  ΩʔɺόϦϡʔͷϖΞΛόέ οτ΁อଘ͢Δ
    •  όϦϡʔ͸ͲͷΑ͏ͳόΠφ ϦͰ΋Α͍   KEY KEY KEY bucket   VALUE VALUE VALUE
13. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σʔλͷෳ੡
    •  160-­‐bit  ੔਺ͷΩʔྖҬ  =  Ring   •  RingΛ౳ִؒͰύʔςΟγϣϯʹ෼ׂ   •  ύʔςγϣϯΛΫϥελͷ֤ϊʔυʹ ׂΓ౰ͯΔ   node  0   node  1   node  2   node  3  
14. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σʔλͷෳ੡
    •  160-­‐bit  ੔਺ͷΩʔྖҬ  =  Ring   •  RingΛ౳ִؒͰύʔςΟγϣϯʹ෼ׂ   •  ύʔςγϣϯΛΫϥελͷ֤ϊʔυʹ ׂΓ౰ͯΔ   •  bucket  /  key  ͷϋογϡ஋ʹΑΓɺ   อଘ͢ΔύʔςΟγϣϯΛܾఆ   node  0   node  1   node  2   node  3   hash(“bucket/key”)  
15. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σʔλͷෳ੡
    •  160-­‐bit  ੔਺ͷΩʔྖҬ  =  Ring   •  RingΛ౳ִؒͰύʔςΟγϣϯʹ෼ׂ   •  ύʔςγϣϯΛΫϥελͷ֤ϊʔυʹ ׂΓ౰ͯΔ   •  bucket  /  key  ͷϋογϡ஋ʹΑΓɺ   อଘ͢ΔύʔςΟγϣϯΛܾఆ   •  ࿈ଓ͢ΔύʔςΟγϣϯʹෳ੡Λอଘ   node  0   node  1   node  2   node  3   hash(“bucket/key”)  
16. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Ұ࣌ো֐ൃੜ࣌
    •  Ұ࣌తͳϊʔυো֐ʢnode  2ʣ͕ൃੜ   •  PUT,  GET,  DELETEϦΫΤετ͸ɺϑΥʔϧ όοΫϊʔυʢnode  0ʣ΁   node  0   node  1   node  2   node  3   hash(“bucket/key”)  
17. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Ұ࣌ো֐ϊʔυͷ෮چ࣌
    •  Ұ࣌తͳϊʔυো֐ʢnode  2ʣ͕ൃੜ   •  PUT,  GET,  DELETEϦΫΤετ͸ɺϑΥʔϧ όοΫϊʔυʢnode  0ʣ΁   •  ো֐ϊʔυͷ෮چʢnode  2ʣ   •  “Handoff”ʹΑΓɺσʔλΛϑΥʔϧόο Ϋϊʔυʢnode  0ʣ͔Β෮چϊʔυ ʢnode  2ʣ΁Ҡߦ   •  ௨ৗӡ༻Λ࠶։   node  0   node  1   node  2   node  3   hash(“bucket/key”)  
18. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σϞ 1.  1ϊʔυ͚ͩͷΫϥελͰ࢝ΊΔ 2.  10 ݸͷ
    Key/Value Λ௥Ճ 3.  3ϊʔυΛΫϥελʹ௥Ճ (Join) 4.  ࠷ॳͷϊʔυΛؚΊɺ2ϊʔυΛ kill -9 5.  ͢΂ͯͷΩʔΛऔಘ
19. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. •  ಛఆͷbucket  /  keyΛࢦఆ͠ɺඥ͍ͮͨΦϒδΣΫτΛऔಘ   •  ಛఆͷbucketsʹඥ͍͍ͮͯΔશͯͷkeyΛऔಘ   Ωʔࢦఆ
    •  ؆୯ͳΫΤϦݴޠʢHTTPʣΛαϙʔτͨ͠શจݕࡧ   •  JSON,  plain  text,  XML  and  ErlangͳͲͷmime-­‐typeΛαϙʔτ
    •  ·ͩβ  (Yokozuna)   શจݕࡧ
    •  ΫΤϦʔʹ࢖༻Ͱ͖ΔλάɺΛΦϒδΣΫτอଘ࣌ʹઃఆ   •  ΦϒδΣΫτͷλά໊͓Αͼ஋Λࢦఆͯ͠ΦϒδΣΫτΛ औಘ   Secondary Index (2i) •  σʔλͷ໰͍߹ΘͤɺϑΟϧλϦϯάͷ෼ࢄɺղੳͱूܭ   •  JavaScript  ͱ
    Erlang   MapReduce Riak಺ͷσʔλ΁ͷΞΫηε  
20. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • ӡ༻ͷ༰қ͞
    • ߴՄ༻ੑ
    • ਫฏ֦ுੑ
    ·ͱΊ
    
21. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED.
22. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • Riak  KVS  ͷ্ʹ࣮૷
      • ؆୯ʹ࢖͑ΔΫϥ΢υετϨʔδ   • AWS  S3  ޓ׵
    API     ಛ௃
    
23. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Riak CS Architecture
    Stanchion ΦϒδΣΫτ ૢ࡞
    block
    block
    block
    block
    block
    block
    manifest
    ü  ߴՄ༻ੑ ü  ෼ࢄ഑ஔ ü  ෳ੡ όέοτ ૢ࡞
    Ϣʔβૢ࡞ɺ Ϩϙʔτ
    S3 REST API
    
    
24. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. APIͱΠϯλϑΣʔε
        •  AWS  S3  REST  API४ڌ   •  Ұൠతͳ
    S3  ޲͚ϥΠϒϥϦɺπʔϧΛར༻Մೳ   •  REST  GET,  PUTͱDELETE  ΦϖϨʔγϣϯ   •  S3-­‐style  ACLsɺόέοτϙϦγʔ
     
25. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σϞ
    DragonDisk http://www.dragondisk.com
    s3cmd http://s3tools.org/s3cmd
    
26. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • Riak  KVS  ͷ্ʹ࣮૷
      • ؆୯ʹ࢖͑ΔΫϥ΢υετϨʔδ   • AWS  S3  ޓ׵
    API     ·ͱΊ
    
27. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Enterprise ঎༻൛
    σʔληϯλʔؒϨϓϦέʔγϣϯ   24x7  αϙʔτ  
28. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σʔληϯλʔؒ
    ϨϓϦέʔγϣϯ
    
    
    ෳ਺
    DC  ؒͰͷɺยํ޲·ͨ͸   ྆ํ޲ͷσʔλಉظ     ར༻໨త   •  ॏେࡂ֐ԼͰ΋αʔϏεΛܧଓ   •  σʔλϩʔΧϦςΟ
      •  ΞΫςΟϒόοΫΞοϓ   •  Produc_onΫϥελʔͱStage༻   ΫϥελʔʹΑΓݕূ؀ڥΛߏங   Primary   Cluster  (DC#1)   Secondary   Cluster  (DC#2)   Secondary   Cluster  (DC#3)   Client   Update  
29. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Use  Cases  /  Case  Studies  
30. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ओͳΧελϚʔ
    h`p://basho.com/riak-­‐users/ʹߋʹଟ͘ͷΧελϚʔ৘ใ͕ޚ࠲͍·͢ɻ  
31. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ੡඼৘ใͷอ؅
    •  ֦ுੑͱϢʔβΤΫεϖϦΤϯεͷύϑΥ ʔϚϯεվળ   •  bestbuy.comͱখചΓళฯͰ࢖༻͢ΔΦϯ ϥΠϯ੡඼ΧλϩάɺϨʔςΟϯάʹRiak Λબ୒ɻ   •  Holiday  ShoppingʢΫϦεϚεηʔϧͳ Ͳʣ࣌ʹ߹Θͤͯϊʔυ௥Ճɻ   •  Bestbuy.comͷϗʔϜϖʔδϨϯμϦϯά ͷSLA͸̍ඵҎ಺ɻ   •  2013೥ͷHoliday  seasonʹ͸ɺSKU਺Ͱ26ඦ ສ͔Β500ඦສ΁֦ுΛ૝ఆɻ   •  Amazon  AWSͷෳ਺Availability  Zone্ʹRiak ΫϥελʔΛߏங  
32. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ΦϯϥΠϯ޿ࠂ
    •  DCϨϓϦέʔγϣϯͱ֦ுੑͷඞཁ͔Β   MySQL  →  Cassandra  →  Riak  ͱϦϓϨʔε   •  ϢʔβΞΫςΟϏςΟσʔλ͓ΑͼτϥϑΟο ΫσʔλΛอଘ   •  τϥϑΟοΫσʔλ͸MySQL͔ΒҠߦʢෳ਺DC ؒͷσʔλϨϓϦέʔγϣϯ͕ඞཁʣ   •  Ϣʔβσʔλ͸Cassandra͔ΒҠߦʢbackward   compa_bility͕֬อͰ͖͍ͯͳ͍͜ͱ͕աڈͷ ϦϦʔεʹ͋Γɺ৴༻Ͱ͖ͳ͍ͨΊܾஅʣ   •  5ΧॴͷDCؒͰσʔλϨϓϦέʔγϣϯ   •  2011೥ʹ4  trillionͷadσʔλΛѻ͏ɻ   advertising conomic potential of uding OpenX mpTime) provide a bining ad serving, an ad exchange, a Supply Side Platform, and ach year. OpenX uses Riak for user and trafficking data behind its data services API. They selected Riak due to its highly available, low-latency, redundant architecture. OpenX also uses Riak’s  multi- datacenter replication across several data centers, providing up-to-date data throughout its global infrastructure. For more details about how OpenX uses Riak, check out the video of Anthony Molinaro, OpenX engineer, speaking at RICON2012,  Basho’s  2012   developer conference. ng technology provider. gencies, mobile operators, active and measurable obile devices. In 2009, rly all of the broadcasters le operators. With the dly, they needed to move to an architecture that could gracefully new platform because it is distributed, scalable, and highly mes of traffic. they opted to build two geographically separated, mirrored sites ation feature. As Marcus Kern, VP of Technology at Velti, ver  140  customers.  We  cannot  afford  a  single  minute  of   d exceed our requirements for scale, data durability, and 2009೥࣌఺  
33. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ϝχϡʔγεςϜ
    •  ౰ॳAmazon  S3Λ࢖༻;  On  demandϝχϡʔͷϥ΢ϯ υτϦοϓϨΠςϯγʔͷ௿ݮΛୡ੒   •  ߴ଎ͳಡΈग़͠/ॻࠐΈͷͨΊRiakΛબ୒   •  Video  On  demandϝχϡʔͷΞάϦήʔτʹඞཁͳ σʔλΛอଘ   •  ϚʔέςΟϯάΩϟϯϖʔϯʹকདྷ࢖༻͢ΔͰ͋Ζ ͏Ϣʔβؔ࿈ͷ৘ใΛอଘ   •  ຖ೔ɺϦϞʔτίϯτϩʔϥ͔Βͷ25MΫϦοΫ਺ Λॲཧ   •  On  demandϝχϡʔͷͨΊʹ̏ΧॴͷDCʹRiakΫϥ ελʔΛߏஙɻ   •  ϚʔέςΟϯά༻ʹ̐ͭͷΫϥελʔΛߏங  
34. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. •  MySQL,  Oracle͔ΒϨΠςϯγʔվળ͓Αͼίετ ௿ݮͨΊRiak΁ϦϓϨʔε   •  Electronic  Health  Recordsʢॲํᝦ৘ใʣΛҩऀ΍ ϝσΟΧϧγεςϜ͕༷ʑͳσόΠε͔ΒΞΫη ε   •  5.5MͷશσϯϚʔΫࠃຽ޲͚   Danish Health Services ϔϧεέΞ৘ใ؅ཧ
    
35. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Riak CS  ͷϢʔεέʔε
    ύϒϦοΫ
    Ϋϥ΢υεετϨʔδ
    AWSҎ֎ͷS3४ڌͷ ετϨʔδ   Ϋϥ΢υυϥΠϒ   (ҰൠతͳίϯςϯπετϨʔδ)   Backup-­‐as-­‐a-­‐Service   ΞʔΧΠϒετϨʔδ
    ࣾһͱࣾ಺෦໳ͷͨ ΊͷετϨʔδ
    
36. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. :BIPP
    +"1"/
    •  ΠϯλʔωοτγϣοϐϯάαΠτͷϓ ϥοτϑΥʔϜΛYahoo!δϟύϯ͕ఏڙ   •  γϣοϐϯάαΠτͷը૾σʔλΛRiak   CSʹετΞ   •  ొ࿥ΦϒδΣΫτ਺ɿ20ສ݅ʢ2012೥຤ ࣌఺ʣ   •  ϦΫΤετ਺ɿ450  req/sec   •  Ϩεϙϯεɿ10ms  –  80ms   •  ߏஙɿ1೔ɻ   •  S3ޓ׵Ϋϥ΢υετϨʔδαʔϏεΛఏڙ   •  2ΧॴͷDCؒͰσʔλϨϓϦέʔγϣϯ  
37. ©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Questions? •  Twitter: @BashoJapan •  ML: riak-users-jp@lists.basho.com •  Basho/TED @ లࣔϒʔε •  Me: shino@basho.com @itawasa
38. Basho Technologies 2013-06-22 ΦʔϓϯιʔεΧϯϑΝϨϯε 2013 @ ໊ݹ԰