Share
Embed
Riak and Riak CS @ Open Source Conference 名古屋 2013
Riak は高い可用性と運用の簡単さにフォーカスした分散キーバリューストア。Riak CS は Riak の利点を引き継ぎ、S3 互換の Rest API を備えるクラウドストレージ。これらふたつの利点と特徴を説明します。
shino
June 25, 2013
Transcript
38 slides
-
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. & ࢄ KVS
Ϋϥυ ετϨʔδ Basho δϟύϯגࣜձࣾ Senior Software Engineer ࣰݪढ़Ұ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • ΫϥυετϨʔδ Riak
CS • ࢄΩʔόϦϡʔετΞ Riak • ॲཧܥ Erlang/OTP • OS Linux/Solaris/*BSD Riak & Riak CS : ελοΫ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • ΫϥυετϨʔδ Riak
CS • ࢄΩʔόϦϡʔετΞ Riak • ॲཧܥ Erlang/OTP • OS Linux/Solaris/*BSD Φʔϓϯιʔε! Erlang Public License (Mozilla Public License ੜ) -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • ΫϥυετϨʔδ Riak
CS • ࢄΩʔόϦϡʔετΞ Riak • ॲཧܥ Erlang/OTP • OS Linux/Solaris/*BSD Φʔϓϯιʔε! Erlang Public License (Mozilla Public License ੜ) Apache License 2.0 code, wiki, issues on github.com -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ʮͲΜͳނোͰ σʔλৗʹ߹͍ͯ͠·͢͠ γεςϜ͕
ࢭ·Δ͜ͱ͋Γ·ͤΜʯ ͱ͍͏γεςϜଘࡏ͠·ͤΜ = CAPఆཧ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Ωʔίϯηϓτ1 ϊʔυͯ͢ରɺಉ •
ෳϊʔυͰΫϥελΛߏ • ͢ͷϊʔυରͰɺ Ϛελʔ୯Ұোͳ͍ • ͯ͢ͷϊʔυಉͰɺ ϦεΤετΛࡹ͖ɺσʔλΛ อ࣋͢Δ node node node node node -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Ωʔίϯηϓτ2 Ωʔ/ όϦϡʔ/
όέοτ • ΩʔɺόϦϡʔͷϖΞΛόέ οτอଘ͢Δ • όϦϡʔͲͷΑ͏ͳόΠφ ϦͰΑ͍ KEY KEY KEY bucket VALUE VALUE VALUE -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σʔλͷෳ •
160-‐bit ͷΩʔྖҬ = Ring • RingΛִؒͰύʔςΟγϣϯʹׂ • ύʔςγϣϯΛΫϥελͷ֤ϊʔυʹ ׂΓͯΔ node 0 node 1 node 2 node 3 -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σʔλͷෳ •
160-‐bit ͷΩʔྖҬ = Ring • RingΛִؒͰύʔςΟγϣϯʹׂ • ύʔςγϣϯΛΫϥελͷ֤ϊʔυʹ ׂΓͯΔ • bucket / key ͷϋογϡʹΑΓɺ อଘ͢ΔύʔςΟγϣϯΛܾఆ node 0 node 1 node 2 node 3 hash(“bucket/key”) -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σʔλͷෳ •
160-‐bit ͷΩʔྖҬ = Ring • RingΛִؒͰύʔςΟγϣϯʹׂ • ύʔςγϣϯΛΫϥελͷ֤ϊʔυʹ ׂΓͯΔ • bucket / key ͷϋογϡʹΑΓɺ อଘ͢ΔύʔςΟγϣϯΛܾఆ • ࿈ଓ͢ΔύʔςΟγϣϯʹෳΛอଘ node 0 node 1 node 2 node 3 hash(“bucket/key”) -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Ұ࣌োൃੜ࣌ •
Ұ࣌తͳϊʔυোʢnode 2ʣ͕ൃੜ • PUT, GET, DELETEϦΫΤετɺϑΥʔϧ όοΫϊʔυʢnode 0ʣ node 0 node 1 node 2 node 3 hash(“bucket/key”) -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Ұ࣌োϊʔυͷ෮چ࣌ •
Ұ࣌తͳϊʔυোʢnode 2ʣ͕ൃੜ • PUT, GET, DELETEϦΫΤετɺϑΥʔϧ όοΫϊʔυʢnode 0ʣ • োϊʔυͷ෮چʢnode 2ʣ • “Handoff”ʹΑΓɺσʔλΛϑΥʔϧόο Ϋϊʔυʢnode 0ʣ͔Β෮چϊʔυ ʢnode 2ʣҠߦ • ௨ৗӡ༻Λ࠶։ node 0 node 1 node 2 node 3 hash(“bucket/key”) -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σϞ 1. 1ϊʔυ͚ͩͷΫϥελͰ࢝ΊΔ
2. 10 ݸͷKey/Value ΛՃ 3. 3ϊʔυΛΫϥελʹՃ (Join) 4. ࠷ॳͷϊʔυΛؚΊɺ2ϊʔυΛ kill -9 5. ͯ͢ͷΩʔΛऔಘ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • ಛఆͷbucket /
keyΛࢦఆ͠ɺඥ͍ͮͨΦϒδΣΫτΛऔಘ • ಛఆͷbucketsʹඥ͍͍ͮͯΔશͯͷkeyΛऔಘ Ωʔࢦఆ • ؆୯ͳΫΤϦݴޠʢHTTPʣΛαϙʔτͨ͠શจݕࡧ • JSON, plain text, XML and ErlangͳͲͷmime-‐typeΛαϙʔτ • ·ͩβ (Yokozuna) શจݕࡧ • ΫΤϦʔʹ༻Ͱ͖ΔλάɺΛΦϒδΣΫτอଘ࣌ʹઃఆ • ΦϒδΣΫτͷλά໊͓ΑͼΛࢦఆͯ͠ΦϒδΣΫτΛ औಘ Secondary Index (2i) • σʔλͷ͍߹ΘͤɺϑΟϧλϦϯάͷࢄɺղੳͱूܭ • JavaScript ͱErlang MapReduce RiakͷσʔλͷΞΫηε -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • Riak KVS ͷ্ʹ࣮
• ؆୯ʹ͑ΔΫϥυετϨʔδ • AWS S3 ޓAPI ಛ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Riak CS Architecture
Stanchion ΦϒδΣΫτ ૢ࡞ block block block block block block manifest ü ߴՄ༻ੑ ü ࢄஔ ü ෳ όέοτ ૢ࡞ Ϣʔβૢ࡞ɺ Ϩϙʔτ S3 REST API -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. APIͱΠϯλϑΣʔε
• AWS S3 REST API४ڌ • ҰൠతͳS3 ͚ϥΠϒϥϦɺπʔϧΛར༻Մೳ • REST GET, PUTͱDELETE ΦϖϨʔγϣϯ • S3-‐style ACLsɺόέοτϙϦγʔ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σϞ DragonDisk
http://www.dragondisk.com s3cmd http://s3tools.org/s3cmd -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • Riak KVS ͷ্ʹ࣮
• ؆୯ʹ͑ΔΫϥυετϨʔδ • AWS S3 ޓAPI ·ͱΊ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. σʔληϯλʔؒ ϨϓϦέʔγϣϯ
ෳDC ؒͰͷɺยํ·ͨ ྆ํͷσʔλಉظ ར༻త • ॏେࡂԼͰαʔϏεΛܧଓ • σʔλϩʔΧϦςΟ • ΞΫςΟϒόοΫΞοϓ • Produc_onΫϥελʔͱStage༻ ΫϥελʔʹΑΓݕূڥΛߏங Primary Cluster (DC#1) Secondary Cluster (DC#2) Secondary Cluster (DC#3) Client Update -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ใͷอ •
֦ுੑͱϢʔβΤΫεϖϦΤϯεͷύϑΥ ʔϚϯεվળ • bestbuy.comͱখചΓళฯͰ༻͢ΔΦϯ ϥΠϯΧλϩάɺϨʔςΟϯάʹRiak Λબɻ • Holiday ShoppingʢΫϦεϚεηʔϧͳ Ͳʣ࣌ʹ߹ΘͤͯϊʔυՃɻ • Bestbuy.comͷϗʔϜϖʔδϨϯμϦϯά ͷSLA̍ඵҎɻ • 2013ͷHoliday seasonʹɺSKUͰ26ඦ ສ͔Β500ඦສ֦ுΛఆɻ • Amazon AWSͷෳAvailability Zone্ʹRiak ΫϥελʔΛߏங -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ΦϯϥΠϯࠂ • DCϨϓϦέʔγϣϯͱ֦ுੑͷඞཁ͔Β
MySQL → Cassandra → Riak ͱϦϓϨʔε • ϢʔβΞΫςΟϏςΟσʔλ͓ΑͼτϥϑΟο ΫσʔλΛอଘ • τϥϑΟοΫσʔλMySQL͔ΒҠߦʢෳDC ؒͷσʔλϨϓϦέʔγϣϯ͕ඞཁʣ • ϢʔβσʔλCassandra͔ΒҠߦʢbackward compa_bility͕֬อͰ͖͍ͯͳ͍͜ͱ͕աڈͷ ϦϦʔεʹ͋Γɺ৴༻Ͱ͖ͳ͍ͨΊܾஅʣ • 5ΧॴͷDCؒͰσʔλϨϓϦέʔγϣϯ • 2011ʹ4 trillionͷadσʔλΛѻ͏ɻ advertising conomic potential of uding OpenX mpTime) provide a bining ad serving, an ad exchange, a Supply Side Platform, and ach year. OpenX uses Riak for user and trafficking data behind its data services API. They selected Riak due to its highly available, low-latency, redundant architecture. OpenX also uses Riak’s multi- datacenter replication across several data centers, providing up-to-date data throughout its global infrastructure. For more details about how OpenX uses Riak, check out the video of Anthony Molinaro, OpenX engineer, speaking at RICON2012, Basho’s 2012 developer conference. ng technology provider. gencies, mobile operators, active and measurable obile devices. In 2009, rly all of the broadcasters le operators. With the dly, they needed to move to an architecture that could gracefully new platform because it is distributed, scalable, and highly mes of traffic. they opted to build two geographically separated, mirrored sites ation feature. As Marcus Kern, VP of Technology at Velti, ver 140 customers. We cannot afford a single minute of d exceed our requirements for scale, data durability, and 2009࣌ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. ϝχϡʔγεςϜ • ॳAmazon
S3Λ༻; On demandϝχϡʔͷϥϯ υτϦοϓϨΠςϯγʔͷݮΛୡ • ߴͳಡΈग़͠/ॻࠐΈͷͨΊRiakΛબ • Video On demandϝχϡʔͷΞάϦήʔτʹඞཁͳ σʔλΛอଘ • ϚʔέςΟϯάΩϟϯϖʔϯʹকདྷ༻͢ΔͰ͋Ζ ͏Ϣʔβؔ࿈ͷใΛอଘ • ຖɺϦϞʔτίϯτϩʔϥ͔Βͷ25MΫϦοΫ Λॲཧ • On demandϝχϡʔͷͨΊʹ̏ΧॴͷDCʹRiakΫϥ ελʔΛߏஙɻ • ϚʔέςΟϯά༻ʹ̐ͭͷΫϥελʔΛߏங -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. • MySQL, Oracle͔ΒϨΠςϯγʔվળ͓Αͼίετ
ݮͨΊRiakϦϓϨʔε • Electronic Health RecordsʢॲํᝦใʣΛҩऀ ϝσΟΧϧγεςϜ͕༷ʑͳσόΠε͔ΒΞΫη ε • 5.5MͷશσϯϚʔΫࠃຽ͚ Danish Health Services ϔϧεέΞใཧ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Riak CS ͷϢʔεέʔε
ύϒϦοΫ ΫϥυεετϨʔδ AWSҎ֎ͷS3४ڌͷ ετϨʔδ ΫϥυυϥΠϒ (ҰൠతͳίϯςϯπετϨʔδ) Backup-‐as-‐a-‐Service ΞʔΧΠϒετϨʔδ ࣾһͱࣾ෦ͷͨ ΊͷετϨʔδ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. :BIPP+"1"/ • ΠϯλʔωοτγϣοϐϯάαΠτͷϓ
ϥοτϑΥʔϜΛYahoo!δϟύϯ͕ఏڙ • γϣοϐϯάαΠτͷը૾σʔλΛRiak CSʹετΞ • ొΦϒδΣΫτɿ20ສ݅ʢ2012 ࣌ʣ • ϦΫΤετɿ450 req/sec • Ϩεϙϯεɿ10ms – 80ms • ߏஙɿ1ɻ • S3ޓΫϥυετϨʔδαʔϏεΛఏڙ • 2ΧॴͷDCؒͰσʔλϨϓϦέʔγϣϯ -
©2013 BASHO TECHNOLOGIES INC. ALL RIGHTS RESERVED. Questions? • Twitter:
@BashoJapan • ML: [email protected] • Basho/TED @ లࣔϒʔε • Me: [email protected] @itawasa