Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Riak on EC2

UENISHI Kota
September 28, 2013

Riak on EC2

AWSでRiakを使うときのいろいろお話しました。

UENISHI Kota

September 28, 2013
Tweet

More Decks by UENISHI Kota

Other Decks in Technology

Transcript

  1. Riak on EC2
    JAWS Festa Osaka
    ্੢߁ଠ
    2013/9/28

    View Slide

  2. Who are you??
    •  ্੢߁ଠ (@kuenishi)

    •  Bashoδϟύϯ

    •  ෼ࢄγεςϜྺ6೥
    •  Bashoྺ1೥ɺ΍ͬͱίʔ
    υॻ͕࣌ؒ͘Ͱ͖͖ͯͨ

    •  ޷͖ͳ˓˓

    View Slide

  3. Agenda
    •  What is Riak

    •  U NO EC2?

    •  Riak + EC2 = ?

    •  ߏ੒ྫ

    •  ࣄྫʢলུ͞Ε·ͨ͠ʣ

    •  Software Design 10/18 ങͬͯͶʂ

    View Slide

  4. WAT IS RIAK?

    View Slide

  5. ཧ૝ͷσʔλϕʔε
    ʮͲΜͳނো͕ىͬͯ͜΋

    σʔλΛৗʹ੔߹ͤͭͭ͞

    ɹαʔϏε΋ࢭ·Δ͜ͱ͸ͳ͍

    ɹσʔλϕʔεʯ


    ͸ଘࡏ͠·ͤΜ = CAPఆཧ

    View Slide

  6. •  Amazon Dynamo࿦จΛϕʔεʹઃܭ͞Εͨ෼ࢄKVS

    •  Φʔϓϯιʔε

    •  ҎԼʹॏ఺Λஔ͍͍ͯΔ

    •  Մ༻ੑʢAvailabilityʣ

    •  ଱ো֐ੑʢFault-toleranceʣ

    •  ӡ༻ͷ༰қੑʢOperational simplicityʣ

    •  εέʔϥϏϦςΟʢScalabilityʣ



    View Slide

  7. Dynamo ࿦จΑΓ
    •  Dynamo: Amazon’s Highly Available Key-value Store

    •  Մ༻ੑୈҰɿৗʹॻ͖ࠐΈ͕Ͱ͖Δ͜ͱ

    View Slide

  8. Riak͸ Erlang/OTP্Ͱಈ࡞
    •  ΫϥελϦϯά΍ϊʔυؒͷϝοηʔδϯάͳͲ͸ɺErlang/
    OTPʹ·͔͍ͤͯΔ

    •  Erlang/OTPͱ͸ʁ

    •  ݩʑ͸ΤϦΫιϯ͕ࣾి࿩ަ׵ػϓϩάϥϜͷͨΊʹ։ൃɻ

    View Slide

  9. Riak͸෼ࢄKVS
    ɾRiak ͸ӬଓੑΛ࣋ͬͨ Key Value Store

    View Slide

  10. RiakͷσʔλͱRDBͷσʔλൺֱ
    3JBLͷσʔλߏ଄ 3%#ͷσʔλߏ଄
    PCKFDU SFDPSEʢߦʣ
    #VDLFU 5BCMF ද

    K V PK

    View Slide

  11. ɾ1ͭͷRiakϊʔυ͸ɺOSϓϩηε্͸1ͭͷErlangΞϓϦ

    View Slide

  12. ɾ෼ࢄσʔλϕʔεͰ͋ΔRiak͸ෳ਺ϊʔυͰߏ੒

    View Slide

  13. ɾBucket͸ɺKeyΛ෼ྨ͢ΔͨΊͷ໊લۭؒ

    View Slide

  14. ɾσʔλ͸ҟͳΔRiakϊʔυʹࣗಈෳ੡͞ΕΔ

    Riakʹ͓͚Δ
    σʔλͷࣗಈϨϓϦέʔγϣϯ
    K V K V
    OPEF OPEF OPEF OPEF

    View Slide

  15. ɾΞΫηεͰ͖ͳ͍ϊʔυ΁ͷΞΫηε͸εΩοϓɻ

    View Slide

  16. ɾো֐ͳͲͰࣦΘΕͨσʔλͷෳ੡͸ࣗಈ෮چ͞ΕΔ

    Riakʹ͓͚Δ
    σʔλͷࣗಈ෮چ
    OPEF OPEF OPEF OPEF
    OPEF OPEF OPEF OPEF

    View Slide

  17. ɾϊʔυ௥Ճ࣌ͳͲɺඞཁʹԠͯ͡࠶഑ஔ͕ߦΘΕΔ

    Riakʹ͓͚Δ
    σʔλͷࣗಈ࠶഑ஔ
    OPEF OPEF OPEF OPEF
    OPEF OPEF OPEF OPEF
    +PJO

    View Slide

  18. ɾෳ਺ͷRiakΫϥελʔؒͰɺσʔλͷϨϓϦέʔγϣϯ

    View Slide

  19. RiakͷAPI
    •  HTTP  API  or  Protocol  Buffers  
    •  put,  get,  delete,  list_keys,  counter  
    •  PBͳΒ:  http://github.com/basho/riak_pb  
    •  src/riak.proto,  src/riak_kv.proto  
    •  Ruby,  Java,  Python,  Erlang,  PHP,  …  

    View Slide

  20. ͭ·Γ… Riak͸
    •  Amazon Dynamo࿦จΛϕʔεʹઃܭ͞Εͨ෼ࢄKVS

    •  ҎԼʹॏ఺Λஔ͍͍ͯΔ

    •  Մ༻ੑʢAvailabilityʣ

    •  ଱ো֐ੑʢFault-toleranceʣ

    •  ӡ༻ͷ༰қੑʢOperational simplicityʣ

    •  εέʔϥϏϦςΟʢScalabilityʣ



    View Slide

  21. ͔ࣗ͠͠લͷϋʔυͩͱ…
    •  εέʔϧ͕େ͖͍ˍܭըՄೳͳΒ͍҆

    •  ϋʔυ΢ΣΞ΍ωοτϫʔΫ͔Βࣗ෼޷
    Έʹ࡞ΕΔ

    •  ෛՙ΍ྔʹ೾͕͋ͬͯɺ
    ࠷େ஋ʹ߹Θͤͯௐୡ͠
    ͳ͍ͱ͍͚ͳ͍ͱׂߴ

    •  ࡉ͔͍ͱ͜Ζ·Ͱߟ͑ͯ
    ߏ੒͠ͳ͍ͱ͍͚ͳ͍

    View Slide

  22. ͍͘Βιϑτ΢ΣΞͰؤுͬͯ΋
    •  Մ༻ੑ

    •  ωοτϫʔΫ΍ిݯ͕མ͍ͪͯͨΒҙຯ͕ͳ͍

    •  ଱ো֐ੑ

    •  ωοτϫʔΫ΍ిݯ͕୔ࢁյΕ͍ͯͨΒҙຯ͕ͳ͍

    •  ӡ༻

    •  ϋʔυ΢ΣΞ΍ωοτϫʔΫͷӡ༻͸େม

    •  εέʔϥϏϦςΟ

    •  ؊৺ͳͱ͖ʹαʔόʔΛௐୡͰ͖ͳ͚Ε͹ҙຯ͕ͳ͍

    View Slide

  23. OMG

    View Slide

  24. BUT… YOU KNOW EC2

    View Slide

  25. AWS EC2
    •  Amazon ͕ࣗ෼ͨͪͷԾ૝Ϛγϯ؅ཧͷΠϯϑϥΛ
    αʔϏεԽͨ͠΋ͷ

    •  ҎԼͷಛ௃

    •  Մ༻ੑʢAvailabilityʣ

    •  ଱ো֐ੑʢFault-toleranceʣ

    •  ӡ༻ͷ༰қੑʢOperational simplicityʣ

    •  εέʔϥϏϦςΟʢScalabilityʣ



    View Slide

  26. AWS EC2͕͋Ε͹…
    •  Մ༻ੑ

    •  ωοτϫʔΫ΍ిݯ͸جຊతʹམͪͳ͍ͱࢥ͍͍ͬͯ

    •  ଱ো֐ੑ

    •  ωοτϫʔΫ΍ిݯ͸جຊతʹ৑௕Խ͞Ε͍ͯΔ

    •  ӡ༻

    •  ϋʔυ΢ΣΞ΍ωοτϫʔΫͷӡ༻͸΍Βͳ͍͍ͯ͘

    •  εέʔϥϏϦςΟ

    •  ৽͍͠Ϛγϯ΋σΟεΫ΋APIͰҰൃήοτʂ

    View Slide

  27. ͜ΕͰউͭΔ

    View Slide

  28. RIAK + EC2 =
    EASY OPERATION AND SCALABLE SYSTEM

    View Slide

  29. Riakͷߏங
    •  Bashoͷapt-lineΛ௥Ճ

    •  Πϯετʔϧ

    •  sudo  apt-­‐get  install  riak  
    •  ઃఆϑΝΠϧΛ2ͭࣗ෼޲͚ʹมߋ

    •  ΧʔωϧपΓΛ͍͔ͭ͘ઃఆʢεέδϡʔϥͳͲͳͲʣ

    •  ެࣜAMI΋͋ΔΑʂami-­‐7d50c77c  
    •  ˠ http://docs.basho.com/riak/latest

    View Slide

  30. Riakͷӡ༻ͷૢ࡞ 1/2
    •  ϊʔυΛىಈ

    •  ϊʔυΛΫϥελʹ௥Ճ

    •  riak  start  
    •  riak-­‐admin  cluster  join  \  
    [email protected]  
    •  riak-­‐admin  cluster  plan  
    •  riak-­‐admin  cluster  commit  
    •  σʔλ഑ஔΛܭࢉ

    •  ΞΫηεΛྲྀ࢝͠ΊΔ

    View Slide

  31. Riakͷӡ༻ͷૢ࡞ 2/2
    •  ϊʔυΛΫϥελ͔Β֎͢

    •  σʔλ഑ஔΛ࠶ܭࢉ

    •  σʔλΛ࠶഑ஔ

    •  riak-­‐admin  down  \  
    [email protected]  
    •  riak-­‐admin  cluster  plan  
    •  riak-­‐admin  cluster  commit  

    View Slide

  32. ͓͢͢Ίπʔϧ
    •  AnsibleͷެࣜPlaybookϥΠϒϥϦʹ͋Γ·͢

    •  ߏ੒ྫ΋͋Γ·͢

    •  https://github.com/ansible/ansible  
    •  https://github.com/ansible/ansible-­‐examples/
    tree/master/riak  

    View Slide

  33. ͜Μͳӡ༻͕૊Ί·͢
    ৸Δ

    ໷தʹෛՙ͕
    ্͕Δ

    APIͰϊʔυ
    ࣗಈ௥Ճ

    ໷தʹຊ౰ʹඞ
    ཁͳ਺·Ͱϊʔ
    υΛݮΒ͢

    ى͖Δ

    ʢே͕͖ͨͷͰʣ

    ࡢ໷ͷ࠷େෛՙ
    ΛΈͯɺϊʔυ୆
    ਺ͱൺֱ

    View Slide

  34. খ࢝͘͞ΊΔͳΒ
    ˡWebServers

    •  m1.xlarge x3

    •  Elastic Block Storage
    256GB x3, >2000PIOPS

    •  ηΩϡϦςΟάϧʔϓ

    •  TCP 4369, 6000-7999, 8087,
    8098, 8099

    View Slide

  35. ;ͭ͏ʹ࢝ΊΔͳΒ
    ˡWebServers

    •  m1.xlarge x5

    •  Elastic Block Storage
    256GB x5, >2000PIOPS

    •  ηΩϡϦςΟάϧʔϓ

    •  TCP 4369, 6000-7999, 8087,
    8098, 8099

    View Slide

  36. ˓˓͕૿͖͑ͯͨΒ
    •  ୹ظతʢ1ʙ10िؒʣରࡦ

    •  ϚγϯΛ૿΍͢

    •  ௕ظతରࡦ

    •  ϚγϯεϖοΫΛ্͛Δ

    View Slide

  37. ෛՙ͕૿͖͑ͯͨΒ
    ˡWebServers

    •  m3.2xlarge x5

    •  Elastic Block Storage
    512GB x5, >4000PIOPS

    •  ηΩϡϦςΟάϧʔϓ

    •  TCP 4369, 6000-7999, 8087,
    8098, 8099

    View Slide

  38. σʔλ͕૿͖͑ͯͨΒ
    ˡWebServers

    •  m1.xlarge x5

    •  Elastic Block Storage
    2TB x5, >2000PIOPS

    •  ηΩϡϦςΟάϧʔϓ

    •  TCP 4369, 6000-7999, 8087,
    8098, 8099

    View Slide

  39. ෛՙ͕૿͖͑ͯͨΒ̎
    ˡWebServers

    •  hi1.4xlarge x5

    •  SSD

    •  ηΩϡϦςΟάϧʔϓ

    •  TCP 4369, 6000-7999, 8087,
    8098, 8099

    File

    Archive

    View Slide

  40. όοΫΞοϓઓུ
    •  tarball ΛS3ʹ

    •  LVM Snapshot

    •  ͍Ζ͍Ζ͋Γ·͕͢…

    •  MDC Replication

    View Slide

  41. Availability Zone Availability Zone
    ;ͨͭͷΫϥελΛϨϓϦέʔγϣϯͰಉظ͢Δ͜ͱ͕Ͱ͖Δ

    όοΫΞοϓ΍ෛՙ෼ࢄͳͲʹ

    View Slide

  42. RIAK + EC2 USECASES (ࣄྫ)ɹলུ͞Ε·ͨ͠

    View Slide

  43. ·ͱΊ
    •  Riak ͱ EC2 ͷ૊Έ߹Θͤ͸ͱͯ΋Α͍

    •  ؆୯ʹ࢖͑Δˍ҆ఆ͍ͯ͠Δ

    •  ׂͱଟ͘ͷϢʔβʔ͕EC2্ͰRiakΛ࢖͍ͬͯΔ

    એ఻

    ɾιϑτ΢ΣΞσβΠϯͱ͍͏ࡶࢽʹ࿈ࡌ͍ͯ͠·͢

    ɾ11݄߸ʢ10/18ൃച͸ͪΐ͏ͲRiak on EC2ͷࡉ࿦͕ʣ

    View Slide

  44. One more thing…
    •  ͍Ζ͍Ζݴ͍·͕ͨ͠ɺࣗ෼ͰؤுΔΑΓ΋100ഒָ
    ʹAWS্Ͱεέʔϧ͢ΔDBΛ࡞Δํ๏͕͋Γ·͢

    •  ϙνϙν͢Δ͚ͩͰOK

    View Slide

  45. Go to www.engineyard.com

    View Slide

  46. ࣭໰ʁ
    •  Repo: https://github.com/basho/riak

    •  Twitterެࣜ: @BashoJapan or #riakjp

    •  Riak ML: [email protected]

    •  Me: [email protected]

    View Slide