Upgrade to Pro — share decks privately, control downloads, hide ads and more …

分散ユニークID採番機 katsubushi と Web アプリケーションへの応用例 / katsubushi

分散ユニークID採番機 katsubushi と Web アプリケーションへの応用例 / katsubushi

YAPC::Fukuoka

Ca6281fff64797dc419b78f51f25c0a5?s=128

FUJIWARA Shunichiro
PRO

July 01, 2017
Tweet

Transcript

  1. ෼ࢄϢχʔΫID࠾൪ػ katsubushi ͱ Web ΞϓϦέʔγϣϯ ΁ͷԠ༻ྫ 2017.07.01 YAPC::Fukuoka @fujiwara

  2. @fujiwara ٕज़෦ github.com/fujiwara sfujiwara.hatenablog.com

  3. Game & Community

  4. katsubushi - ෼ࢄϢχʔΫID࠾൪ػ != katsuobushi != katsubushi

  5. katsubushi - ෼ࢄϢχʔΫID࠾൪ػ ෼ࢄ = ୯Ұͷ (host | node |

    process) ʹґଘ͠ͳ͍ ϢχʔΫ = ෳ਺ͷ katsubushi Ͱൃߦͨ͠ ID ͕ॏෳ͠ͳ͍ Lobi Ͱͷ DB γϟʔσΟϯάͷཁ੥ʹ൐ͬͯ஀ੜ
  6. Lobi Ͱͷ DB γϟʔσΟϯά MySQL Ͱී௨ͷ ୯Ұ master ෳ਺ slave

    ߏ੒ (ʙ2014) σʔλɺΞΫηε૿ՃʹΑΓ୯Ұ master ͕ݶքʹ (Ұ࣌తʹ i2.8xlarge Πϯελϯεʹͯ࣌ؒ͠Ք͗… !!!) ➡ ΍ΉΛಘͣɺDB Λ෼ׂ͢Δ͜ͱʹ (2015.02) ڞ௨DB(ϢʔβͳͲ) x 1 άϧʔϓνϟοτؔ࿈DB x 2 ←ඞཁʹͳΕ͹૿΍ͤΔ
  7. DB γϟʔσΟϯά࣌ͷ՝୊ ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑ ID࠾൪ σʔλҠߦɺ੾Γସ͑ (ຊτʔΫͰ͸লུ)

  8. DB γϟʔσΟϯά࣌ͷ՝୊ ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑

  9. ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑ άϧʔϓ͕Ͳͷγϟʔυʹଐ͢Δ͔ͷmapΛڞ௨DBʹ࣋ͭ ! γϟʔυҠಈ͠ͳ͚Ε͹ΩϟογϡՄೳ ΫΤϦʹؚ·ΕΔςʔϒϧͱΧϥϜͷ஋ΛΈͯ… ! γϟʔυΛಛఆͰ͖Ε͹1γϟʔυʹ౤͛Δ ! Ͱ͖ͳ͚Ε͹ෳ਺γϟʔυʹ౤͛ͯ݁ՌΛ·ͱΊΔ Teng

    ʹຐվ଄Λࢪͨ͠ Teng::Sharding Λ࡞੒(ඇެ։…)
  10. DB γϟʔσΟϯά࣌ͷ՝୊ ID ࠾൪

  11. ID ͷཁ݅ ֤γϟʔυͰॏෳ͠ͳ͍ϢχʔΫͳIDΛৼΓ͍ͨʂ 64bitҎԼͷ੔਺஋ ! طʹ਺஋ͱͯ͠ѻ͏લఏͷίʔυ͕ࢁ΄Ͳ͋Δ ! DB ʹೖΕΔʹ͸ 64

    bit ҎԼ͕Α͍ ࣌ܥྻͷঢॱʹͳΔ͜ͱ (࣮༻্໰୊ͳ͍ਫ਼౓Ͱ) ! ࣌ܥྻͰιʔτ͢ΔͨΊʹ ORDER BY id (DESC) ͍ͯ͠Δ
  12. ͭ·Γ

  13. UUID ͸࠾༻Ͱ͖ͳ͍ !

  14. ʮFacebook, Twitter, Instagram౳͕Ͳ͏΍ͬͯIDΛੜ੒͍ͯ͠ Δͷ͔ ·ͱΊ1ʯΛࢀߟʹݕ౼ 1 http://qiita.com/daisy1754/items/98a6e6b17d8161eab081

  15. Twitter Snowflake github.com/twitter/snowflake λΠϜελϯϓΛݩʹੜ੒ͨ͠ 64bit ੔਺ Instagram Snowflake like ͳ

    64bit ID Λ PL/PGSQL Ͱ Flickr MySQL ͷ Auto Increment Λ2୆༻ҙ ͦΕͧΕح਺ͱۮ਺୲౰Ͱ +2 ͍ͯ͘͠
  16. None
  17. ݁࿦ = Snowflake ํࣜ +------------------------------------------------------+ | 64 bit | +--+-----------------+----------------+----------------+

    | 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit | +--+-----------------+----------------+----------------+ Timestamp : Epoch ͔Βͷܦա࣌ؒ(msec) WorkerID : Ϋϥελ಺ͰҰҙͷ ID Sequence : ಉҰ Timestamp ಺ͷ࿈൪
  18. ݁࿦ = Snowflake ํࣜ + ಠࣗσʔϞϯ ಠࣗσʔϞϯ ! Go Ͱ࣮૷͠ɺ1ϗετʹ1ϓϩηεىಈ͢Δ

    ! ଟݴޠɺϛυϧ΢ΣΞ͔Β࠾൪Մೳʹ ௨৴ϓϩτίϧ͸ memcached text protocol ! ܰྔɾ࣮૷͕؆୯ (GET ͔͠͠ͳ͍ͷͰ) ! ֤छݴޠͰ࢖͑Δ ! PerlͷΫϥΠΞϯτ͕଎͍ (Cache::Memcached::Fast)
  19. katsubushi ஀ੜ github.com/kayac/go-katsubushi ໊લͷ༝དྷ Lobiͱ͍͑͹͵͜͞Μ ➡ Snowflake (ઇͷ݁থɺࡉ͔͍) ➡ ͵͜

    Ͷ͜ ͔͚Β ࡉ͔͍ ➡ ׍અʂ
  20. katsubushi ͷಛੑͱੑೳ

  21. ID ͷߏ଄ 329508644217425920 (10ਐදݱ) +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID

    | Sequence | +-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z bitͷ഑෼͸ Snowflake ͱಉ༷ Epoch 2015-01-01T00:00:00Z (Snowflakeͱ͸ҟͳΔ) ࣌ܥྻͰঢॱɺID ͔Βൃߦ࣌ࠁͷٯࢉ͕Մೳ (msecਫ਼౓)
  22. ID ͸͍ͭ·Ͱ࢖͑Δͷ͔ Timestamp 41bit 2015-01-01T00:00:00 UTC + 0b1{41} msec =>

    2084-09-06T15:47:35.551 UTC ໿69೥ؒ
  23. 1ඵؒʹൃߦͰ͖ΔID਺ 1ms ͋ͨΓ 12bit (4096) ͷ Sequence Λ΋ͭ = 4,096,000/sec

    (ཧ࿦஋) WorkerID 10bit (1024) ͳͷͰΫϥελશମͰ͸ 4,194,304,000 (≒32bit) ຖඵ໿32bitਐΉ
  24. ࣮ࡍͷੑೳ $ go test -bench Generate BenchmarkGenerateID-4 5000000 244 ns/op

    1ID ੜ੒ʹ 244ns 244 ns x 4,096,000 ≒ 1 sec Go ಺෦Ͱͷൃ൪͸ཧ࿦஋ݶք·ͰՄೳ (Sequence ͕ 4095 Λ௒͑Δͱ࣍ͷ Timestamp ·Ͱ sleep)
  25. ωοτϫʔΫαʔόʔͱͯ͠ͷੑೳ Client: Go katsubushi.Client on AWS c4.8xlarge (36 cores) Server:

    katsubushi v1.3.0 on AWS c3.2xlarge (8 cores) ಉҰAZͷผϗετͰܭଌ ฒྻ౓ fetch=1 fetch=10 fetch=100 1 5,567 52,003 258,077 10 52,936 434,977 1,190,018 100 151,188 826,200 1,164,833 Ұ౓ʹ 100 ID Λൃߦ͢Ε͹ 100ສ/sec Ҏ্
  26. None
  27. katsubushi ͷ࢖͍͔ͨ

  28. Install Docker $ docker pull katsubushi/katsubushi $ docker run -p

    11212:11212 katsubushi/katsubushi Release binary github.com/kayac/go-katsubushi/releases
  29. katsubushi ͷ࢖͍͔ͨ worker-id Λࢦఆͯ͠ىಈ͢Δͱ TCP 11212 Λ Listen $ katsubushi

    -worker-id 1 2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:152 Listening at [::]:11212 2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:153 Worker ID = 1 Docker ͷ৔߹͸؀ڥม਺Ͱ $ docker run -p 11212:11212 -e worker_id=2 katsubushi/katsubushi 2017-06-29T05:58:39.847Z INFO go-katsubushi/app.go:152 Listening at [::]:11212 2017-06-29T05:58:39.848Z INFO go-katsubushi/app.go:153 Worker ID = 2
  30. katsubushi ͷ࢖͍͔ͨ nc ίϚϯυͰखͰ஻ͬͯΈ·͠ΐ͏ GET ͷ key ͸೚ҙͷ஋Ͱ OK $

    nc localhost 11212 GET id VALUE id 0 18 329508644217425920 END
  31. katsubushi ͷ࢖͍͔ͨ ෳ਺IDΛ·ͱΊͯऔಘ΋Ͱ͖·͢ GET id1 id2 VALUE id1 0 18

    329508654694797312 VALUE id2 0 18 329508654694797313 END
  32. katsubushi ͷ࢖͍͔ͨ ී௨ͷ memcached client ͕࢖͑·͢ (ςΩετϓϩτίϧͷΈ) use Cache::Memcached; $c

    = Cache::Memcached->new( servers => ["127.0.0.1:11212"] ); $c->get("id"); #= 329521618642538496 Perl Cache::Memcached, C::M::Fast, C::M::AnyEvent Go github.com/bradfitz/gomemcache Ruby memcached(gem)
  33. katsubushi ؔ࿈ϥΠϒϥϦ

  34. Perl - ΫϥΠΞϯτϥΠϒϥϦ metacpan.org/pod/Katsubushi::Client use Katubushi::Client; my $client = Katsubushi::Client->new({

    servers => ["127.0.0.1:11212", "10.8.0.1:11212"], }); my $id = $client->fetch; my @ids = $client->fetch_multi(3); serversʹࢦఆͨ͠ϗετͷઌ಄͔Βऔಘ ࣦഊͨ͠Β࣍ͷϗετ΁
  35. Perl - ID ͱ UNIX time ͷ૬ޓม׵ use Katsubushi::Converter #

    ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ my $now = time(); my $id = Katsubushi::Converter::epoch_to_id($now); # ID͕ൃߦ͞Εͨ࣌ࠁΛunix timeͰऔಘ my $t = Katsubushi::Converter::id_to_epoch($id); katsubushi2epoch, epoch2katsubushi ίϚϯυ΋ೖΔ
  36. Go - ΫϥΠΞϯτϥΠϒϥϦ godoc.org/github.com/kayac/go-katsubushi#Client import "github.com/kayac/go-katsubushi" func main() { client

    := katsubushi.NewClient("127.0.0.1:11212", "10.8.0.1:11212") id, _ := client.Fetch() // uint64, error ids, _ := client.FetchMulti(3) // []uint64, error } ෳ਺ϗετͷࢼߦॱ͸ Perl ൛ͱಉ͡
  37. Go - ID ͱ time.Time ͷ૬ޓม׵ godoc.org/github.com/kayac/go-katsubushi#ToID // ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ

    t1 := time.Now() id := katsubushi.ToID(t1) // ID͕ൃߦ͞Εͨ࣌ࠁΛ time.Time Ͱฦ͢ t2 := katsubushi.ToTime(id)
  38. fluent-plugin-katsubushi rubygems.org/gems/fluent-plugin-katsubushi Fluentd ͷϑΟϧλʔϓϥάΠϯ ϑΟϧλʔΛ௨ա͢Δϩάʹ katsubushi Ͱൃߦͨ͠ ID Λ஫ೖ <filter>

    @type katsubushi id_key id # ஫ೖ͢Δkey໊ host localhost # katsubushi ͷϗετ໊ port 11212 # katsubushi ͷϙʔτ൪߸ </filter>
  39. fluent-plugin-katsubushi fluent-plugin-katsubushi Λ௨ա͢Δͱ ID ͕஫ೖ͞ΕΔ {"message":"foo"} ! {"message":"foo","id":329518490450919424} ϩάʹ unique

    ͳ ID ΛৼΓ͍ͨʂ ͚ͲൃߦՕॴͰ͸ৼΕͳ͍ʂͱ͍͏৔߹ʹ
  40. katsubushi ͷӡ༻

  41. worker-id ΛҰҙʹ͢Δ ӡ༻͍ͯ͠Δશͯͷ katsubushi process ͷ worker-id ΛҰҙʹ͢Δͷ͕ඞਢ worker-id ॏෳ

    ➡ ಉҰ࣌ࠁʹൃߦ͞ΕͨID͕ॏෳ͢Δ ➡ ഁ໓ +------------------------------------------------------+ | 64 bit | +--+-----------------+----------------+----------------+ | 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit | +--+-----------------+----------------+----------------+
  42. worker-id ΛҰҙʹ͢Δ worker-id ͸ 1ʙ1023 ͷ੔਺஋ ݻఆαʔόͰಈ͔͢ͳΒద੾ʹखͰ͚ͭΔ ಈతʹαʔό͕ىಈ͢Δ؀ڥͰ͸…ʁ ͨͱ͑͹ IP

    ΞυϨεͷୈ 3,4 octet Λ࢖͏ 192.168.0.10 ➡ 0 * 256 + 10 = 10 192.168.1.20 ➡ 1 * 256 + 20 = 266 192.168.2.30 ➡ 2 * 256 + 30 = 542
  43. worker-id ΛҰҙʹ͢Δ ίϯςφ؀ڥͰ͸…ʁ Redis Λ࢖ͬͨ worker-id ࣗಈׂΓ౰ͯػೳ (ޙड़)

  44. stats memcached ͷ stats ίϚϯυΛ࣮૷ $ echo "STATS" | nc

    localhost 11212 STAT pid 1 STAT uptime 51014 STAT time 1498716107 STAT version 1.3.0 STAT curr_connections 1 STAT total_connections 8 STAT cmd_get 18 STAT get_hits 20 STAT get_misses 0 END
  45. key ஋ͷྫ ҙຯ pid 1 Process ID uptime 51014 ىಈ͔ͯ͠Βͷܦաඵ਺

    time 1498716107 ݱࡏ࣌ࠁ UNIX time version 1.3.0 όʔδϣϯ curr_connections 1 ݱࡏ઀ଓ͍ͯ͠ΔTCPίωΫγϣϯ ਺ total_connections 8 ىಈ͔ͯ͠Β઀ଓͨ͠ίωΫγϣϯ ਺ (ྦྷܭ) cmd_get 18 GET ίϚϯυΛॲཧͨ͠ճ਺ (ྦྷܭ) get_hits 20 ൃߦͨ͠ ID ਺ (ྦྷܭ) get_misses 0 ൃߦͰ͖ͳ͔ͬͨ ID ਺ (ྦྷܭ)
  46. ؂ࢹ͢΂͖஋ key curr_connections ଟ͗͢ͳ͍͔ uptime ר͖໭͍ͬͯΔ = ࠶ىಈͯ͠ ͍Δ get_misses

    ൃੜ = ࣌ࠁ͕ר͖໭͍ͬͯΔ
  47. ࣌ࠁר͖໭Γ࣌ͷରԠ katsubushi ID ͷҰҙੑ͸࣌ࠁʹґଘ ࣌ࠁ͕ਐΉͷ͸໰୊ͳ͍͕ɺר͖໭Δͱ…(ͨͱ͑͹Ӟඵ) աڈʹൃߦͨ͠ ID ͱಉ͡ ID Λൃߦͯ͠͠·͏Մೳੑ

    ➡ ίʔυ಺Ͱ͸ר͖໭Βͳ͍Α͏ʹอޢ
  48. ҎલͷλΠϜελϯϓ͕དྷͨΒΤϥʔ func (g *Generator) NextID() (uint64, error) { g.lock.Lock() defer

    g.lock.Unlock() ts := g.timestamp() // for rewind of server clock if ts < g.lastTimestamp { return 0, errors.New("system clock was rollbacked") }
  49. Go 1.9 ͷ monotonic time Λ࢖͏(༧ఆ) Go 1.9 Ͱ monotonic

    time ͕ಋೖ͞ΕΔ 3 ىಈ࣌ʹ time.Now() ͨ࣌͠ࠁ͔Βͷࠩ෼Λར༻͢Ε͹ OS࣌ࠁ͕ר͖໭ͬͯ΋ӨڹΛड͚ͳ͍ 3 https://tip.golang.org/doc/go1.9#monotonic-time
  50. ఀࢭ͍ͯ͠Δؒʹ࣌ࠁ͕໭ͬͨΒ… katsubushi ϓϩηε಺Ͱ͸ݕ஌͠Α͏͕ͳ͍ ରԠࡦɿࣗಈͰ࠶ىಈ͠ͳ͍ Perl / Go ͷΫϥΠΞϯτ͸ଞͷϗετʹ fallback Ͱ͖Δ

    ઃఆͯ͋͠Δશͯͷϗετ͕ಉ࣌ʹམͪͳ͚Ε͹ ID औಘͰ͖Δ
  51. ҆ఆੑ 2015೥ʹϦϦʔε͔ͯ͠Β2೥ؒ Lobiͷຊ൪؀ڥͷ΄΅શͯͷϗετͰՔಇ͍ͯ͠Δ͕ ௨ৗར༻Ͱϓϩηε͕མͪͨ͜ͱ͸ͳ͍ 2 ϝϞϦϦʔΫ΋ܦݧͳ͠ 2 ͨͩ͠ memcached protocolͰਖ਼͘͠ΞΫηεͨ͠৔߹ɻෆਖ਼ͳίϚϯυͰΫϥογϡͨ͜͠ͱ͸͋ͬͨ

    github.com/kayac/ go-katsubushi/pull/8
  52. WebΞϓϦέʔγϣϯ΁ͷԠ༻

  53. Request ID ʹΑΔ ΞϓϦέʔγϣϯ௥੻

  54. Request ID Λ katsubushi Ͱൃߦ nginx͕ϦΫΤετΛड͚෇͚ͨ࣌఺Ͱ katsubushi ID Λൃߦ ޙॲཧʹҾ͖౉͢͜ͱͰ֤छॲཧͷ௥੻Λ༰қʹ

    nginx ͱ katsubushi ͕ memcached protocol Ͱ௨৴
  55. IDΛҾ͖ճͯ͠ ϩάʹه࿥

  56. katsubushi ➡ nginx ΞΫηεຖʹ nginx lua ͕ ID Λऔಘͯ͠ม਺ $req_id

    ʹઃఆ upstream katsubushi { server 127.0.0.1:11212; keepalive 10; } location /katsubushi { internal; set $memcached_key "id"; # key͸ͳΜͰ΋OK memcached_pass katsubushi; } set $req_id 0; access_by_lua_block { local res = ngx.location.capture("/katsubushi") if res then ngx.var.req_id = res.body end }
  57. nginx Ͱϩάʹग़ྗ log_format ltsv 'req_id:$req_id\t' 'host:$remote_addr\t' 'user:$remote_user\t' 'time:$time_iso8601\t' 'method:$request_method\t' ...

  58. nginx ➡ App(Plack) ϦΫΤετϔομ X-Request-ID Λઃఆ upstream app { server

    unix:/var/tmp/app.socket; } location / { proxy_set_header X-Request-ID $req_id; proxy_pass http://app; }
  59. App(Plack) ➡ local %ENV Plack::Middleware::SetLocalEnv Plack ͷ $env ͔Β local

    %ENV ʹ஋Λઃఆ͢Δ ΞϓϦέʔγϣϯͷͲ͔͜ΒͰ΋ $ENV{REQUEST_ID} ͰࢀরՄ # app.psgi use Plack::Builder; builder { enable "SetLocalEnv", REQUEST_ID => "HTTP_X_REQUEST_ID", ; # ... $app; }
  60. Plack::Middleware::SetLocalEnv ࣮૷͸͜Ε͚ͩ local ͳͷͰείʔϓΛൈ͚Δͱ(ϦΫΤετॲཧ͕ऴΘΔͱ) ஋͕ݩʹ໭Δ sub call { my ($self,

    $env) = @_; local %ENV = %ENV; for my $key (keys %$self) { $ENV{$key} = $env->{ $self->{$key} } if exists $env->{$self->{$key}}; } $self->app->($env); }
  61. App(Plack) ➡ app.log (Log::Minimal) $Log::Minimal::(PRINT|DIE) Ͱग़ྗΛΧελϚΠζ use Log::Minimal; BEGIN {

    $Log::Minimal::PRINT = sub { my ($time, $type, $message, $trace) = @_; my $req_id = $ENV{REQUEST_ID} // 0; warn "[$$][$req_id][$type] $message at $trace\n"; }; } warnf("something bad!"); [64302][329733861698727936][WARN] something bad! at .../App.pm line XX
  62. App(Plack) ➡ Fluentd ϑΝΠϧΛհͣ͞௚઀ Fluent::Logger Ͱૹ৴͢Δ৔߹ ϝοηʔδʹ req_id Λ஫ೖ package

    Logger; use Fluent::Logger; sub log { my ($self, $tag, $msg) = @_; $msg->{req_id} = $ENV{REQUEST_ID} // 0; $logger->post($tag, $msg); # $logger is Fluent::Logger }
  63. App(Plack) ➡ MySQL ΫΤϦͷઌ಄ʹSQLίϝϯτͰ /* req_id=... */ ΛຒΊࠐΉ use parent

    'Teng::Sharding'; around ["execute", "do"] => sub { my ($orig, $self, $sql, @rest) = @_; my $req_id = $ENV{REQUEST_ID} // 0; return $orig->($self, "/* req_id=${req_id} */\n${sql}", @rest); }; # Time: 170629 3:27:47 # User@Host: xxx[xxx] @ [192.0.2.1] # Query_time: 1.080420 Lock_time: 0.000040 Rows_sent: 5 Rows_examined: 909850 SET timestamp=1498674467; /* req_id=329689349781536768 */ SELECT ..... FROM ...
  64. ͍ΖΜͳى఺Ͱௐ͕ࠪͰ͖Δ ΞΫηεϩάͰ status:500 ➡ ΞϓϦͷΤϥʔϩάͱর߹ slow.log ʹࠔͬͨΫΤϦ͕ه࿥͞Ε͍ͯΔ ඇಉظ job ͕ࣦഊ

    ➡ ൃੜݩʹͳͬͨHTTPϦΫΤετΛΞΫηεϩάͰಛఆ
  65. ଞܗࣜͷ Request ID ͱͷൺֱ $request_id (nginx >= 1.11.0) unique request

    identifier generated from 16 random bytes, in hexadecimal 4 ྫ: 9f53d64cf47dd966a7462f5a737b5ade ௕͍ ࣌ࠁ৘ใؚ͕·Εͳ͍ 4 http://nginx.org/en/docs/http/ngxhttpcoremodule.html#varrequest_id
  66. ೔࣌ͰύʔςΟγϣχϯάΛߦ͏ ςʔϒϧͷϓϥΠϚϦΩʔͱͯ͠࢖͏

  67. ҰఆظؒͰݹ͍σʔλΛফ͍ͨ͠ςʔϒϧ(௨஌) Redis ʹ id ͷΈอ࣋ͯ͠ index ʹ͍ͯ͠Δ id auto_increment, created_date

    ͷෳ߹Primary Key CREATE TABLE `notification` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `created_date` datetime NOT NULL, /* other columns */ PRIMARY KEY (`id`,`created_date`) ) ENGINE=InnoDB AUTO_INCREMENT=xxxxx DEFAULT CHARSET=utf8 PARTITION BY RANGE COLUMNS(created_date) (PARTITION p20170602 VALUES LESS THAN ('2017-06-02') ENGINE = InnoDB, PARTITION p20170603 VALUES LESS THAN ('2017-06-03') ENGINE = InnoDB, PARTITION p20170604 VALUES LESS THAN ('2017-06-04') ENGINE = InnoDB, .....
  68. id + datetime ύʔςΟγϣχϯάͷ໰୊ idͷΈͰݕࡧ͢ΔΫΤϦͰ͸מΓࠐΊͳ͍ͨΊ શύʔςΟγϣϯΛݕࡧ͢Δ (30೔෼࢒ͨ͢Ίʹ30 partition੾Δͱ30ഒͷݕࡧίετ) ➡ Α͘΍Δ

    WHERE id IN (?,?,....) ͕஗͍ ➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍͸ͣͷύʔςΟγϣϯʹ සൟʹࢀর͕૸Δ (InnoDB Buffer Poolͷແବ)
  69. ! katsubushi ID Ͱ RANGE ύʔςΟγϣϯΛ੾Ε͹…

  70. katsubushi ID ͷΈͰ1೔୯ҐͷύʔςΟγϣχϯά CREATE TABLE `notification` ( `id` bigint(20) unsigned

    NOT NULL, /* AUTO_INCREMENT ͠ͳ͍ */ `created_date` datetime NOT NULL, /* ෆཁ͔΋͠Εͳ͍ */ /* other columns */ PRIMARY KEY (`id`) /* pkey ͸ id ͷΈ */ ) ENGINE=InnoDB DEFAULT CHARSET=utf8 PARTITION BY RANGE COLUMNS(id) (PARTITION p20170602 VALUES LESS THAN (319852589875200000) ENGINE = InnoDB, /* ^ 2017-06-02 00:00:00 JST */ PARTITION p20170603 VALUES LESS THAN (320214977740800000) ENGINE = InnoDB, /* ^ 2017-06-03 00:00:00 JST */ PARTITION p20170604 VALUES LESS THAN (320577365606400000) ENGINE = InnoDB, /* ^ 2017-06-04 00:00:00 JST */ .....
  71. katsubushi ID only ύʔςΟγϣχϯάͷར఺ id ͷΈͷݕࡧΫΤϦ͕ࣗಈతʹמΓࠐ·ΕΔ ➡ Α͘΍Δ WHERE id

    IN (?,?,....) ͕࠷దԽ͞ΕΔ ➡ Row Reads ܹݮ ➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍ύʔςΟγϣϯʹ͸ ࢀর͕૸Βͳ͍ (InnoDB Buffer Poolͷޮ཰Up)
  72. katsubushi ID only ύʔςΟγϣχϯάͷܽ఺? Ͳͷ஋Λڥʹ੾Ε͹͍͍ͷ͔͕௚ײతͰ͸ͳ͍ ! Katsubushi::Converter Ͱܭࢉ ಛఆͷ೔͚ͩࢀর͍ͨ͠ !

    ೔ͷൣғΛ ID ʹม׵ͯ͠ WHERE ۟ʹೖΕΕ͹OK
  73. ίϯςφ؀ڥ΁ͷదԠ

  74. ίϯςφ؀ڥͰ͸ worker-id ΛҰҙʹ ͢Δͷ͕೉͍͠!

  75. ίϯςφ؀ڥͰ worker-id ΛҰҙʹ͢Δ ❌ ਓྗͰͷݻఆׂΓ౰ͯ͸Ͱ͖ͳ͍ ੜ·Εͯ͸ফ͑Δͷ͕ίϯςφ ❌ IPΞυϨεΛ࢖͏͜ͱ΋Ͱ͖ͳ͍ ϗετͱNAT͍ͯ͠ΔͱɺผϗετͰಉҰͷ IP

    ΞυϨεΛ ࢖͍ͬͯΔίϯςφ͕ଘࡏ͠͏Δ Ͳ͏͢Ε͹…
  76. Redis Ͱഉଞతʹ஋Λऔಘͤ͞Δ ͙͢ࢥ͍ͭ͘ͷ͸ू߹ܕͰ SPOP > SADD id_pool 1 2 3

    4 # 4ཁૉͷू߹Λ࡞Δ > SPOP id_pool # ू߹͔ΒϥϯμϜʹҰͭऔΓग़͢ "3" > SMEMBERS id_pool # ࢒Γ "1" "2" "4" Simple is best?
  77. SPOP ํࣜͷ໰୊ 1. ϓϩηεىಈ࣌ʹ SPOP Ͱऔಘ 2. ϓϩηεऴྃ࣌ʹ SADD Ͱฦ٫

    औΓग़͞Εͨ஋͕ฦ٫͞ΕΔ͜ͱ͕อূͰ͖ͳ͍ ϓϩηε͸ਖ਼ৗऴྃ͢Δͱ͸ݶΒͳ͍ ➡ ௕ظؒӡ༻͍ͯ͠Δͱฦ٫͞Εͳ͍஋͕ ཷ·͍ͬͯͬͯރׇ͢ΔڪΕ
  78. ϝϯςφϯεϑϦʔͳํ๏Λߟ͍͑ͨ

  79. raus

  80. raus github.com/fujiwara/raus "RAnged Unique id Supplier" RedisΛ࢖ͬͯҰఆൣғͷ੔਺IDΛ ॏෳͳׂ͘Γ౰ͯΔͨΊͷϥΠϒϥϦ

  81. PubSub + SET NX + GETSET + EXPIRE 1. ੜ͖͍ͯΔϓϩηε͕

    PubSub channel ʹอ͍࣋ͯ͠ΔIDΛ ྲྀ͢ 2. อ͍࣋ͯ͠ΔIDͷ Key Λ GETSET + EXPIRE Ͱߋ৽͠ଓ͚Δ 3. ৽نϓϩηε͸ PubSub channel Λ؍࡯ͯ͠ɺ࢖ΘΕ͍ͯͳ ͦ͏ͳIDΛ SET NX Ͱऔಘ 4. औಘͰ͖ͨΒ PubSub + GETSET + EXPIRE Λଓ͚Δ
  82. 1. IDΛอ͍࣋ͯ͠Δطଘ raus ͸ PUBLISH + GETSET Λఆظతʹߦ͏

  83. 2. ৽ن raus ͸ SUBSCRIBE ͯ͠ଞͷ raus ͕࢖͍ͬͯΔIDΛ೺Ѳ

  84. 3. ৽ن raus ͸ଞͷ raus ͕࢖ͬͯͳ͍(͸ͣͷ) key Λ SET NX

    Ͱഉଞऔಘ
  85. 4. ID ΛऔಘͰ͖ͨΒ PUBLISH + GETSET Λఆظ࣮ߦ

  86. raus ͕ϝϯςφϯεϑϦʔͳཧ༝

  87. raus ͕ϝϯςφϯεϑϦʔͳཧ༝ • ࢮΜͩϓϩηε͸ GETSET ͠ͳ͍ͷͰ key ͕࣌ݶ։์͞ΕΔ • ૄ௨Ͱ͖ͳ͘ͳͬͯ΋Ұఆ࣌ؒ͸

    key ͕อ࣋͞Ε͍ͯΔͷͰ ଞ͕औಘͰ͖ͳ͍ • ૄ௨͕ճ෮ͨ͠ޙʹGETSETʹࣦഊͨ͠ (=ଞͷϓϩηε͕։์͞Εͨ key Λऔಘͯ͠͠·ͬͨ)৔߹ ଈ࠲ʹࢮ͵
  88. katsubushi + raus $ katsubushi -redis redis://192.0.2.2:6379 Waiting for worker-id

    automated assignment (between 1 and 1023) with redis://192.0.2.2:6379 xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 <= ଞͷkatsubushi͕woker_id=2Λอ͍࣋ͯ͠Δ xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 ͷΛsubscribeͰ؍࡯͍ͯ͠Δ xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 candidate ids: [1 3 4 5 6 7 8 9 10 11] <= 2 Ҏ֎ͷΛީิʹͯͦ͠ͷத͔ΒϥϯμϜʹ trying to get lock key raus:id:8 <= SET NX got lock for 8 <= ੒ޭͨ͠ͷͰ 8 Λऔಘ Assigned worker-id: 8 Listening at [::]:11212 Worker ID = 8 <= PUBLISH + GETSET ΛཪͰଓ͚͍ͯΔ
  89. katsubushi + raus worker_id=8 Λऔಘͨ͠ katsubushi ͕ಈ͍͍ͯΔঢ়ଶͰ key Λ্ॻ͖ͯ͠ΈΔͱ… $

    redis-cli set raus:id:8 xxx unexpected uuid got: xxx panic: unexpected uuid got: xxx goroutine 42 [running]: main.assignWorkerID.func1(0xc4201462c0, 0xc42006f020) src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:230 +0xb0 created by main.assignWorkerID src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:235 +0x4fe
  90. ·ͱΊ

  91. ·ͱΊ • Snowflake ͱಉ༷ͷΞϧΰϦζϜͰ෼ࢄ؀ڥͰϢχʔΫͳID ΛൃߦͰ͖Δ katsubushi • ܰྔͳͷͰ DB ͷߦ

    ID Ҏ֎ʹ΋ར༻Մೳ ➡ Request ID Ͱॲཧ௥੻ • ࣌ࠁؚ͕·ΕΔಛੑΛੜ͔ͯ͠ύʔςΟγϣχϯάʹ΋ • ίϯςφ؀ڥʹରԠ͢Δ Redis Λ࢖ͬͨࣗಈΫϥελϦϯά