分散ユニークID採番機 katsubushi と Web アプリケーションへの応用例 / katsubushi

YAPC::Fukuoka

Transcript

1. ෼ࢄϢχʔΫID࠾൪ػ katsubushi ͱ Web ΞϓϦέʔγϣϯ ΁ͷԠ༻ྫ 2017.07.01 YAPC::Fukuoka @fujiwara
2. @fujiwara ٕज़෦ github.com/fujiwara sfujiwara.hatenablog.com
3. Game & Community
4. katsubushi - ෼ࢄϢχʔΫID࠾൪ػ != katsuobushi != katsubushi
5. katsubushi - ෼ࢄϢχʔΫID࠾൪ػ ෼ࢄ = ୯Ұͷ (host | node | process) ʹґଘ͠ͳ͍ ϢχʔΫ = ෳ਺ͷ katsubushi Ͱൃߦͨ͠ ID ͕ॏෳ͠ͳ͍ Lobi Ͱͷ DB γϟʔσΟϯάͷཁ੥ʹ൐ͬͯ஀ੜ
6. Lobi Ͱͷ DB γϟʔσΟϯά MySQL Ͱී௨ͷ ୯Ұ master ෳ਺ slave ߏ੒ (ʙ2014) σʔλɺΞΫηε૿ՃʹΑΓ୯Ұ master ͕ݶքʹ (Ұ࣌తʹ i2.8xlarge Πϯελϯεʹͯ࣌ؒ͠Ք͗… !!!) ➡ ΍ΉΛಘͣɺDB Λ෼ׂ͢Δ͜ͱʹ (2015.02) ڞ௨DB(ϢʔβͳͲ) x 1 άϧʔϓνϟοτؔ࿈DB x 2 ←ඞཁʹͳΕ͹૿΍ͤΔ
7. DB γϟʔσΟϯά࣌ͷ՝୊ ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑ ID࠾൪ σʔλҠߦɺ੾Γସ͑ (ຊτʔΫͰ͸লུ)
8. DB γϟʔσΟϯά࣌ͷ՝୊ ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑
9. ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑ άϧʔϓ͕Ͳͷγϟʔυʹଐ͢Δ͔ͷmapΛڞ௨DBʹ࣋ͭ ! γϟʔυҠಈ͠ͳ͚Ε͹ΩϟογϡՄೳ ΫΤϦʹؚ·ΕΔςʔϒϧͱΧϥϜͷ஋ΛΈͯ… ! γϟʔυΛಛఆͰ͖Ε͹1γϟʔυʹ౤͛Δ ! Ͱ͖ͳ͚Ε͹ෳ਺γϟʔυʹ౤͛ͯ݁ՌΛ·ͱΊΔ Teng ʹຐվ଄Λࢪͨ͠ Teng::Sharding Λ࡞੒(ඇެ։…)
10. DB γϟʔσΟϯά࣌ͷ՝୊ ID ࠾൪
11. ID ͷཁ݅ ֤γϟʔυͰॏෳ͠ͳ͍ϢχʔΫͳIDΛৼΓ͍ͨʂ 64bitҎԼͷ੔਺஋ ! طʹ਺஋ͱͯ͠ѻ͏લఏͷίʔυ͕ࢁ΄Ͳ͋Δ ! DB ʹೖΕΔʹ͸ 64 bit ҎԼ͕Α͍ ࣌ܥྻͷঢॱʹͳΔ͜ͱ (࣮༻্໰୊ͳ͍ਫ਼౓Ͱ) ! ࣌ܥྻͰιʔτ͢ΔͨΊʹ ORDER BY id (DESC) ͍ͯ͠Δ
12. ͭ·Γ
13. UUID ͸࠾༻Ͱ͖ͳ͍ !
14. ʮFacebook, Twitter, Instagram౳͕Ͳ͏΍ͬͯIDΛੜ੒͍ͯ͠ Δͷ͔ ·ͱΊ1ʯΛࢀߟʹݕ౼ 1 http://qiita.com/daisy1754/items/98a6e6b17d8161eab081
15. Twitter Snowflake github.com/twitter/snowflake λΠϜελϯϓΛݩʹੜ੒ͨ͠ 64bit ੔਺ Instagram Snowflake like ͳ 64bit ID Λ PL/PGSQL Ͱ Flickr MySQL ͷ Auto Increment Λ2୆༻ҙ ͦΕͧΕح਺ͱۮ਺୲౰Ͱ +2 ͍ͯ͘͠
17. ݁࿦ = Snowflake ํࣜ +------------------------------------------------------+ | 64 bit | +--+-----------------+----------------+----------------+ | 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit | +--+-----------------+----------------+----------------+ Timestamp : Epoch ͔Βͷܦա࣌ؒ(msec) WorkerID : Ϋϥελ಺ͰҰҙͷ ID Sequence : ಉҰ Timestamp ಺ͷ࿈൪
18. ݁࿦ = Snowflake ํࣜ + ಠࣗσʔϞϯ ಠࣗσʔϞϯ ! Go Ͱ࣮૷͠ɺ1ϗετʹ1ϓϩηεىಈ͢Δ ! ଟݴޠɺϛυϧ΢ΣΞ͔Β࠾൪Մೳʹ ௨৴ϓϩτίϧ͸ memcached text protocol ! ܰྔɾ࣮૷͕؆୯ (GET ͔͠͠ͳ͍ͷͰ) ! ֤छݴޠͰ࢖͑Δ ! PerlͷΫϥΠΞϯτ͕଎͍ (Cache::Memcached::Fast)
19. katsubushi ஀ੜ github.com/kayac/go-katsubushi ໊લͷ༝དྷ Lobiͱ͍͑͹͵͜͞Μ ➡ Snowflake (ઇͷ݁থɺࡉ͔͍) ➡ ͵͜ Ͷ͜ ͔͚Β ࡉ͔͍ ➡ ׍અʂ
20. katsubushi ͷಛੑͱੑೳ
21. ID ͷߏ଄ 329508644217425920 (10ਐදݱ) +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID | Sequence | +-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z bitͷ഑෼͸ Snowflake ͱಉ༷ Epoch 2015-01-01T00:00:00Z (Snowflakeͱ͸ҟͳΔ) ࣌ܥྻͰঢॱɺID ͔Βൃߦ࣌ࠁͷٯࢉ͕Մೳ (msecਫ਼౓)
22. ID ͸͍ͭ·Ͱ࢖͑Δͷ͔ Timestamp 41bit 2015-01-01T00:00:00 UTC + 0b1{41} msec => 2084-09-06T15:47:35.551 UTC ໿69೥ؒ
23. 1ඵؒʹൃߦͰ͖ΔID਺ 1ms ͋ͨΓ 12bit (4096) ͷ Sequence Λ΋ͭ = 4,096,000/sec (ཧ࿦஋) WorkerID 10bit (1024) ͳͷͰΫϥελશମͰ͸ 4,194,304,000 (≒32bit) ຖඵ໿32bitਐΉ
24. ࣮ࡍͷੑೳ $ go test -bench Generate BenchmarkGenerateID-4 5000000 244 ns/op 1ID ੜ੒ʹ 244ns 244 ns x 4,096,000 ≒ 1 sec Go ಺෦Ͱͷൃ൪͸ཧ࿦஋ݶք·ͰՄೳ (Sequence ͕ 4095 Λ௒͑Δͱ࣍ͷ Timestamp ·Ͱ sleep)
25. ωοτϫʔΫαʔόʔͱͯ͠ͷੑೳ Client: Go katsubushi.Client on AWS c4.8xlarge (36 cores) Server: katsubushi v1.3.0 on AWS c3.2xlarge (8 cores) ಉҰAZͷผϗετͰܭଌ ฒྻ౓ fetch=1 fetch=10 fetch=100 1 5,567 52,003 258,077 10 52,936 434,977 1,190,018 100 151,188 826,200 1,164,833 Ұ౓ʹ 100 ID Λൃߦ͢Ε͹ 100ສ/sec Ҏ্
27. katsubushi ͷ࢖͍͔ͨ
28. Install Docker $ docker pull katsubushi/katsubushi $ docker run -p 11212:11212 katsubushi/katsubushi Release binary github.com/kayac/go-katsubushi/releases
29. katsubushi ͷ࢖͍͔ͨ worker-id Λࢦఆͯ͠ىಈ͢Δͱ TCP 11212 Λ Listen $ katsubushi -worker-id 1 2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:152 Listening at [::]:11212 2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:153 Worker ID = 1 Docker ͷ৔߹͸؀ڥม਺Ͱ $ docker run -p 11212:11212 -e worker_id=2 katsubushi/katsubushi 2017-06-29T05:58:39.847Z INFO go-katsubushi/app.go:152 Listening at [::]:11212 2017-06-29T05:58:39.848Z INFO go-katsubushi/app.go:153 Worker ID = 2
30. katsubushi ͷ࢖͍͔ͨ nc ίϚϯυͰखͰ஻ͬͯΈ·͠ΐ͏ GET ͷ key ͸೚ҙͷ஋Ͱ OK $ nc localhost 11212 GET id VALUE id 0 18 329508644217425920 END
31. katsubushi ͷ࢖͍͔ͨ ෳ਺IDΛ·ͱΊͯऔಘ΋Ͱ͖·͢ GET id1 id2 VALUE id1 0 18 329508654694797312 VALUE id2 0 18 329508654694797313 END
32. katsubushi ͷ࢖͍͔ͨ ී௨ͷ memcached client ͕࢖͑·͢ (ςΩετϓϩτίϧͷΈ) use Cache::Memcached; $c = Cache::Memcached->new( servers => ["127.0.0.1:11212"] ); $c->get("id"); #= 329521618642538496 Perl Cache::Memcached, C::M::Fast, C::M::AnyEvent Go github.com/bradfitz/gomemcache Ruby memcached(gem)
33. katsubushi ؔ࿈ϥΠϒϥϦ
34. Perl - ΫϥΠΞϯτϥΠϒϥϦ metacpan.org/pod/Katsubushi::Client use Katubushi::Client; my $client = Katsubushi::Client->new({ servers => ["127.0.0.1:11212", "10.8.0.1:11212"], }); my $id = $client->fetch; my @ids = $client->fetch_multi(3); serversʹࢦఆͨ͠ϗετͷઌ಄͔Βऔಘ ࣦഊͨ͠Β࣍ͷϗετ΁
35. Perl - ID ͱ UNIX time ͷ૬ޓม׵ use Katsubushi::Converter # ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ my $now = time(); my $id = Katsubushi::Converter::epoch_to_id($now); # ID͕ൃߦ͞Εͨ࣌ࠁΛunix timeͰऔಘ my $t = Katsubushi::Converter::id_to_epoch($id); katsubushi2epoch, epoch2katsubushi ίϚϯυ΋ೖΔ
36. Go - ΫϥΠΞϯτϥΠϒϥϦ godoc.org/github.com/kayac/go-katsubushi#Client import "github.com/kayac/go-katsubushi" func main() { client := katsubushi.NewClient("127.0.0.1:11212", "10.8.0.1:11212") id, _ := client.Fetch() // uint64, error ids, _ := client.FetchMulti(3) // []uint64, error } ෳ਺ϗετͷࢼߦॱ͸ Perl ൛ͱಉ͡
37. Go - ID ͱ time.Time ͷ૬ޓม׵ godoc.org/github.com/kayac/go-katsubushi#ToID // ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ t1 := time.Now() id := katsubushi.ToID(t1) // ID͕ൃߦ͞Εͨ࣌ࠁΛ time.Time Ͱฦ͢ t2 := katsubushi.ToTime(id)
38. fluent-plugin-katsubushi rubygems.org/gems/fluent-plugin-katsubushi Fluentd ͷϑΟϧλʔϓϥάΠϯ ϑΟϧλʔΛ௨ա͢Δϩάʹ katsubushi Ͱൃߦͨ͠ ID Λ஫ೖ <filter> @type katsubushi id_key id # ஫ೖ͢Δkey໊ host localhost # katsubushi ͷϗετ໊ port 11212 # katsubushi ͷϙʔτ൪߸ </filter>
39. fluent-plugin-katsubushi fluent-plugin-katsubushi Λ௨ա͢Δͱ ID ͕஫ೖ͞ΕΔ {"message":"foo"} ! {"message":"foo","id":329518490450919424} ϩάʹ unique ͳ ID ΛৼΓ͍ͨʂ ͚ͲൃߦՕॴͰ͸ৼΕͳ͍ʂͱ͍͏৔߹ʹ
40. katsubushi ͷӡ༻
41. worker-id ΛҰҙʹ͢Δ ӡ༻͍ͯ͠Δશͯͷ katsubushi process ͷ worker-id ΛҰҙʹ͢Δͷ͕ඞਢ worker-id ॏෳ ➡ ಉҰ࣌ࠁʹൃߦ͞ΕͨID͕ॏෳ͢Δ ➡ ഁ໓ +------------------------------------------------------+ | 64 bit | +--+-----------------+----------------+----------------+ | 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit | +--+-----------------+----------------+----------------+
42. worker-id ΛҰҙʹ͢Δ worker-id ͸ 1ʙ1023 ͷ੔਺஋ ݻఆαʔόͰಈ͔͢ͳΒద੾ʹखͰ͚ͭΔ ಈతʹαʔό͕ىಈ͢Δ؀ڥͰ͸…ʁ ͨͱ͑͹ IP ΞυϨεͷୈ 3,4 octet Λ࢖͏ 192.168.0.10 ➡ 0 * 256 + 10 = 10 192.168.1.20 ➡ 1 * 256 + 20 = 266 192.168.2.30 ➡ 2 * 256 + 30 = 542
43. worker-id ΛҰҙʹ͢Δ ίϯςφ؀ڥͰ͸…ʁ Redis Λ࢖ͬͨ worker-id ࣗಈׂΓ౰ͯػೳ (ޙड़)
44. stats memcached ͷ stats ίϚϯυΛ࣮૷ $ echo "STATS" | nc localhost 11212 STAT pid 1 STAT uptime 51014 STAT time 1498716107 STAT version 1.3.0 STAT curr_connections 1 STAT total_connections 8 STAT cmd_get 18 STAT get_hits 20 STAT get_misses 0 END
45. key ஋ͷྫ ҙຯ pid 1 Process ID uptime 51014 ىಈ͔ͯ͠Βͷܦաඵ਺ time 1498716107 ݱࡏ࣌ࠁ UNIX time version 1.3.0 όʔδϣϯ curr_connections 1 ݱࡏ઀ଓ͍ͯ͠ΔTCPίωΫγϣϯ ਺ total_connections 8 ىಈ͔ͯ͠Β઀ଓͨ͠ίωΫγϣϯ ਺ (ྦྷܭ) cmd_get 18 GET ίϚϯυΛॲཧͨ͠ճ਺ (ྦྷܭ) get_hits 20 ൃߦͨ͠ ID ਺ (ྦྷܭ) get_misses 0 ൃߦͰ͖ͳ͔ͬͨ ID ਺ (ྦྷܭ)
46. ؂ࢹ͢΂͖஋ key curr_connections ଟ͗͢ͳ͍͔ uptime ר͖໭͍ͬͯΔ = ࠶ىಈͯ͠ ͍Δ get_misses ൃੜ = ࣌ࠁ͕ר͖໭͍ͬͯΔ
47. ࣌ࠁר͖໭Γ࣌ͷରԠ katsubushi ID ͷҰҙੑ͸࣌ࠁʹґଘ ࣌ࠁ͕ਐΉͷ͸໰୊ͳ͍͕ɺר͖໭Δͱ…(ͨͱ͑͹Ӟඵ) աڈʹൃߦͨ͠ ID ͱಉ͡ ID Λൃߦͯ͠͠·͏Մೳੑ ➡ ίʔυ಺Ͱ͸ר͖໭Βͳ͍Α͏ʹอޢ
48. ҎલͷλΠϜελϯϓ͕དྷͨΒΤϥʔ func (g *Generator) NextID() (uint64, error) { g.lock.Lock() defer g.lock.Unlock() ts := g.timestamp() // for rewind of server clock if ts < g.lastTimestamp { return 0, errors.New("system clock was rollbacked") }
49. Go 1.9 ͷ monotonic time Λ࢖͏(༧ఆ) Go 1.9 Ͱ monotonic time ͕ಋೖ͞ΕΔ 3 ىಈ࣌ʹ time.Now() ͨ࣌͠ࠁ͔Βͷࠩ෼Λར༻͢Ε͹ OS࣌ࠁ͕ר͖໭ͬͯ΋ӨڹΛड͚ͳ͍ 3 https://tip.golang.org/doc/go1.9#monotonic-time
50. ఀࢭ͍ͯ͠Δؒʹ࣌ࠁ͕໭ͬͨΒ… katsubushi ϓϩηε಺Ͱ͸ݕ஌͠Α͏͕ͳ͍ ରԠࡦɿࣗಈͰ࠶ىಈ͠ͳ͍ Perl / Go ͷΫϥΠΞϯτ͸ଞͷϗετʹ fallback Ͱ͖Δ ઃఆͯ͋͠Δશͯͷϗετ͕ಉ࣌ʹམͪͳ͚Ε͹ ID औಘͰ͖Δ
51. ҆ఆੑ 2015೥ʹϦϦʔε͔ͯ͠Β2೥ؒ Lobiͷຊ൪؀ڥͷ΄΅શͯͷϗετͰՔಇ͍ͯ͠Δ͕ ௨ৗར༻Ͱϓϩηε͕མͪͨ͜ͱ͸ͳ͍ 2 ϝϞϦϦʔΫ΋ܦݧͳ͠ 2 ͨͩ͠ memcached protocolͰਖ਼͘͠ΞΫηεͨ͠৔߹ɻෆਖ਼ͳίϚϯυͰΫϥογϡͨ͜͠ͱ͸͋ͬͨ github.com/kayac/ go-katsubushi/pull/8
52. WebΞϓϦέʔγϣϯ΁ͷԠ༻
53. Request ID ʹΑΔ ΞϓϦέʔγϣϯ௥੻
54. Request ID Λ katsubushi Ͱൃߦ nginx͕ϦΫΤετΛड͚෇͚ͨ࣌఺Ͱ katsubushi ID Λൃߦ ޙॲཧʹҾ͖౉͢͜ͱͰ֤छॲཧͷ௥੻Λ༰қʹ nginx ͱ katsubushi ͕ memcached protocol Ͱ௨৴
55. IDΛҾ͖ճͯ͠ ϩάʹه࿥
56. katsubushi ➡ nginx ΞΫηεຖʹ nginx lua ͕ ID Λऔಘͯ͠ม਺ $req_id ʹઃఆ upstream katsubushi { server 127.0.0.1:11212; keepalive 10; } location /katsubushi { internal; set $memcached_key "id"; # key͸ͳΜͰ΋OK memcached_pass katsubushi; } set $req_id 0; access_by_lua_block { local res = ngx.location.capture("/katsubushi") if res then ngx.var.req_id = res.body end }
57. nginx Ͱϩάʹग़ྗ log_format ltsv 'req_id:$req_id\t' 'host:$remote_addr\t' 'user:$remote_user\t' 'time:$time_iso8601\t' 'method:$request_method\t' ...
58. nginx ➡ App(Plack) ϦΫΤετϔομ X-Request-ID Λઃఆ upstream app { server unix:/var/tmp/app.socket; } location / { proxy_set_header X-Request-ID $req_id; proxy_pass http://app; }
59. App(Plack) ➡ local %ENV Plack::Middleware::SetLocalEnv Plack ͷ $env ͔Β local %ENV ʹ஋Λઃఆ͢Δ ΞϓϦέʔγϣϯͷͲ͔͜ΒͰ΋ $ENV{REQUEST_ID} ͰࢀরՄ # app.psgi use Plack::Builder; builder { enable "SetLocalEnv", REQUEST_ID => "HTTP_X_REQUEST_ID", ; # ... $app; }
60. Plack::Middleware::SetLocalEnv ࣮૷͸͜Ε͚ͩ local ͳͷͰείʔϓΛൈ͚Δͱ(ϦΫΤετॲཧ͕ऴΘΔͱ) ஋͕ݩʹ໭Δ sub call { my ($self, $env) = @_; local %ENV = %ENV; for my $key (keys %$self) { $ENV{$key} = $env->{ $self->{$key} } if exists $env->{$self->{$key}}; } $self->app->($env); }
61. App(Plack) ➡ app.log (Log::Minimal) $Log::Minimal::(PRINT|DIE) Ͱग़ྗΛΧελϚΠζ use Log::Minimal; BEGIN { $Log::Minimal::PRINT = sub { my ($time, $type, $message, $trace) = @_; my $req_id = $ENV{REQUEST_ID} // 0; warn "[$$][$req_id][$type] $message at $trace\n"; }; } warnf("something bad!"); [64302][329733861698727936][WARN] something bad! at .../App.pm line XX
62. App(Plack) ➡ Fluentd ϑΝΠϧΛհͣ͞௚઀ Fluent::Logger Ͱૹ৴͢Δ৔߹ ϝοηʔδʹ req_id Λ஫ೖ package Logger; use Fluent::Logger; sub log { my ($self, $tag, $msg) = @_; $msg->{req_id} = $ENV{REQUEST_ID} // 0; $logger->post($tag, $msg); # $logger is Fluent::Logger }
63. App(Plack) ➡ MySQL ΫΤϦͷઌ಄ʹSQLίϝϯτͰ /* req_id=... */ ΛຒΊࠐΉ use parent 'Teng::Sharding'; around ["execute", "do"] => sub { my ($orig, $self, $sql, @rest) = @_; my $req_id = $ENV{REQUEST_ID} // 0; return $orig->($self, "/* req_id=${req_id} */\n${sql}", @rest); }; # Time: 170629 3:27:47 # User@Host: xxx[xxx] @ [192.0.2.1] # Query_time: 1.080420 Lock_time: 0.000040 Rows_sent: 5 Rows_examined: 909850 SET timestamp=1498674467; /* req_id=329689349781536768 */ SELECT ..... FROM ...
64. ͍ΖΜͳى఺Ͱௐ͕ࠪͰ͖Δ ΞΫηεϩάͰ status:500 ➡ ΞϓϦͷΤϥʔϩάͱর߹ slow.log ʹࠔͬͨΫΤϦ͕ه࿥͞Ε͍ͯΔ ඇಉظ job ͕ࣦഊ ➡ ൃੜݩʹͳͬͨHTTPϦΫΤετΛΞΫηεϩάͰಛఆ
65. ଞܗࣜͷ Request ID ͱͷൺֱ $request_id (nginx >= 1.11.0) unique request identifier generated from 16 random bytes, in hexadecimal 4 ྫ: 9f53d64cf47dd966a7462f5a737b5ade ௕͍ ࣌ࠁ৘ใؚ͕·Εͳ͍ 4 http://nginx.org/en/docs/http/ngxhttpcoremodule.html#varrequest_id
66. ೔࣌ͰύʔςΟγϣχϯάΛߦ͏ ςʔϒϧͷϓϥΠϚϦΩʔͱͯ͠࢖͏
67. ҰఆظؒͰݹ͍σʔλΛফ͍ͨ͠ςʔϒϧ(௨஌) Redis ʹ id ͷΈอ࣋ͯ͠ index ʹ͍ͯ͠Δ id auto_increment, created_date ͷෳ߹Primary Key CREATE TABLE `notification` ( `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, `created_date` datetime NOT NULL, /* other columns */ PRIMARY KEY (`id`,`created_date`) ) ENGINE=InnoDB AUTO_INCREMENT=xxxxx DEFAULT CHARSET=utf8 PARTITION BY RANGE COLUMNS(created_date) (PARTITION p20170602 VALUES LESS THAN ('2017-06-02') ENGINE = InnoDB, PARTITION p20170603 VALUES LESS THAN ('2017-06-03') ENGINE = InnoDB, PARTITION p20170604 VALUES LESS THAN ('2017-06-04') ENGINE = InnoDB, .....
68. id + datetime ύʔςΟγϣχϯάͷ໰୊ idͷΈͰݕࡧ͢ΔΫΤϦͰ͸מΓࠐΊͳ͍ͨΊ શύʔςΟγϣϯΛݕࡧ͢Δ (30೔෼࢒ͨ͢Ίʹ30 partition੾Δͱ30ഒͷݕࡧίετ) ➡ Α͘΍Δ WHERE id IN (?,?,....) ͕஗͍ ➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍͸ͣͷύʔςΟγϣϯʹ සൟʹࢀর͕૸Δ (InnoDB Buffer Poolͷແବ)
69. ! katsubushi ID Ͱ RANGE ύʔςΟγϣϯΛ੾Ε͹…
70. katsubushi ID ͷΈͰ1೔୯ҐͷύʔςΟγϣχϯά CREATE TABLE `notification` ( `id` bigint(20) unsigned NOT NULL, /* AUTO_INCREMENT ͠ͳ͍ */ `created_date` datetime NOT NULL, /* ෆཁ͔΋͠Εͳ͍ */ /* other columns */ PRIMARY KEY (`id`) /* pkey ͸ id ͷΈ */ ) ENGINE=InnoDB DEFAULT CHARSET=utf8 PARTITION BY RANGE COLUMNS(id) (PARTITION p20170602 VALUES LESS THAN (319852589875200000) ENGINE = InnoDB, /* ^ 2017-06-02 00:00:00 JST */ PARTITION p20170603 VALUES LESS THAN (320214977740800000) ENGINE = InnoDB, /* ^ 2017-06-03 00:00:00 JST */ PARTITION p20170604 VALUES LESS THAN (320577365606400000) ENGINE = InnoDB, /* ^ 2017-06-04 00:00:00 JST */ .....
71. katsubushi ID only ύʔςΟγϣχϯάͷར఺ id ͷΈͷݕࡧΫΤϦ͕ࣗಈతʹמΓࠐ·ΕΔ ➡ Α͘΍Δ WHERE id IN (?,?,....) ͕࠷దԽ͞ΕΔ ➡ Row Reads ܹݮ ➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍ύʔςΟγϣϯʹ͸ ࢀর͕૸Βͳ͍ (InnoDB Buffer Poolͷޮ཰Up)
72. katsubushi ID only ύʔςΟγϣχϯάͷܽ఺? Ͳͷ஋Λڥʹ੾Ε͹͍͍ͷ͔͕௚ײతͰ͸ͳ͍ ! Katsubushi::Converter Ͱܭࢉ ಛఆͷ೔͚ͩࢀর͍ͨ͠ ! ೔ͷൣғΛ ID ʹม׵ͯ͠ WHERE ۟ʹೖΕΕ͹OK
73. ίϯςφ؀ڥ΁ͷదԠ
74. ίϯςφ؀ڥͰ͸ worker-id ΛҰҙʹ ͢Δͷ͕೉͍͠!
75. ίϯςφ؀ڥͰ worker-id ΛҰҙʹ͢Δ ❌ ਓྗͰͷݻఆׂΓ౰ͯ͸Ͱ͖ͳ͍ ੜ·Εͯ͸ফ͑Δͷ͕ίϯςφ ❌ IPΞυϨεΛ࢖͏͜ͱ΋Ͱ͖ͳ͍ ϗετͱNAT͍ͯ͠ΔͱɺผϗετͰಉҰͷ IP ΞυϨεΛ ࢖͍ͬͯΔίϯςφ͕ଘࡏ͠͏Δ Ͳ͏͢Ε͹…
76. Redis Ͱഉଞతʹ஋Λऔಘͤ͞Δ ͙͢ࢥ͍ͭ͘ͷ͸ू߹ܕͰ SPOP > SADD id_pool 1 2 3 4 # 4ཁૉͷू߹Λ࡞Δ > SPOP id_pool # ू߹͔ΒϥϯμϜʹҰͭऔΓग़͢ "3" > SMEMBERS id_pool # ࢒Γ "1" "2" "4" Simple is best?
77. SPOP ํࣜͷ໰୊ 1. ϓϩηεىಈ࣌ʹ SPOP Ͱऔಘ 2. ϓϩηεऴྃ࣌ʹ SADD Ͱฦ٫ औΓग़͞Εͨ஋͕ฦ٫͞ΕΔ͜ͱ͕อূͰ͖ͳ͍ ϓϩηε͸ਖ਼ৗऴྃ͢Δͱ͸ݶΒͳ͍ ➡ ௕ظؒӡ༻͍ͯ͠Δͱฦ٫͞Εͳ͍஋͕ ཷ·͍ͬͯͬͯރׇ͢ΔڪΕ
78. ϝϯςφϯεϑϦʔͳํ๏Λߟ͍͑ͨ
79. raus
80. raus github.com/fujiwara/raus "RAnged Unique id Supplier" RedisΛ࢖ͬͯҰఆൣғͷ੔਺IDΛ ॏෳͳׂ͘Γ౰ͯΔͨΊͷϥΠϒϥϦ
81. PubSub + SET NX + GETSET + EXPIRE 1. ੜ͖͍ͯΔϓϩηε͕ PubSub channel ʹอ͍࣋ͯ͠ΔIDΛ ྲྀ͢ 2. อ͍࣋ͯ͠ΔIDͷ Key Λ GETSET + EXPIRE Ͱߋ৽͠ଓ͚Δ 3. ৽نϓϩηε͸ PubSub channel Λ؍࡯ͯ͠ɺ࢖ΘΕ͍ͯͳ ͦ͏ͳIDΛ SET NX Ͱऔಘ 4. औಘͰ͖ͨΒ PubSub + GETSET + EXPIRE Λଓ͚Δ
82. 1. IDΛอ͍࣋ͯ͠Δطଘ raus ͸ PUBLISH + GETSET Λఆظతʹߦ͏
83. 2. ৽ن raus ͸ SUBSCRIBE ͯ͠ଞͷ raus ͕࢖͍ͬͯΔIDΛ೺Ѳ
84. 3. ৽ن raus ͸ଞͷ raus ͕࢖ͬͯͳ͍(͸ͣͷ) key Λ SET NX Ͱഉଞऔಘ
85. 4. ID ΛऔಘͰ͖ͨΒ PUBLISH + GETSET Λఆظ࣮ߦ
86. raus ͕ϝϯςφϯεϑϦʔͳཧ༝
87. raus ͕ϝϯςφϯεϑϦʔͳཧ༝ • ࢮΜͩϓϩηε͸ GETSET ͠ͳ͍ͷͰ key ͕࣌ݶ։์͞ΕΔ • ૄ௨Ͱ͖ͳ͘ͳͬͯ΋Ұఆ࣌ؒ͸ key ͕อ࣋͞Ε͍ͯΔͷͰ ଞ͕औಘͰ͖ͳ͍ • ૄ௨͕ճ෮ͨ͠ޙʹGETSETʹࣦഊͨ͠ (=ଞͷϓϩηε͕։์͞Εͨ key Λऔಘͯ͠͠·ͬͨ)৔߹ ଈ࠲ʹࢮ͵
88. katsubushi + raus $ katsubushi -redis redis://192.0.2.2:6379 Waiting for worker-id automated assignment (between 1 and 1023) with redis://192.0.2.2:6379 xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 <= ଞͷkatsubushi͕woker_id=2Λอ͍࣋ͯ͠Δ xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 ͷΛsubscribeͰ؍࡯͍ͯ͠Δ xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 candidate ids: [1 3 4 5 6 7 8 9 10 11] <= 2 Ҏ֎ͷΛީิʹͯͦ͠ͷத͔ΒϥϯμϜʹ trying to get lock key raus:id:8 <= SET NX got lock for 8 <= ੒ޭͨ͠ͷͰ 8 Λऔಘ Assigned worker-id: 8 Listening at [::]:11212 Worker ID = 8 <= PUBLISH + GETSET ΛཪͰଓ͚͍ͯΔ
89. katsubushi + raus worker_id=8 Λऔಘͨ͠ katsubushi ͕ಈ͍͍ͯΔঢ়ଶͰ key Λ্ॻ͖ͯ͠ΈΔͱ… $ redis-cli set raus:id:8 xxx unexpected uuid got: xxx panic: unexpected uuid got: xxx goroutine 42 [running]: main.assignWorkerID.func1(0xc4201462c0, 0xc42006f020) src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:230 +0xb0 created by main.assignWorkerID src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:235 +0x4fe
90. ·ͱΊ
91. ·ͱΊ • Snowflake ͱಉ༷ͷΞϧΰϦζϜͰ෼ࢄ؀ڥͰϢχʔΫͳID ΛൃߦͰ͖Δ katsubushi • ܰྔͳͷͰ DB ͷߦ ID Ҏ֎ʹ΋ར༻Մೳ ➡ Request ID Ͱॲཧ௥੻ • ࣌ࠁؚ͕·ΕΔಛੑΛੜ͔ͯ͠ύʔςΟγϣχϯάʹ΋ • ίϯςφ؀ڥʹରԠ͢Δ Redis Λ࢖ͬͨࣗಈΫϥελϦϯά