分散ユニークID採番機 katsubushi と Web アプリケーションへの応用例 / katsubushi

෼ࢄϢχʔΫID࠾൪ػ
katsubushi ͱ
Web ΞϓϦέʔγϣϯ
΁ͷԠ༻ྫ
2017.07.01 YAPC::Fukuoka
@fujiwara

View Slide

@fujiwara
ٕज़෦
github.com/fujiwara
sfujiwara.hatenablog.com

View Slide

Game & Community

View Slide

katsubushi - ෼ࢄϢχʔΫID࠾൪ػ
!= katsuobushi
!= katsubushi

View Slide

katsubushi - ෼ࢄϢχʔΫID࠾൪ػ
෼ࢄ = ୯Ұͷ (host | node | process) ʹґଘ͠ͳ͍
ϢχʔΫ = ෳ਺ͷ katsubushi Ͱൃߦͨ͠ ID ͕ॏෳ͠ͳ͍
Lobi Ͱͷ DB γϟʔσΟϯάͷཁ੥ʹ൐ͬͯ஀ੜ

View Slide

Lobi Ͱͷ DB γϟʔσΟϯά
MySQL Ͱී௨ͷ ୯Ұ master ෳ਺ slave ߏ੒ (ʙ2014)
σʔλɺΞΫηε૿ՃʹΑΓ୯Ұ master ͕ݶքʹ
(Ұ࣌తʹ i2.8xlarge Πϯελϯεʹͯ࣌ؒ͠Ք͗… !!!)
➡ ΍ΉΛಘͣɺDB Λ෼ׂ͢Δ͜ͱʹ (2015.02)
ڞ௨DB(ϢʔβͳͲ) x 1
άϧʔϓνϟοτؔ࿈DB x 2 ←ඞཁʹͳΕ͹૿΍ͤΔ

View Slide

DB γϟʔσΟϯά࣌ͷ՝୊
ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑
ID࠾൪
σʔλҠߦɺ੾Γସ͑ (ຊτʔΫͰ͸লུ)

View Slide


View Slide

άϧʔϓ͕Ͳͷγϟʔυʹଐ͢Δ͔ͷmapΛڞ௨DBʹ࣋ͭ
! γϟʔυҠಈ͠ͳ͚Ε͹ΩϟογϡՄೳ
ΫΤϦʹؚ·ΕΔςʔϒϧͱΧϥϜͷ஋ΛΈͯ…
! γϟʔυΛಛఆͰ͖Ε͹1γϟʔυʹ౤͛Δ
! Ͱ͖ͳ͚Ε͹ෳ਺γϟʔυʹ౤͛ͯ݁ՌΛ·ͱΊΔ
Teng ʹຐվ଄Λࢪͨ͠ Teng::Sharding Λ࡞੒(ඇެ։…)

View Slide

ID ࠾൪

View Slide

ID ͷཁ݅
֤γϟʔυͰॏෳ͠ͳ͍ϢχʔΫͳIDΛৼΓ͍ͨʂ
64bitҎԼͷ੔਺஋
! طʹ਺஋ͱͯ͠ѻ͏લఏͷίʔυ͕ࢁ΄Ͳ͋Δ
! DB ʹೖΕΔʹ͸ 64 bit ҎԼ͕Α͍
࣌ܥྻͷঢॱʹͳΔ͜ͱ (࣮༻্໰୊ͳ͍ਫ਼౓Ͱ)
! ࣌ܥྻͰιʔτ͢ΔͨΊʹ ORDER BY id (DESC) ͍ͯ͠Δ

View Slide

ͭ·Γ

View Slide

UUID ͸࠾༻Ͱ͖ͳ͍ !

View Slide

ʮFacebook, Twitter, Instagram౳͕Ͳ͏΍ͬͯIDΛੜ੒͍ͯ͠
Δͷ͔ ·ͱΊ1ʯΛࢀߟʹݕ౼
1 http://qiita.com/daisy1754/items/98a6e6b17d8161eab081

View Slide

Twitter
Snowflake github.com/twitter/snowflake
λΠϜελϯϓΛݩʹੜ੒ͨ͠ 64bit ੔਺
Instagram
Snowflake like ͳ 64bit ID Λ PL/PGSQL Ͱ
Flickr
MySQL ͷ Auto Increment Λ2୆༻ҙ
ͦΕͧΕح਺ͱۮ਺୲౰Ͱ +2 ͍ͯ͘͠

View Slide

View Slide

݁࿦ = Snowflake ํࣜ
+------------------------------------------------------+
| 64 bit |
+--+-----------------+----------------+----------------+
| 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit |
+--+-----------------+----------------+----------------+
Timestamp : Epoch ͔Βͷܦա࣌ؒ(msec)
WorkerID : Ϋϥελ಺ͰҰҙͷ ID
Sequence : ಉҰ Timestamp ಺ͷ࿈൪

View Slide

݁࿦ = Snowflake ํࣜ + ಠࣗσʔϞϯ
ಠࣗσʔϞϯ
! Go Ͱ࣮૷͠ɺ1ϗετʹ1ϓϩηεىಈ͢Δ
! ଟݴޠɺϛυϧ΢ΣΞ͔Β࠾൪Մೳʹ
௨৴ϓϩτίϧ͸ memcached text protocol
! ܰྔɾ࣮૷͕؆୯ (GET ͔͠͠ͳ͍ͷͰ)
! ֤छݴޠͰ࢖͑Δ
! PerlͷΫϥΠΞϯτ͕଎͍ (Cache::Memcached::Fast)

View Slide

katsubushi ஀ੜ
github.com/kayac/go-katsubushi
໊લͷ༝དྷ
Lobiͱ͍͑͹͵͜͞Μ
➡ Snowﬂake (ઇͷ݁থɺࡉ͔͍)
➡ ͵͜ Ͷ͜ ͔͚Β ࡉ͔͍
➡ ׍અʂ

View Slide

katsubushi ͷಛੑͱੑೳ

View Slide

ID ͷߏ଄
329508644217425920 (10ਐදݱ)
+-+-----------------------------------------+----------+------------+
| | Timestamp | WorkerID | Sequence |
+-+-----------------------------------------+----------+------------+
|0|00001001001001010100110010111011011100100|0000000001|000000000000|
+-+-----------------------------------------+----------+------------+
| | 78,560,982,756ms since epoch | 1| 0|
+-+-----------------------------------------+----------+------------+
= 2017-06-28T06:29:42.756Z
bitͷ഑෼͸ Snowﬂake ͱಉ༷
Epoch 2015-01-01T00:00:00Z (Snowﬂakeͱ͸ҟͳΔ)
࣌ܥྻͰঢॱɺID ͔Βൃߦ࣌ࠁͷٯࢉ͕Մೳ (msecਫ਼౓)

View Slide

ID ͸͍ͭ·Ͱ࢖͑Δͷ͔
Timestamp 41bit
2015-01-01T00:00:00 UTC + 0b1{41} msec
=> 2084-09-06T15:47:35.551 UTC
໿69೥ؒ

View Slide

1ඵؒʹൃߦͰ͖ΔID਺
1ms ͋ͨΓ 12bit (4096) ͷ Sequence Λ΋ͭ
= 4,096,000/sec (ཧ࿦஋)
WorkerID 10bit (1024) ͳͷͰΫϥελશମͰ͸
4,194,304,000 (≒32bit)
ຖඵ໿32bitਐΉ

View Slide

࣮ࡍͷੑೳ
$ go test -bench Generate
BenchmarkGenerateID-4 5000000 244 ns/op
1ID ੜ੒ʹ 244ns
244 ns x 4,096,000 ≒ 1 sec
Go ಺෦Ͱͷൃ൪͸ཧ࿦஋ݶք·ͰՄೳ
(Sequence ͕ 4095 Λ௒͑Δͱ࣍ͷ Timestamp ·Ͱ sleep)

View Slide

ωοτϫʔΫαʔόʔͱͯ͠ͷੑೳ
Client: Go katsubushi.Client on AWS c4.8xlarge (36 cores)
Server: katsubushi v1.3.0 on AWS c3.2xlarge (8 cores)
ಉҰAZͷผϗετͰܭଌ
ฒྻ౓ fetch=1 fetch=10 fetch=100
1 5,567 52,003 258,077
10 52,936 434,977 1,190,018
100 151,188 826,200 1,164,833
Ұ౓ʹ 100 ID Λൃߦ͢Ε͹ 100ສ/sec Ҏ্

View Slide

View Slide

katsubushi ͷ࢖͍͔ͨ

View Slide

Install
Docker
$ docker pull katsubushi/katsubushi
$ docker run -p 11212:11212 katsubushi/katsubushi
Release binary
github.com/kayac/go-katsubushi/releases

View Slide

worker-id Λࢦఆͯ͠ىಈ͢Δͱ TCP 11212 Λ Listen
$ katsubushi -worker-id 1
2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:152 Listening at [::]:11212
2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:153 Worker ID = 1
Docker ͷ৔߹͸؀ڥม਺Ͱ
$ docker run -p 11212:11212 -e worker_id=2 katsubushi/katsubushi
2017-06-29T05:58:39.847Z INFO go-katsubushi/app.go:152 Listening at [::]:11212
2017-06-29T05:58:39.848Z INFO go-katsubushi/app.go:153 Worker ID = 2

View Slide

nc ίϚϯυͰखͰ஻ͬͯΈ·͠ΐ͏
GET ͷ key ͸೚ҙͷ஋Ͱ OK
$ nc localhost 11212
GET id
VALUE id 0 18
329508644217425920
END

View Slide

ෳ਺IDΛ·ͱΊͯऔಘ΋Ͱ͖·͢
GET id1 id2
VALUE id1 0 18
329508654694797312
VALUE id2 0 18
329508654694797313
END

View Slide

ී௨ͷ memcached client ͕࢖͑·͢
(ςΩετϓϩτίϧͷΈ)
use Cache::Memcached;
$c = Cache::Memcached->new( servers => ["127.0.0.1:11212"] );
$c->get("id"); #= 329521618642538496
Perl Cache::Memcached, C::M::Fast, C::M::AnyEvent
Go github.com/bradﬁtz/gomemcache
Ruby memcached(gem)

View Slide

katsubushi ؔ࿈ϥΠϒϥϦ

View Slide

Perl - ΫϥΠΞϯτϥΠϒϥϦ
metacpan.org/pod/Katsubushi::Client
use Katubushi::Client;
my $client = Katsubushi::Client->new({
servers => ["127.0.0.1:11212", "10.8.0.1:11212"],
});
my $id = $client->fetch;
my @ids = $client->fetch_multi(3);
serversʹࢦఆͨ͠ϗετͷઌ಄͔Βऔಘ
ࣦഊͨ͠Β࣍ͷϗετ΁

View Slide

Perl - ID ͱ UNIX time ͷ૬ޓม׵
use Katsubushi::Converter
# ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ
my $now = time();
my $id = Katsubushi::Converter::epoch_to_id($now);
# ID͕ൃߦ͞Εͨ࣌ࠁΛunix timeͰऔಘ
my $t = Katsubushi::Converter::id_to_epoch($id);
katsubushi2epoch, epoch2katsubushi ίϚϯυ΋ೖΔ

View Slide

Go - ΫϥΠΞϯτϥΠϒϥϦ
godoc.org/github.com/kayac/go-katsubushi#Client
import "github.com/kayac/go-katsubushi"
func main() {
client := katsubushi.NewClient("127.0.0.1:11212", "10.8.0.1:11212")
id, _ := client.Fetch() // uint64, error
ids, _ := client.FetchMulti(3) // []uint64, error
}
ෳ਺ϗετͷࢼߦॱ͸ Perl ൛ͱಉ͡

View Slide

Go - ID ͱ time.Time ͷ૬ޓม׵
godoc.org/github.com/kayac/go-katsubushi#ToID
// ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ
t1 := time.Now()
id := katsubushi.ToID(t1)
// ID͕ൃߦ͞Εͨ࣌ࠁΛ time.Time Ͱฦ͢
t2 := katsubushi.ToTime(id)

View Slide

fluent-plugin-katsubushi
rubygems.org/gems/ﬂuent-plugin-katsubushi
Fluentd ͷϑΟϧλʔϓϥάΠϯ
ϑΟϧλʔΛ௨ա͢Δϩάʹ katsubushi Ͱൃߦͨ͠ ID Λ஫ೖ

@type katsubushi
id_key id # ஫ೖ͢Δkey໊
host localhost # katsubushi ͷϗετ໊
port 11212 # katsubushi ͷϙʔτ൪߸

View Slide

fluent-plugin-katsubushi
ﬂuent-plugin-katsubushi Λ௨ա͢Δͱ ID ͕஫ೖ͞ΕΔ
{"message":"foo"}
!
{"message":"foo","id":329518490450919424}
ϩάʹ unique ͳ ID ΛৼΓ͍ͨʂ
͚ͲൃߦՕॴͰ͸ৼΕͳ͍ʂͱ͍͏৔߹ʹ

View Slide

katsubushi ͷӡ༻

View Slide

worker-id ΛҰҙʹ͢Δ
ӡ༻͍ͯ͠Δશͯͷ katsubushi process ͷ
worker-id ΛҰҙʹ͢Δͷ͕ඞਢ
worker-id ॏෳ ➡ ಉҰ࣌ࠁʹൃߦ͞ΕͨID͕ॏෳ͢Δ ➡ ഁ໓
+------------------------------------------------------+
| 64 bit |
+--+-----------------+----------------+----------------+
| 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit |
+--+-----------------+----------------+----------------+

View Slide

worker-id ͸ 1ʙ1023 ͷ੔਺஋
ݻఆαʔόͰಈ͔͢ͳΒద੾ʹखͰ͚ͭΔ
ಈతʹαʔό͕ىಈ͢Δ؀ڥͰ͸…ʁ
ͨͱ͑͹ IP ΞυϨεͷୈ 3,4 octet Λ࢖͏
192.168.0.10 ➡ 0 * 256 + 10 = 10
192.168.1.20 ➡ 1 * 256 + 20 = 266
192.168.2.30 ➡ 2 * 256 + 30 = 542

View Slide

ίϯςφ؀ڥͰ͸…ʁ
Redis Λ࢖ͬͨ worker-id ࣗಈׂΓ౰ͯػೳ (ޙड़)

View Slide

stats
memcached ͷ stats ίϚϯυΛ࣮૷
$ echo "STATS" | nc localhost 11212
STAT pid 1
STAT uptime 51014
STAT time 1498716107
STAT version 1.3.0
STAT curr_connections 1
STAT total_connections 8
STAT cmd_get 18
STAT get_hits 20
STAT get_misses 0
END

View Slide

key ஋ͷྫ ҙຯ
pid 1 Process ID
uptime 51014 ىಈ͔ͯ͠Βͷܦաඵ਺
time 1498716107 ݱࡏ࣌ࠁ UNIX time
version 1.3.0 όʔδϣϯ
curr_connections 1 ݱࡏ઀ଓ͍ͯ͠ΔTCPίωΫγϣϯ
਺
total_connections 8 ىಈ͔ͯ͠Β઀ଓͨ͠ίωΫγϣϯ
਺ (ྦྷܭ)
cmd_get 18 GET ίϚϯυΛॲཧͨ͠ճ਺ (ྦྷܭ)
get_hits 20 ൃߦͨ͠ ID ਺ (ྦྷܭ)
get_misses 0 ൃߦͰ͖ͳ͔ͬͨ ID ਺ (ྦྷܭ)

View Slide

؂ࢹ͢΂͖஋
key
curr_connections ଟ͗͢ͳ͍͔
uptime ר͖໭͍ͬͯΔ = ࠶ىಈͯ͠
͍Δ
get_misses ൃੜ = ࣌ࠁ͕ר͖໭͍ͬͯΔ

View Slide

࣌ࠁר͖໭Γ࣌ͷରԠ
katsubushi ID ͷҰҙੑ͸࣌ࠁʹґଘ
࣌ࠁ͕ਐΉͷ͸໰୊ͳ͍͕ɺר͖໭Δͱ…(ͨͱ͑͹Ӟඵ)
աڈʹൃߦͨ͠ ID ͱಉ͡ ID Λൃߦͯ͠͠·͏Մೳੑ
➡ ίʔυ಺Ͱ͸ר͖໭Βͳ͍Α͏ʹอޢ

View Slide

ҎલͷλΠϜελϯϓ͕དྷͨΒΤϥʔ
func (g *Generator) NextID() (uint64, error) {
g.lock.Lock()
defer g.lock.Unlock()
ts := g.timestamp()
// for rewind of server clock
if ts < g.lastTimestamp {
return 0, errors.New("system clock was rollbacked")
}

View Slide

Go 1.9 ͷ monotonic time Λ࢖͏(༧ఆ)
Go 1.9 Ͱ monotonic time ͕ಋೖ͞ΕΔ 3
ىಈ࣌ʹ time.Now() ͨ࣌͠ࠁ͔Βͷࠩ෼Λར༻͢Ε͹
OS࣌ࠁ͕ר͖໭ͬͯ΋ӨڹΛड͚ͳ͍
3 https://tip.golang.org/doc/go1.9#monotonic-time

View Slide

ఀࢭ͍ͯ͠Δؒʹ࣌ࠁ͕໭ͬͨΒ…
katsubushi ϓϩηε಺Ͱ͸ݕ஌͠Α͏͕ͳ͍
ରԠࡦɿࣗಈͰ࠶ىಈ͠ͳ͍
Perl / Go ͷΫϥΠΞϯτ͸ଞͷϗετʹ fallback Ͱ͖Δ
ઃఆͯ͋͠Δશͯͷϗετ͕ಉ࣌ʹམͪͳ͚Ε͹ ID औಘͰ͖Δ

View Slide

҆ఆੑ
2015೥ʹϦϦʔε͔ͯ͠Β2೥ؒ
Lobiͷຊ൪؀ڥͷ΄΅શͯͷϗετͰՔಇ͍ͯ͠Δ͕
௨ৗར༻Ͱϓϩηε͕མͪͨ͜ͱ͸ͳ͍ 2
ϝϞϦϦʔΫ΋ܦݧͳ͠
2 ͨͩ͠ memcached protocolͰਖ਼͘͠ΞΫηεͨ͠৔߹ɻෆਖ਼ͳίϚϯυͰΫϥογϡͨ͜͠ͱ͸͋ͬͨ github.com/kayac/
go-katsubushi/pull/8

View Slide

WebΞϓϦέʔγϣϯ΁ͷԠ༻

View Slide

Request ID ʹΑΔ
ΞϓϦέʔγϣϯ௥੻

View Slide

Request ID Λ katsubushi Ͱൃߦ
nginx͕ϦΫΤετΛड͚෇͚ͨ࣌఺Ͱ katsubushi ID Λൃߦ
ޙॲཧʹҾ͖౉͢͜ͱͰ֤छॲཧͷ௥੻Λ༰қʹ
nginx ͱ katsubushi ͕ memcached protocol Ͱ௨৴

View Slide

IDΛҾ͖ճͯ͠
ϩάʹه࿥

View Slide

katsubushi ➡ nginx
ΞΫηεຖʹ nginx lua ͕ ID Λऔಘͯ͠ม਺ $req_id ʹઃఆ
upstream katsubushi {
server 127.0.0.1:11212;
keepalive 10;
}
location /katsubushi {
internal;
set $memcached_key "id"; # key͸ͳΜͰ΋OK
memcached_pass katsubushi;
}
set $req_id 0;
access_by_lua_block {
local res = ngx.location.capture("/katsubushi")
if res then
ngx.var.req_id = res.body
end
}

View Slide

nginx Ͱϩάʹग़ྗ
log_format ltsv
'req_id:$req_id\t'
'host:$remote_addr\t'
'user:$remote_user\t'
'time:$time_iso8601\t'
'method:$request_method\t'
...

View Slide

nginx ➡ App(Plack)
ϦΫΤετϔομ X-Request-ID Λઃఆ
upstream app {
server unix:/var/tmp/app.socket;
}
location / {
proxy_set_header X-Request-ID $req_id;
proxy_pass http://app;
}

View Slide

App(Plack) ➡ local %ENV
Plack::Middleware::SetLocalEnv
Plack ͷ $env ͔Β local %ENV ʹ஋Λઃఆ͢Δ
ΞϓϦέʔγϣϯͷͲ͔͜ΒͰ΋ $ENV{REQUEST_ID} ͰࢀরՄ
# app.psgi
use Plack::Builder;
builder {
enable "SetLocalEnv",
REQUEST_ID => "HTTP_X_REQUEST_ID",
;
# ...
$app;
}

View Slide

Plack::Middleware::SetLocalEnv
࣮૷͸͜Ε͚ͩ
local ͳͷͰείʔϓΛൈ͚Δͱ(ϦΫΤετॲཧ͕ऴΘΔͱ)
஋͕ݩʹ໭Δ
sub call {
my ($self, $env) = @_;
local %ENV = %ENV;
for my $key (keys %$self) {
$ENV{$key} = $env->{ $self->{$key} }
if exists $env->{$self->{$key}};
}
$self->app->($env);
}

View Slide

App(Plack) ➡ app.log (Log::Minimal)
$Log::Minimal::(PRINT|DIE) Ͱग़ྗΛΧελϚΠζ
use Log::Minimal;
BEGIN {
$Log::Minimal::PRINT = sub {
my ($time, $type, $message, $trace) = @_;
my $req_id = $ENV{REQUEST_ID} // 0;
warn "[$$][$req_id][$type] $message at $trace\n";
};
}
warnf("something bad!");
[64302][329733861698727936][WARN] something bad! at .../App.pm line XX

View Slide

App(Plack) ➡ Fluentd
ϑΝΠϧΛհͣ͞௚઀ Fluent::Logger Ͱૹ৴͢Δ৔߹
ϝοηʔδʹ req_id Λ஫ೖ
package Logger;
use Fluent::Logger;
sub log {
my ($self, $tag, $msg) = @_;
$msg->{req_id} = $ENV{REQUEST_ID} // 0;
$logger->post($tag, $msg); # $logger is Fluent::Logger
}

View Slide

App(Plack) ➡ MySQL
ΫΤϦͷઌ಄ʹSQLίϝϯτͰ /* req_id=... */ ΛຒΊࠐΉ
use parent 'Teng::Sharding';
around ["execute", "do"] => sub {
my ($orig, $self, $sql, @rest) = @_;
my $req_id = $ENV{REQUEST_ID} // 0;
return $orig->($self, "/* req_id=${req_id} */\n${sql}", @rest);
};
# Time: 170629 3:27:47
# [email protected]: xxx[xxx] @ [192.0.2.1]
# Query_time: 1.080420 Lock_time: 0.000040 Rows_sent: 5 Rows_examined: 909850
SET timestamp=1498674467;
/* req_id=329689349781536768 */
SELECT ..... FROM ...

View Slide

͍ΖΜͳى఺Ͱௐ͕ࠪͰ͖Δ
ΞΫηεϩάͰ status:500
➡ ΞϓϦͷΤϥʔϩάͱর߹
slow.log ʹࠔͬͨΫΤϦ͕ه࿥͞Ε͍ͯΔ
ඇಉظ job ͕ࣦഊ
➡ ൃੜݩʹͳͬͨHTTPϦΫΤετΛΞΫηεϩάͰಛఆ

View Slide

ଞܗࣜͷ Request ID ͱͷൺֱ
$request_id (nginx >= 1.11.0)
unique request identiﬁer generated from 16 random bytes, in
hexadecimal 4
ྫ: 9f53d64cf47dd966a7462f5a737b5ade
௕͍
࣌ࠁ৘ใؚ͕·Εͳ͍
4 http://nginx.org/en/docs/http/ngxhttpcoremodule.html#varrequest_id

View Slide

೔࣌ͰύʔςΟγϣχϯάΛߦ͏
ςʔϒϧͷϓϥΠϚϦΩʔͱͯ͠࢖͏

View Slide

ҰఆظؒͰݹ͍σʔλΛফ͍ͨ͠ςʔϒϧ(௨஌)
Redis ʹ id ͷΈอ࣋ͯ͠ index ʹ͍ͯ͠Δ
id auto_increment, created_date ͷෳ߹Primary Key
CREATE TABLE `notification` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`created_date` datetime NOT NULL,
/* other columns */
PRIMARY KEY (`id`,`created_date`)
) ENGINE=InnoDB AUTO_INCREMENT=xxxxx DEFAULT CHARSET=utf8
PARTITION BY RANGE COLUMNS(created_date)
(PARTITION p20170602 VALUES LESS THAN ('2017-06-02') ENGINE = InnoDB,
PARTITION p20170603 VALUES LESS THAN ('2017-06-03') ENGINE = InnoDB,
PARTITION p20170604 VALUES LESS THAN ('2017-06-04') ENGINE = InnoDB,
.....

View Slide

id + datetime ύʔςΟγϣχϯάͷ໰୊
idͷΈͰݕࡧ͢ΔΫΤϦͰ͸מΓࠐΊͳ͍ͨΊ
શύʔςΟγϣϯΛݕࡧ͢Δ
(30೔෼࢒ͨ͢Ίʹ30 partition੾Δͱ30ഒͷݕࡧίετ)
➡ Α͘΍Δ WHERE id IN (?,?,....) ͕஗͍
➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍͸ͣͷύʔςΟγϣϯʹ
සൟʹࢀর͕૸Δ (InnoDB Buffer Poolͷແବ)

View Slide

! katsubushi ID Ͱ RANGE
ύʔςΟγϣϯΛ੾Ε͹…

View Slide

katsubushi ID ͷΈͰ1೔୯ҐͷύʔςΟγϣχϯά
CREATE TABLE `notification` (
`id` bigint(20) unsigned NOT NULL, /* AUTO_INCREMENT ͠ͳ͍ */
`created_date` datetime NOT NULL, /* ෆཁ͔΋͠Εͳ͍ */
/* other columns */
PRIMARY KEY (`id`) /* pkey ͸ id ͷΈ */
) ENGINE=InnoDB DEFAULT CHARSET=utf8
PARTITION BY RANGE COLUMNS(id)
(PARTITION p20170602 VALUES LESS THAN (319852589875200000) ENGINE = InnoDB,
/* ^ 2017-06-02 00:00:00 JST */
PARTITION p20170603 VALUES LESS THAN (320214977740800000) ENGINE = InnoDB,
/* ^ 2017-06-03 00:00:00 JST */
PARTITION p20170604 VALUES LESS THAN (320577365606400000) ENGINE = InnoDB,
/* ^ 2017-06-04 00:00:00 JST */
.....

View Slide

katsubushi ID only ύʔςΟγϣχϯάͷར఺
id ͷΈͷݕࡧΫΤϦ͕ࣗಈతʹמΓࠐ·ΕΔ
➡ Α͘΍Δ WHERE id IN (?,?,....) ͕࠷దԽ͞ΕΔ
➡ Row Reads ܹݮ
➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍ύʔςΟγϣϯʹ͸
ࢀর͕૸Βͳ͍ (InnoDB Buffer Poolͷޮ཰Up)

View Slide

katsubushi ID only ύʔςΟγϣχϯάͷܽ఺?
Ͳͷ஋Λڥʹ੾Ε͹͍͍ͷ͔͕௚ײతͰ͸ͳ͍
! Katsubushi::Converter Ͱܭࢉ
ಛఆͷ೔͚ͩࢀর͍ͨ͠
! ೔ͷൣғΛ ID ʹม׵ͯ͠ WHERE ۟ʹೖΕΕ͹OK

View Slide

ίϯςφ؀ڥ΁ͷదԠ

View Slide

ίϯςφ؀ڥͰ͸ worker-id ΛҰҙʹ
͢Δͷ͕೉͍͠!

View Slide

ίϯςφ؀ڥͰ worker-id ΛҰҙʹ͢Δ
❌ ਓྗͰͷݻఆׂΓ౰ͯ͸Ͱ͖ͳ͍
ੜ·Εͯ͸ফ͑Δͷ͕ίϯςφ
❌ IPΞυϨεΛ࢖͏͜ͱ΋Ͱ͖ͳ͍
ϗετͱNAT͍ͯ͠ΔͱɺผϗετͰಉҰͷ IP ΞυϨεΛ
࢖͍ͬͯΔίϯςφ͕ଘࡏ͠͏Δ
Ͳ͏͢Ε͹…

View Slide

Redis Ͱഉଞతʹ஋Λऔಘͤ͞Δ
͙͢ࢥ͍ͭ͘ͷ͸ू߹ܕͰ SPOP
> SADD id_pool 1 2 3 4 # 4ཁૉͷू߹Λ࡞Δ
> SPOP id_pool # ू߹͔ΒϥϯμϜʹҰͭऔΓग़͢
"3"
> SMEMBERS id_pool # ࢒Γ
"1"
"2"
"4"
Simple is best?

View Slide

SPOP ํࣜͷ໰୊
1. ϓϩηεىಈ࣌ʹ SPOP Ͱऔಘ
2. ϓϩηεऴྃ࣌ʹ SADD Ͱฦ٫
औΓग़͞Εͨ஋͕ฦ٫͞ΕΔ͜ͱ͕อূͰ͖ͳ͍
ϓϩηε͸ਖ਼ৗऴྃ͢Δͱ͸ݶΒͳ͍
➡ ௕ظؒӡ༻͍ͯ͠Δͱฦ٫͞Εͳ͍஋͕
ཷ·͍ͬͯͬͯރׇ͢ΔڪΕ

View Slide

ϝϯςφϯεϑϦʔͳํ๏Λߟ͍͑ͨ

View Slide

raus

View Slide

raus
github.com/fujiwara/raus
"RAnged Unique id Supplier"
RedisΛ࢖ͬͯҰఆൣғͷ੔਺IDΛ
ॏෳͳׂ͘Γ౰ͯΔͨΊͷϥΠϒϥϦ

View Slide

PubSub + SET NX + GETSET + EXPIRE
1. ੜ͖͍ͯΔϓϩηε͕ PubSub channel ʹอ͍࣋ͯ͠ΔIDΛ
ྲྀ͢
2. อ͍࣋ͯ͠ΔIDͷ Key Λ GETSET + EXPIRE Ͱߋ৽͠ଓ͚Δ
3. ৽نϓϩηε͸ PubSub channel Λ؍࡯ͯ͠ɺ࢖ΘΕ͍ͯͳ
ͦ͏ͳIDΛ SET NX Ͱऔಘ
4. औಘͰ͖ͨΒ PubSub + GETSET + EXPIRE Λଓ͚Δ

View Slide

1. IDΛอ͍࣋ͯ͠Δطଘ raus ͸ PUBLISH + GETSET Λఆظతʹߦ͏

View Slide

2. ৽ن raus ͸ SUBSCRIBE ͯ͠ଞͷ raus ͕࢖͍ͬͯΔIDΛ೺Ѳ

View Slide

3. ৽ن raus ͸ଞͷ raus ͕࢖ͬͯͳ͍(͸ͣͷ) key Λ SET NX Ͱഉଞऔಘ

View Slide

4. ID ΛऔಘͰ͖ͨΒ PUBLISH + GETSET Λఆظ࣮ߦ

View Slide

raus ͕ϝϯςφϯεϑϦʔͳཧ༝

View Slide

raus ͕ϝϯςφϯεϑϦʔͳཧ༝
• ࢮΜͩϓϩηε͸ GETSET ͠ͳ͍ͷͰ key ͕࣌ݶ։์͞ΕΔ
• ૄ௨Ͱ͖ͳ͘ͳͬͯ΋Ұఆ࣌ؒ͸ key ͕อ࣋͞Ε͍ͯΔͷͰ
ଞ͕औಘͰ͖ͳ͍
• ૄ௨͕ճ෮ͨ͠ޙʹGETSETʹࣦഊͨ͠
(=ଞͷϓϩηε͕։์͞Εͨ key Λऔಘͯ͠͠·ͬͨ)৔߹
ଈ࠲ʹࢮ͵

View Slide

katsubushi + raus
$ katsubushi -redis redis://192.0.2.2:6379
Waiting for worker-id automated assignment (between 1 and 1023) with redis://192.0.2.2:6379
xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 <= ଞͷkatsubushi͕woker_id=2Λอ͍࣋ͯ͠Δ
xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 ͷΛsubscribeͰ؍࡯͍ͯ͠Δ
xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2
xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2
candidate ids: [1 3 4 5 6 7 8 9 10 11] <= 2 Ҏ֎ͷΛީิʹͯͦ͠ͷத͔ΒϥϯμϜʹ
trying to get lock key raus:id:8 <= SET NX
got lock for 8 <= ੒ޭͨ͠ͷͰ 8 Λऔಘ
Assigned worker-id: 8
Listening at [::]:11212
Worker ID = 8
<= PUBLISH + GETSET ΛཪͰଓ͚͍ͯΔ

View Slide

katsubushi + raus
worker_id=8 Λऔಘͨ͠ katsubushi ͕ಈ͍͍ͯΔঢ়ଶͰ
key Λ্ॻ͖ͯ͠ΈΔͱ…
$ redis-cli set raus:id:8 xxx
unexpected uuid got: xxx
panic: unexpected uuid got: xxx
goroutine 42 [running]:
main.assignWorkerID.func1(0xc4201462c0, 0xc42006f020)
src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:230 +0xb0
created by main.assignWorkerID
src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:235 +0x4fe

View Slide

·ͱΊ

View Slide

·ͱΊ
• Snowﬂake ͱಉ༷ͷΞϧΰϦζϜͰ෼ࢄ؀ڥͰϢχʔΫͳID
ΛൃߦͰ͖Δ katsubushi
• ܰྔͳͷͰ DB ͷߦ ID Ҏ֎ʹ΋ར༻Մೳ
➡ Request ID Ͱॲཧ௥੻
• ࣌ࠁؚ͕·ΕΔಛੑΛੜ͔ͯ͠ύʔςΟγϣχϯάʹ΋
• ίϯςφ؀ڥʹରԠ͢Δ Redis Λ࢖ͬͨࣗಈΫϥελϦϯά

View Slide

分散ユニークID採番機 katsubushi と Web アプリケーションへの応用例 / katsubushi

分散ユニークID採番機 katsubushi と Web アプリケーションへの応用例 / katsubushi

FUJIWARA Shunichiro

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Featured

Transcript