Slide 1
Slide 1 text
ࢄϢχʔΫID࠾൪ػ
katsubushi ͱ
Web ΞϓϦέʔγϣϯ
ͷԠ༻ྫ
2017.07.01 YAPC::Fukuoka
@fujiwara
Slide 2
Slide 2 text
@fujiwara
ٕज़෦
github.com/fujiwara
sfujiwara.hatenablog.com
Slide 3
Slide 3 text
Game & Community
Slide 4
Slide 4 text
katsubushi - ࢄϢχʔΫID࠾൪ػ
!= katsuobushi
!= katsubushi
Slide 5
Slide 5 text
katsubushi - ࢄϢχʔΫID࠾൪ػ
ࢄ = ୯Ұͷ (host | node | process) ʹґଘ͠ͳ͍
ϢχʔΫ = ෳͷ katsubushi Ͱൃߦͨ͠ ID ͕ॏෳ͠ͳ͍
Lobi Ͱͷ DB γϟʔσΟϯάͷཁʹͬͯੜ
Slide 6
Slide 6 text
Lobi Ͱͷ DB γϟʔσΟϯά
MySQL Ͱී௨ͷ ୯Ұ master ෳ slave ߏ (ʙ2014)
σʔλɺΞΫηε૿ՃʹΑΓ୯Ұ master ͕ݶքʹ
(Ұ࣌తʹ i2.8xlarge Πϯελϯεʹͯ࣌ؒ͠Ք͗… !!!)
➡ ΉΛಘͣɺDB Λׂ͢Δ͜ͱʹ (2015.02)
ڞ௨DB(ϢʔβͳͲ) x 1
άϧʔϓνϟοτؔ࿈DB x 2 ←ඞཁʹͳΕ૿ͤΔ
Slide 7
Slide 7 text
DB γϟʔσΟϯά࣌ͷ՝
ΞϓϦέʔγϣϯίʔυͷॻ͖͑
ID࠾൪
σʔλҠߦɺΓସ͑ (ຊτʔΫͰলུ)
Slide 8
Slide 8 text
DB γϟʔσΟϯά࣌ͷ՝
ΞϓϦέʔγϣϯίʔυͷॻ͖͑
Slide 9
Slide 9 text
ΞϓϦέʔγϣϯίʔυͷॻ͖͑
άϧʔϓ͕Ͳͷγϟʔυʹଐ͢Δ͔ͷmapΛڞ௨DBʹ࣋ͭ
! γϟʔυҠಈ͠ͳ͚ΕΩϟογϡՄೳ
ΫΤϦʹؚ·ΕΔςʔϒϧͱΧϥϜͷΛΈͯ…
! γϟʔυΛಛఆͰ͖Ε1γϟʔυʹ͛Δ
! Ͱ͖ͳ͚Εෳγϟʔυʹ͛ͯ݁ՌΛ·ͱΊΔ
Teng ʹຐվΛࢪͨ͠ Teng::Sharding Λ࡞(ඇެ։…)
Slide 10
Slide 10 text
DB γϟʔσΟϯά࣌ͷ՝
ID ࠾൪
Slide 11
Slide 11 text
ID ͷཁ݅
֤γϟʔυͰॏෳ͠ͳ͍ϢχʔΫͳIDΛৼΓ͍ͨʂ
64bitҎԼͷ
! طʹͱͯ͠ѻ͏લఏͷίʔυ͕ࢁ΄Ͳ͋Δ
! DB ʹೖΕΔʹ 64 bit ҎԼ͕Α͍
࣌ܥྻͷঢॱʹͳΔ͜ͱ (࣮༻্ͳ͍ਫ਼Ͱ)
! ࣌ܥྻͰιʔτ͢ΔͨΊʹ ORDER BY id (DESC) ͍ͯ͠Δ
Slide 12
Slide 12 text
ͭ·Γ
Slide 13
Slide 13 text
UUID ࠾༻Ͱ͖ͳ͍ !
Slide 14
Slide 14 text
ʮFacebook, Twitter, Instagram͕Ͳ͏ͬͯIDΛੜ͍ͯ͠
Δͷ͔ ·ͱΊ1ʯΛࢀߟʹݕ౼
1 http://qiita.com/daisy1754/items/98a6e6b17d8161eab081
Slide 15
Slide 15 text
Twitter
Snowflake github.com/twitter/snowflake
λΠϜελϯϓΛݩʹੜͨ͠ 64bit
Instagram
Snowflake like ͳ 64bit ID Λ PL/PGSQL Ͱ
Flickr
MySQL ͷ Auto Increment Λ2༻ҙ
ͦΕͧΕحͱۮ୲Ͱ +2 ͍ͯ͘͠
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
݁ = Snowflake ํࣜ
+------------------------------------------------------+
| 64 bit |
+--+-----------------+----------------+----------------+
| 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit |
+--+-----------------+----------------+----------------+
Timestamp : Epoch ͔Βͷܦա࣌ؒ(msec)
WorkerID : ΫϥελͰҰҙͷ ID
Sequence : ಉҰ Timestamp ͷ࿈൪
Slide 18
Slide 18 text
݁ = Snowflake ํࣜ + ಠࣗσʔϞϯ
ಠࣗσʔϞϯ
! Go Ͱ࣮͠ɺ1ϗετʹ1ϓϩηεىಈ͢Δ
! ଟݴޠɺϛυϧΣΞ͔Β࠾൪Մೳʹ
௨৴ϓϩτίϧ memcached text protocol
! ܰྔɾ࣮͕؆୯ (GET ͔͠͠ͳ͍ͷͰ)
! ֤छݴޠͰ͑Δ
! PerlͷΫϥΠΞϯτ͕͍ (Cache::Memcached::Fast)
Slide 19
Slide 19 text
katsubushi ੜ
github.com/kayac/go-katsubushi
໊લͷ༝དྷ
Lobiͱ͍͑͵͜͞Μ
➡ Snowflake (ઇͷ݁থɺࡉ͔͍)
➡ ͵͜ Ͷ͜ ͔͚Β ࡉ͔͍
➡ અʂ
Slide 20
Slide 20 text
katsubushi ͷಛੑͱੑೳ
Slide 21
Slide 21 text
ID ͷߏ
329508644217425920 (10ਐදݱ)
+-+-----------------------------------------+----------+------------+
| | Timestamp | WorkerID | Sequence |
+-+-----------------------------------------+----------+------------+
|0|00001001001001010100110010111011011100100|0000000001|000000000000|
+-+-----------------------------------------+----------+------------+
| | 78,560,982,756ms since epoch | 1| 0|
+-+-----------------------------------------+----------+------------+
= 2017-06-28T06:29:42.756Z
bitͷ Snowflake ͱಉ༷
Epoch 2015-01-01T00:00:00Z (SnowflakeͱҟͳΔ)
࣌ܥྻͰঢॱɺID ͔Βൃߦ࣌ࠁͷٯࢉ͕Մೳ (msecਫ਼)
Slide 22
Slide 22 text
ID ͍ͭ·Ͱ͑Δͷ͔
Timestamp 41bit
2015-01-01T00:00:00 UTC + 0b1{41} msec
=> 2084-09-06T15:47:35.551 UTC
69ؒ
Slide 23
Slide 23 text
1ඵؒʹൃߦͰ͖ΔID
1ms ͋ͨΓ 12bit (4096) ͷ Sequence Λͭ
= 4,096,000/sec (ཧ)
WorkerID 10bit (1024) ͳͷͰΫϥελશମͰ
4,194,304,000 (≒32bit)
ຖඵ32bitਐΉ
Slide 24
Slide 24 text
࣮ࡍͷੑೳ
$ go test -bench Generate
BenchmarkGenerateID-4 5000000 244 ns/op
1ID ੜʹ 244ns
244 ns x 4,096,000 ≒ 1 sec
Go ෦Ͱͷൃ൪ཧݶք·ͰՄೳ
(Sequence ͕ 4095 Λ͑Δͱ࣍ͷ Timestamp ·Ͱ sleep)
Slide 25
Slide 25 text
ωοτϫʔΫαʔόʔͱͯ͠ͷੑೳ
Client: Go katsubushi.Client on AWS c4.8xlarge (36 cores)
Server: katsubushi v1.3.0 on AWS c3.2xlarge (8 cores)
ಉҰAZͷผϗετͰܭଌ
ฒྻ fetch=1 fetch=10 fetch=100
1 5,567 52,003 258,077
10 52,936 434,977 1,190,018
100 151,188 826,200 1,164,833
Ұʹ 100 ID Λൃߦ͢Ε 100ສ/sec Ҏ্
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
katsubushi ͷ͍͔ͨ
Slide 28
Slide 28 text
Install
Docker
$ docker pull katsubushi/katsubushi
$ docker run -p 11212:11212 katsubushi/katsubushi
Release binary
github.com/kayac/go-katsubushi/releases
Slide 29
Slide 29 text
katsubushi ͷ͍͔ͨ
worker-id Λࢦఆͯ͠ىಈ͢Δͱ TCP 11212 Λ Listen
$ katsubushi -worker-id 1
2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:152 Listening at [::]:11212
2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:153 Worker ID = 1
Docker ͷ߹ڥมͰ
$ docker run -p 11212:11212 -e worker_id=2 katsubushi/katsubushi
2017-06-29T05:58:39.847Z INFO go-katsubushi/app.go:152 Listening at [::]:11212
2017-06-29T05:58:39.848Z INFO go-katsubushi/app.go:153 Worker ID = 2
Slide 30
Slide 30 text
katsubushi ͷ͍͔ͨ
nc ίϚϯυͰखͰͬͯΈ·͠ΐ͏
GET ͷ key ҙͷͰ OK
$ nc localhost 11212
GET id
VALUE id 0 18
329508644217425920
END
Slide 31
Slide 31 text
katsubushi ͷ͍͔ͨ
ෳIDΛ·ͱΊͯऔಘͰ͖·͢
GET id1 id2
VALUE id1 0 18
329508654694797312
VALUE id2 0 18
329508654694797313
END
Slide 32
Slide 32 text
katsubushi ͷ͍͔ͨ
ී௨ͷ memcached client ͕͑·͢
(ςΩετϓϩτίϧͷΈ)
use Cache::Memcached;
$c = Cache::Memcached->new( servers => ["127.0.0.1:11212"] );
$c->get("id"); #= 329521618642538496
Perl Cache::Memcached, C::M::Fast, C::M::AnyEvent
Go github.com/bradfitz/gomemcache
Ruby memcached(gem)
Slide 33
Slide 33 text
katsubushi ؔ࿈ϥΠϒϥϦ
Slide 34
Slide 34 text
Perl - ΫϥΠΞϯτϥΠϒϥϦ
metacpan.org/pod/Katsubushi::Client
use Katubushi::Client;
my $client = Katsubushi::Client->new({
servers => ["127.0.0.1:11212", "10.8.0.1:11212"],
});
my $id = $client->fetch;
my @ids = $client->fetch_multi(3);
serversʹࢦఆͨ͠ϗετͷઌ಄͔Βऔಘ
ࣦഊͨ͠Β࣍ͷϗετ
Slide 35
Slide 35 text
Perl - ID ͱ UNIX time ͷ૬ޓม
use Katsubushi::Converter
# ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ
my $now = time();
my $id = Katsubushi::Converter::epoch_to_id($now);
# ID͕ൃߦ͞Εͨ࣌ࠁΛunix timeͰऔಘ
my $t = Katsubushi::Converter::id_to_epoch($id);
katsubushi2epoch, epoch2katsubushi ίϚϯυೖΔ
Slide 36
Slide 36 text
Go - ΫϥΠΞϯτϥΠϒϥϦ
godoc.org/github.com/kayac/go-katsubushi#Client
import "github.com/kayac/go-katsubushi"
func main() {
client := katsubushi.NewClient("127.0.0.1:11212", "10.8.0.1:11212")
id, _ := client.Fetch() // uint64, error
ids, _ := client.FetchMulti(3) // []uint64, error
}
ෳϗετͷࢼߦॱ Perl ൛ͱಉ͡
Slide 37
Slide 37 text
Go - ID ͱ time.Time ͷ૬ޓม
godoc.org/github.com/kayac/go-katsubushi#ToID
// ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ
t1 := time.Now()
id := katsubushi.ToID(t1)
// ID͕ൃߦ͞Εͨ࣌ࠁΛ time.Time Ͱฦ͢
t2 := katsubushi.ToTime(id)
Slide 38
Slide 38 text
fluent-plugin-katsubushi
rubygems.org/gems/fluent-plugin-katsubushi
Fluentd ͷϑΟϧλʔϓϥάΠϯ
ϑΟϧλʔΛ௨ա͢Δϩάʹ katsubushi Ͱൃߦͨ͠ ID Λೖ
@type katsubushi
id_key id # ೖ͢Δkey໊
host localhost # katsubushi ͷϗετ໊
port 11212 # katsubushi ͷϙʔτ൪߸
Slide 39
Slide 39 text
fluent-plugin-katsubushi
fluent-plugin-katsubushi Λ௨ա͢Δͱ ID ͕ೖ͞ΕΔ
{"message":"foo"}
!
{"message":"foo","id":329518490450919424}
ϩάʹ unique ͳ ID ΛৼΓ͍ͨʂ
͚ͲൃߦՕॴͰৼΕͳ͍ʂͱ͍͏߹ʹ
Slide 40
Slide 40 text
katsubushi ͷӡ༻
Slide 41
Slide 41 text
worker-id ΛҰҙʹ͢Δ
ӡ༻͍ͯ͠Δશͯͷ katsubushi process ͷ
worker-id ΛҰҙʹ͢Δͷ͕ඞਢ
worker-id ॏෳ ➡ ಉҰ࣌ࠁʹൃߦ͞ΕͨID͕ॏෳ͢Δ ➡ ഁ໓
+------------------------------------------------------+
| 64 bit |
+--+-----------------+----------------+----------------+
| 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit |
+--+-----------------+----------------+----------------+
Slide 42
Slide 42 text
worker-id ΛҰҙʹ͢Δ
worker-id 1ʙ1023 ͷ
ݻఆαʔόͰಈ͔͢ͳΒదʹखͰ͚ͭΔ
ಈతʹαʔό͕ىಈ͢ΔڥͰ…ʁ
ͨͱ͑ IP ΞυϨεͷୈ 3,4 octet Λ͏
192.168.0.10 ➡ 0 * 256 + 10 = 10
192.168.1.20 ➡ 1 * 256 + 20 = 266
192.168.2.30 ➡ 2 * 256 + 30 = 542
Slide 43
Slide 43 text
worker-id ΛҰҙʹ͢Δ
ίϯςφڥͰ…ʁ
Redis Λͬͨ worker-id ࣗಈׂΓͯػೳ (ޙड़)
Slide 44
Slide 44 text
stats
memcached ͷ stats ίϚϯυΛ࣮
$ echo "STATS" | nc localhost 11212
STAT pid 1
STAT uptime 51014
STAT time 1498716107
STAT version 1.3.0
STAT curr_connections 1
STAT total_connections 8
STAT cmd_get 18
STAT get_hits 20
STAT get_misses 0
END
Slide 45
Slide 45 text
key ͷྫ ҙຯ
pid 1 Process ID
uptime 51014 ىಈ͔ͯ͠Βͷܦաඵ
time 1498716107 ݱࡏ࣌ࠁ UNIX time
version 1.3.0 όʔδϣϯ
curr_connections 1 ݱࡏଓ͍ͯ͠ΔTCPίωΫγϣϯ
total_connections 8 ىಈ͔ͯ͠Βଓͨ͠ίωΫγϣϯ
(ྦྷܭ)
cmd_get 18 GET ίϚϯυΛॲཧͨ͠ճ (ྦྷܭ)
get_hits 20 ൃߦͨ͠ ID (ྦྷܭ)
get_misses 0 ൃߦͰ͖ͳ͔ͬͨ ID (ྦྷܭ)
Slide 46
Slide 46 text
ࢹ͖͢
key
curr_connections ଟ͗͢ͳ͍͔
uptime ר͖͍ͬͯΔ = ࠶ىಈͯ͠
͍Δ
get_misses ൃੜ = ࣌ࠁ͕ר͖͍ͬͯΔ
Slide 47
Slide 47 text
࣌ࠁר͖Γ࣌ͷରԠ
katsubushi ID ͷҰҙੑ࣌ࠁʹґଘ
࣌ࠁ͕ਐΉͷͳ͍͕ɺר͖Δͱ…(ͨͱ͑Ӟඵ)
աڈʹൃߦͨ͠ ID ͱಉ͡ ID Λൃߦͯ͠͠·͏Մೳੑ
➡ ίʔυͰר͖Βͳ͍Α͏ʹอޢ
Slide 48
Slide 48 text
ҎલͷλΠϜελϯϓ͕དྷͨΒΤϥʔ
func (g *Generator) NextID() (uint64, error) {
g.lock.Lock()
defer g.lock.Unlock()
ts := g.timestamp()
// for rewind of server clock
if ts < g.lastTimestamp {
return 0, errors.New("system clock was rollbacked")
}
Slide 49
Slide 49 text
Go 1.9 ͷ monotonic time Λ͏(༧ఆ)
Go 1.9 Ͱ monotonic time ͕ಋೖ͞ΕΔ 3
ىಈ࣌ʹ time.Now() ͨ࣌͠ࠁ͔ΒͷࠩΛར༻͢Ε
OS࣌ࠁ͕ר͖ͬͯӨڹΛड͚ͳ͍
3 https://tip.golang.org/doc/go1.9#monotonic-time
Slide 50
Slide 50 text
ఀࢭ͍ͯ͠Δؒʹ࣌ࠁ͕ͬͨΒ…
katsubushi ϓϩηεͰݕ͠Α͏͕ͳ͍
ରԠࡦɿࣗಈͰ࠶ىಈ͠ͳ͍
Perl / Go ͷΫϥΠΞϯτଞͷϗετʹ fallback Ͱ͖Δ
ઃఆͯ͋͠Δશͯͷϗετ͕ಉ࣌ʹམͪͳ͚Ε ID औಘͰ͖Δ
Slide 51
Slide 51 text
҆ఆੑ
2015ʹϦϦʔε͔ͯ͠Β2ؒ
Lobiͷຊ൪ڥͷ΄΅શͯͷϗετͰՔಇ͍ͯ͠Δ͕
௨ৗར༻Ͱϓϩηε͕མͪͨ͜ͱͳ͍ 2
ϝϞϦϦʔΫܦݧͳ͠
2 ͨͩ͠ memcached protocolͰਖ਼͘͠ΞΫηεͨ͠߹ɻෆਖ਼ͳίϚϯυͰΫϥογϡͨ͜͠ͱ͋ͬͨ github.com/kayac/
go-katsubushi/pull/8
Slide 52
Slide 52 text
WebΞϓϦέʔγϣϯͷԠ༻
Slide 53
Slide 53 text
Request ID ʹΑΔ
ΞϓϦέʔγϣϯ
Slide 54
Slide 54 text
Request ID Λ katsubushi Ͱൃߦ
nginx͕ϦΫΤετΛड͚͚ͨ࣌Ͱ katsubushi ID Λൃߦ
ޙॲཧʹҾ͖͢͜ͱͰ֤छॲཧͷΛ༰қʹ
nginx ͱ katsubushi ͕ memcached protocol Ͱ௨৴
Slide 55
Slide 55 text
IDΛҾ͖ճͯ͠
ϩάʹه
Slide 56
Slide 56 text
katsubushi ➡ nginx
ΞΫηεຖʹ nginx lua ͕ ID Λऔಘͯ͠ม $req_id ʹઃఆ
upstream katsubushi {
server 127.0.0.1:11212;
keepalive 10;
}
location /katsubushi {
internal;
set $memcached_key "id"; # keyͳΜͰOK
memcached_pass katsubushi;
}
set $req_id 0;
access_by_lua_block {
local res = ngx.location.capture("/katsubushi")
if res then
ngx.var.req_id = res.body
end
}
Slide 57
Slide 57 text
nginx Ͱϩάʹग़ྗ
log_format ltsv
'req_id:$req_id\t'
'host:$remote_addr\t'
'user:$remote_user\t'
'time:$time_iso8601\t'
'method:$request_method\t'
...
Slide 58
Slide 58 text
nginx ➡ App(Plack)
ϦΫΤετϔομ X-Request-ID Λઃఆ
upstream app {
server unix:/var/tmp/app.socket;
}
location / {
proxy_set_header X-Request-ID $req_id;
proxy_pass http://app;
}
Slide 59
Slide 59 text
App(Plack) ➡ local %ENV
Plack::Middleware::SetLocalEnv
Plack ͷ $env ͔Β local %ENV ʹΛઃఆ͢Δ
ΞϓϦέʔγϣϯͷͲ͔͜ΒͰ $ENV{REQUEST_ID} ͰࢀরՄ
# app.psgi
use Plack::Builder;
builder {
enable "SetLocalEnv",
REQUEST_ID => "HTTP_X_REQUEST_ID",
;
# ...
$app;
}
Slide 60
Slide 60 text
Plack::Middleware::SetLocalEnv
࣮͜Ε͚ͩ
local ͳͷͰείʔϓΛൈ͚Δͱ(ϦΫΤετॲཧ͕ऴΘΔͱ)
͕ݩʹΔ
sub call {
my ($self, $env) = @_;
local %ENV = %ENV;
for my $key (keys %$self) {
$ENV{$key} = $env->{ $self->{$key} }
if exists $env->{$self->{$key}};
}
$self->app->($env);
}
Slide 61
Slide 61 text
App(Plack) ➡ app.log (Log::Minimal)
$Log::Minimal::(PRINT|DIE) Ͱग़ྗΛΧελϚΠζ
use Log::Minimal;
BEGIN {
$Log::Minimal::PRINT = sub {
my ($time, $type, $message, $trace) = @_;
my $req_id = $ENV{REQUEST_ID} // 0;
warn "[$$][$req_id][$type] $message at $trace\n";
};
}
warnf("something bad!");
[64302][329733861698727936][WARN] something bad! at .../App.pm line XX
Slide 62
Slide 62 text
App(Plack) ➡ Fluentd
ϑΝΠϧΛհͣ͞ Fluent::Logger Ͱૹ৴͢Δ߹
ϝοηʔδʹ req_id Λೖ
package Logger;
use Fluent::Logger;
sub log {
my ($self, $tag, $msg) = @_;
$msg->{req_id} = $ENV{REQUEST_ID} // 0;
$logger->post($tag, $msg); # $logger is Fluent::Logger
}
Slide 63
Slide 63 text
App(Plack) ➡ MySQL
ΫΤϦͷઌ಄ʹSQLίϝϯτͰ /* req_id=... */ ΛຒΊࠐΉ
use parent 'Teng::Sharding';
around ["execute", "do"] => sub {
my ($orig, $self, $sql, @rest) = @_;
my $req_id = $ENV{REQUEST_ID} // 0;
return $orig->($self, "/* req_id=${req_id} */\n${sql}", @rest);
};
# Time: 170629 3:27:47
# User@Host: xxx[xxx] @ [192.0.2.1]
# Query_time: 1.080420 Lock_time: 0.000040 Rows_sent: 5 Rows_examined: 909850
SET timestamp=1498674467;
/* req_id=329689349781536768 */
SELECT ..... FROM ...
Slide 64
Slide 64 text
͍ΖΜͳىͰௐ͕ࠪͰ͖Δ
ΞΫηεϩάͰ status:500
➡ ΞϓϦͷΤϥʔϩάͱর߹
slow.log ʹࠔͬͨΫΤϦ͕ه͞Ε͍ͯΔ
ඇಉظ job ͕ࣦഊ
➡ ൃੜݩʹͳͬͨHTTPϦΫΤετΛΞΫηεϩάͰಛఆ
Slide 65
Slide 65 text
ଞܗࣜͷ Request ID ͱͷൺֱ
$request_id (nginx >= 1.11.0)
unique request identifier generated from 16 random bytes, in
hexadecimal 4
ྫ: 9f53d64cf47dd966a7462f5a737b5ade
͍
࣌ࠁใؚ͕·Εͳ͍
4 http://nginx.org/en/docs/http/ngxhttpcoremodule.html#varrequest_id
Slide 66
Slide 66 text
࣌ͰύʔςΟγϣχϯάΛߦ͏
ςʔϒϧͷϓϥΠϚϦΩʔͱͯ͠͏
Slide 67
Slide 67 text
ҰఆظؒͰݹ͍σʔλΛফ͍ͨ͠ςʔϒϧ(௨)
Redis ʹ id ͷΈอ࣋ͯ͠ index ʹ͍ͯ͠Δ
id auto_increment, created_date ͷෳ߹Primary Key
CREATE TABLE `notification` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`created_date` datetime NOT NULL,
/* other columns */
PRIMARY KEY (`id`,`created_date`)
) ENGINE=InnoDB AUTO_INCREMENT=xxxxx DEFAULT CHARSET=utf8
PARTITION BY RANGE COLUMNS(created_date)
(PARTITION p20170602 VALUES LESS THAN ('2017-06-02') ENGINE = InnoDB,
PARTITION p20170603 VALUES LESS THAN ('2017-06-03') ENGINE = InnoDB,
PARTITION p20170604 VALUES LESS THAN ('2017-06-04') ENGINE = InnoDB,
.....
Slide 68
Slide 68 text
id + datetime ύʔςΟγϣχϯάͷ
idͷΈͰݕࡧ͢ΔΫΤϦͰמΓࠐΊͳ͍ͨΊ
શύʔςΟγϣϯΛݕࡧ͢Δ
(30ͨ͢Ίʹ30 partitionΔͱ30ഒͷݕࡧίετ)
➡ Α͘Δ WHERE id IN (?,?,....) ͕͍
➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍ͣͷύʔςΟγϣϯʹ
සൟʹࢀর͕Δ (InnoDB Buffer Poolͷແବ)
Slide 69
Slide 69 text
! katsubushi ID Ͱ RANGE
ύʔςΟγϣϯΛΕ…
Slide 70
Slide 70 text
katsubushi ID ͷΈͰ1୯ҐͷύʔςΟγϣχϯά
CREATE TABLE `notification` (
`id` bigint(20) unsigned NOT NULL, /* AUTO_INCREMENT ͠ͳ͍ */
`created_date` datetime NOT NULL, /* ෆཁ͔͠Εͳ͍ */
/* other columns */
PRIMARY KEY (`id`) /* pkey id ͷΈ */
) ENGINE=InnoDB DEFAULT CHARSET=utf8
PARTITION BY RANGE COLUMNS(id)
(PARTITION p20170602 VALUES LESS THAN (319852589875200000) ENGINE = InnoDB,
/* ^ 2017-06-02 00:00:00 JST */
PARTITION p20170603 VALUES LESS THAN (320214977740800000) ENGINE = InnoDB,
/* ^ 2017-06-03 00:00:00 JST */
PARTITION p20170604 VALUES LESS THAN (320577365606400000) ENGINE = InnoDB,
/* ^ 2017-06-04 00:00:00 JST */
.....
Slide 71
Slide 71 text
katsubushi ID only ύʔςΟγϣχϯάͷར
id ͷΈͷݕࡧΫΤϦ͕ࣗಈతʹמΓࠐ·ΕΔ
➡ Α͘Δ WHERE id IN (?,?,....) ͕࠷దԽ͞ΕΔ
➡ Row Reads ܹݮ
➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍ύʔςΟγϣϯʹ
ࢀর͕Βͳ͍ (InnoDB Buffer PoolͷޮUp)
Slide 72
Slide 72 text
katsubushi ID only ύʔςΟγϣχϯάͷܽ?
ͲͷΛڥʹΕ͍͍ͷ͔͕ײతͰͳ͍
! Katsubushi::Converter Ͱܭࢉ
ಛఆͷ͚ͩࢀর͍ͨ͠
! ͷൣғΛ ID ʹมͯ͠ WHERE ۟ʹೖΕΕOK
Slide 73
Slide 73 text
ίϯςφڥͷదԠ
Slide 74
Slide 74 text
ίϯςφڥͰ worker-id ΛҰҙʹ
͢Δͷ͕͍͠!
Slide 75
Slide 75 text
ίϯςφڥͰ worker-id ΛҰҙʹ͢Δ
❌ ਓྗͰͷݻఆׂΓͯͰ͖ͳ͍
ੜ·Εͯফ͑Δͷ͕ίϯςφ
❌ IPΞυϨεΛ͏͜ͱͰ͖ͳ͍
ϗετͱNAT͍ͯ͠ΔͱɺผϗετͰಉҰͷ IP ΞυϨεΛ
͍ͬͯΔίϯςφ͕ଘࡏ͠͏Δ
Ͳ͏͢Ε…
Slide 76
Slide 76 text
Redis ͰഉଞతʹΛऔಘͤ͞Δ
͙͢ࢥ͍ͭ͘ͷू߹ܕͰ SPOP
> SADD id_pool 1 2 3 4 # 4ཁૉͷू߹Λ࡞Δ
> SPOP id_pool # ू߹͔ΒϥϯμϜʹҰͭऔΓग़͢
"3"
> SMEMBERS id_pool # Γ
"1"
"2"
"4"
Simple is best?
Slide 77
Slide 77 text
SPOP ํࣜͷ
1. ϓϩηεىಈ࣌ʹ SPOP Ͱऔಘ
2. ϓϩηεऴྃ࣌ʹ SADD Ͱฦ٫
औΓग़͞Ε͕ͨฦ٫͞ΕΔ͜ͱ͕อূͰ͖ͳ͍
ϓϩηεਖ਼ৗऴྃ͢ΔͱݶΒͳ͍
➡ ظؒӡ༻͍ͯ͠Δͱฦ٫͞Εͳ͍͕
ཷ·͍ͬͯͬͯރׇ͢ΔڪΕ
Slide 78
Slide 78 text
ϝϯςφϯεϑϦʔͳํ๏Λߟ͍͑ͨ
Slide 79
Slide 79 text
raus
Slide 80
Slide 80 text
raus
github.com/fujiwara/raus
"RAnged Unique id Supplier"
RedisΛͬͯҰఆൣғͷIDΛ
ॏෳͳׂ͘ΓͯΔͨΊͷϥΠϒϥϦ
Slide 81
Slide 81 text
PubSub + SET NX + GETSET + EXPIRE
1. ੜ͖͍ͯΔϓϩηε͕ PubSub channel ʹอ͍࣋ͯ͠ΔIDΛ
ྲྀ͢
2. อ͍࣋ͯ͠ΔIDͷ Key Λ GETSET + EXPIRE Ͱߋ৽͠ଓ͚Δ
3. ৽نϓϩηε PubSub channel Λ؍ͯ͠ɺΘΕ͍ͯͳ
ͦ͏ͳIDΛ SET NX Ͱऔಘ
4. औಘͰ͖ͨΒ PubSub + GETSET + EXPIRE Λଓ͚Δ
Slide 82
Slide 82 text
1. IDΛอ͍࣋ͯ͠Δطଘ raus PUBLISH + GETSET Λఆظతʹߦ͏
Slide 83
Slide 83 text
2. ৽ن raus SUBSCRIBE ͯ͠ଞͷ raus ͕͍ͬͯΔIDΛѲ
Slide 84
Slide 84 text
3. ৽ن raus ଞͷ raus ͕ͬͯͳ͍(ͣͷ) key Λ SET NX Ͱഉଞऔಘ
Slide 85
Slide 85 text
4. ID ΛऔಘͰ͖ͨΒ PUBLISH + GETSET Λఆظ࣮ߦ
Slide 86
Slide 86 text
raus ͕ϝϯςφϯεϑϦʔͳཧ༝
Slide 87
Slide 87 text
raus ͕ϝϯςφϯεϑϦʔͳཧ༝
• ࢮΜͩϓϩηε GETSET ͠ͳ͍ͷͰ key ͕࣌ݶ։์͞ΕΔ
• ૄ௨Ͱ͖ͳ͘ͳͬͯҰఆ࣌ؒ key ͕อ࣋͞Ε͍ͯΔͷͰ
ଞ͕औಘͰ͖ͳ͍
• ૄ௨͕ճ෮ͨ͠ޙʹGETSETʹࣦഊͨ͠
(=ଞͷϓϩηε͕։์͞Εͨ key Λऔಘͯ͠͠·ͬͨ)߹
ଈ࠲ʹࢮ͵
Slide 88
Slide 88 text
katsubushi + raus
$ katsubushi -redis redis://192.0.2.2:6379
Waiting for worker-id automated assignment (between 1 and 1023) with redis://192.0.2.2:6379
xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 <= ଞͷkatsubushi͕woker_id=2Λอ͍࣋ͯ͠Δ
xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 ͷΛsubscribeͰ؍͍ͯ͠Δ
xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2
xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2
candidate ids: [1 3 4 5 6 7 8 9 10 11] <= 2 Ҏ֎ͷΛީิʹͯͦ͠ͷத͔ΒϥϯμϜʹ
trying to get lock key raus:id:8 <= SET NX
got lock for 8 <= ޭͨ͠ͷͰ 8 Λऔಘ
Assigned worker-id: 8
Listening at [::]:11212
Worker ID = 8
<= PUBLISH + GETSET ΛཪͰଓ͚͍ͯΔ
Slide 89
Slide 89 text
katsubushi + raus
worker_id=8 Λऔಘͨ͠ katsubushi ͕ಈ͍͍ͯΔঢ়ଶͰ
key Λ্ॻ͖ͯ͠ΈΔͱ…
$ redis-cli set raus:id:8 xxx
unexpected uuid got: xxx
panic: unexpected uuid got: xxx
goroutine 42 [running]:
main.assignWorkerID.func1(0xc4201462c0, 0xc42006f020)
src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:230 +0xb0
created by main.assignWorkerID
src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:235 +0x4fe
Slide 90
Slide 90 text
·ͱΊ
Slide 91
Slide 91 text
·ͱΊ
• Snowflake ͱಉ༷ͷΞϧΰϦζϜͰࢄڥͰϢχʔΫͳID
ΛൃߦͰ͖Δ katsubushi
• ܰྔͳͷͰ DB ͷߦ ID Ҏ֎ʹར༻Մೳ
➡ Request ID Ͱॲཧ
• ࣌ࠁؚ͕·ΕΔಛੑΛੜ͔ͯ͠ύʔςΟγϣχϯάʹ
• ίϯςφڥʹରԠ͢Δ Redis ΛͬͨࣗಈΫϥελϦϯά