Upgrade to Pro — share decks privately, control downloads, hide ads and more …

分散ユニークID採番機 katsubushi と Web アプリケーションへの応用例 / katsubushi

分散ユニークID採番機 katsubushi と Web アプリケーションへの応用例 / katsubushi

YAPC::Fukuoka

FUJIWARA Shunichiro

July 01, 2017
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. ෼ࢄϢχʔΫID࠾൪ػ
    katsubushi ͱ
    Web ΞϓϦέʔγϣϯ
    ΁ͷԠ༻ྫ
    2017.07.01 YAPC::Fukuoka
    @fujiwara

    View full-size slide

  2. @fujiwara
    ٕज़෦
    github.com/fujiwara
    sfujiwara.hatenablog.com

    View full-size slide

  3. Game & Community

    View full-size slide

  4. katsubushi - ෼ࢄϢχʔΫID࠾൪ػ
    != katsuobushi
    != katsubushi

    View full-size slide

  5. katsubushi - ෼ࢄϢχʔΫID࠾൪ػ
    ෼ࢄ = ୯Ұͷ (host | node | process) ʹґଘ͠ͳ͍
    ϢχʔΫ = ෳ਺ͷ katsubushi Ͱൃߦͨ͠ ID ͕ॏෳ͠ͳ͍
    Lobi Ͱͷ DB γϟʔσΟϯάͷཁ੥ʹ൐ͬͯ஀ੜ

    View full-size slide

  6. Lobi Ͱͷ DB γϟʔσΟϯά
    MySQL Ͱී௨ͷ ୯Ұ master ෳ਺ slave ߏ੒ (ʙ2014)
    σʔλɺΞΫηε૿ՃʹΑΓ୯Ұ master ͕ݶքʹ
    (Ұ࣌తʹ i2.8xlarge Πϯελϯεʹͯ࣌ؒ͠Ք͗… !!!)
    ➡ ΍ΉΛಘͣɺDB Λ෼ׂ͢Δ͜ͱʹ (2015.02)
    ڞ௨DB(ϢʔβͳͲ) x 1
    άϧʔϓνϟοτؔ࿈DB x 2 ←ඞཁʹͳΕ͹૿΍ͤΔ

    View full-size slide

  7. DB γϟʔσΟϯά࣌ͷ՝୊
    ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑
    ID࠾൪
    σʔλҠߦɺ੾Γସ͑ (ຊτʔΫͰ͸লུ)

    View full-size slide

  8. DB γϟʔσΟϯά࣌ͷ՝୊
    ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑

    View full-size slide

  9. ΞϓϦέʔγϣϯίʔυͷॻ͖׵͑
    άϧʔϓ͕Ͳͷγϟʔυʹଐ͢Δ͔ͷmapΛڞ௨DBʹ࣋ͭ
    ! γϟʔυҠಈ͠ͳ͚Ε͹ΩϟογϡՄೳ
    ΫΤϦʹؚ·ΕΔςʔϒϧͱΧϥϜͷ஋ΛΈͯ…
    ! γϟʔυΛಛఆͰ͖Ε͹1γϟʔυʹ౤͛Δ
    ! Ͱ͖ͳ͚Ε͹ෳ਺γϟʔυʹ౤͛ͯ݁ՌΛ·ͱΊΔ
    Teng ʹຐվ଄Λࢪͨ͠ Teng::Sharding Λ࡞੒(ඇެ։…)

    View full-size slide

  10. DB γϟʔσΟϯά࣌ͷ՝୊
    ID ࠾൪

    View full-size slide

  11. ID ͷཁ݅
    ֤γϟʔυͰॏෳ͠ͳ͍ϢχʔΫͳIDΛৼΓ͍ͨʂ
    64bitҎԼͷ੔਺஋
    ! طʹ਺஋ͱͯ͠ѻ͏લఏͷίʔυ͕ࢁ΄Ͳ͋Δ
    ! DB ʹೖΕΔʹ͸ 64 bit ҎԼ͕Α͍
    ࣌ܥྻͷঢॱʹͳΔ͜ͱ (࣮༻্໰୊ͳ͍ਫ਼౓Ͱ)
    ! ࣌ܥྻͰιʔτ͢ΔͨΊʹ ORDER BY id (DESC) ͍ͯ͠Δ

    View full-size slide

  12. UUID ͸࠾༻Ͱ͖ͳ͍ !

    View full-size slide

  13. ʮFacebook, Twitter, Instagram౳͕Ͳ͏΍ͬͯIDΛੜ੒͍ͯ͠
    Δͷ͔ ·ͱΊ1ʯΛࢀߟʹݕ౼
    1 http://qiita.com/daisy1754/items/98a6e6b17d8161eab081

    View full-size slide

  14. Twitter
    Snowflake github.com/twitter/snowflake
    λΠϜελϯϓΛݩʹੜ੒ͨ͠ 64bit ੔਺
    Instagram
    Snowflake like ͳ 64bit ID Λ PL/PGSQL Ͱ
    Flickr
    MySQL ͷ Auto Increment Λ2୆༻ҙ
    ͦΕͧΕح਺ͱۮ਺୲౰Ͱ +2 ͍ͯ͘͠

    View full-size slide

  15. ݁࿦ = Snowflake ํࣜ
    +------------------------------------------------------+
    | 64 bit |
    +--+-----------------+----------------+----------------+
    | 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit |
    +--+-----------------+----------------+----------------+
    Timestamp : Epoch ͔Βͷܦա࣌ؒ(msec)
    WorkerID : Ϋϥελ಺ͰҰҙͷ ID
    Sequence : ಉҰ Timestamp ಺ͷ࿈൪

    View full-size slide

  16. ݁࿦ = Snowflake ํࣜ + ಠࣗσʔϞϯ
    ಠࣗσʔϞϯ
    ! Go Ͱ࣮૷͠ɺ1ϗετʹ1ϓϩηεىಈ͢Δ
    ! ଟݴޠɺϛυϧ΢ΣΞ͔Β࠾൪Մೳʹ
    ௨৴ϓϩτίϧ͸ memcached text protocol
    ! ܰྔɾ࣮૷͕؆୯ (GET ͔͠͠ͳ͍ͷͰ)
    ! ֤छݴޠͰ࢖͑Δ
    ! PerlͷΫϥΠΞϯτ͕଎͍ (Cache::Memcached::Fast)

    View full-size slide

  17. katsubushi ஀ੜ
    github.com/kayac/go-katsubushi
    ໊લͷ༝དྷ
    Lobiͱ͍͑͹͵͜͞Μ
    ➡ Snowflake (ઇͷ݁থɺࡉ͔͍)
    ➡ ͵͜ Ͷ͜ ͔͚Β ࡉ͔͍
    ➡ ׍અʂ

    View full-size slide

  18. katsubushi ͷಛੑͱੑೳ

    View full-size slide

  19. ID ͷߏ଄
    329508644217425920 (10ਐදݱ)
    +-+-----------------------------------------+----------+------------+
    | | Timestamp | WorkerID | Sequence |
    +-+-----------------------------------------+----------+------------+
    |0|00001001001001010100110010111011011100100|0000000001|000000000000|
    +-+-----------------------------------------+----------+------------+
    | | 78,560,982,756ms since epoch | 1| 0|
    +-+-----------------------------------------+----------+------------+
    = 2017-06-28T06:29:42.756Z
    bitͷ഑෼͸ Snowflake ͱಉ༷
    Epoch 2015-01-01T00:00:00Z (Snowflakeͱ͸ҟͳΔ)
    ࣌ܥྻͰঢॱɺID ͔Βൃߦ࣌ࠁͷٯࢉ͕Մೳ (msecਫ਼౓)

    View full-size slide

  20. ID ͸͍ͭ·Ͱ࢖͑Δͷ͔
    Timestamp 41bit
    2015-01-01T00:00:00 UTC + 0b1{41} msec
    => 2084-09-06T15:47:35.551 UTC
    ໿69೥ؒ

    View full-size slide

  21. 1ඵؒʹൃߦͰ͖ΔID਺
    1ms ͋ͨΓ 12bit (4096) ͷ Sequence Λ΋ͭ
    = 4,096,000/sec (ཧ࿦஋)
    WorkerID 10bit (1024) ͳͷͰΫϥελશମͰ͸
    4,194,304,000 (≒32bit)
    ຖඵ໿32bitਐΉ

    View full-size slide

  22. ࣮ࡍͷੑೳ
    $ go test -bench Generate
    BenchmarkGenerateID-4 5000000 244 ns/op
    1ID ੜ੒ʹ 244ns
    244 ns x 4,096,000 ≒ 1 sec
    Go ಺෦Ͱͷൃ൪͸ཧ࿦஋ݶք·ͰՄೳ
    (Sequence ͕ 4095 Λ௒͑Δͱ࣍ͷ Timestamp ·Ͱ sleep)

    View full-size slide

  23. ωοτϫʔΫαʔόʔͱͯ͠ͷੑೳ
    Client: Go katsubushi.Client on AWS c4.8xlarge (36 cores)
    Server: katsubushi v1.3.0 on AWS c3.2xlarge (8 cores)
    ಉҰAZͷผϗετͰܭଌ
    ฒྻ౓ fetch=1 fetch=10 fetch=100
    1 5,567 52,003 258,077
    10 52,936 434,977 1,190,018
    100 151,188 826,200 1,164,833
    Ұ౓ʹ 100 ID Λൃߦ͢Ε͹ 100ສ/sec Ҏ্

    View full-size slide

  24. katsubushi ͷ࢖͍͔ͨ

    View full-size slide

  25. Install
    Docker
    $ docker pull katsubushi/katsubushi
    $ docker run -p 11212:11212 katsubushi/katsubushi
    Release binary
    github.com/kayac/go-katsubushi/releases

    View full-size slide

  26. katsubushi ͷ࢖͍͔ͨ
    worker-id Λࢦఆͯ͠ىಈ͢Δͱ TCP 11212 Λ Listen
    $ katsubushi -worker-id 1
    2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:152 Listening at [::]:11212
    2017-06-28T15:21:18.370+0900 INFO go-katsubushi/app.go:153 Worker ID = 1
    Docker ͷ৔߹͸؀ڥม਺Ͱ
    $ docker run -p 11212:11212 -e worker_id=2 katsubushi/katsubushi
    2017-06-29T05:58:39.847Z INFO go-katsubushi/app.go:152 Listening at [::]:11212
    2017-06-29T05:58:39.848Z INFO go-katsubushi/app.go:153 Worker ID = 2

    View full-size slide

  27. katsubushi ͷ࢖͍͔ͨ
    nc ίϚϯυͰखͰ஻ͬͯΈ·͠ΐ͏
    GET ͷ key ͸೚ҙͷ஋Ͱ OK
    $ nc localhost 11212
    GET id
    VALUE id 0 18
    329508644217425920
    END

    View full-size slide

  28. katsubushi ͷ࢖͍͔ͨ
    ෳ਺IDΛ·ͱΊͯऔಘ΋Ͱ͖·͢
    GET id1 id2
    VALUE id1 0 18
    329508654694797312
    VALUE id2 0 18
    329508654694797313
    END

    View full-size slide

  29. katsubushi ͷ࢖͍͔ͨ
    ී௨ͷ memcached client ͕࢖͑·͢
    (ςΩετϓϩτίϧͷΈ)
    use Cache::Memcached;
    $c = Cache::Memcached->new( servers => ["127.0.0.1:11212"] );
    $c->get("id"); #= 329521618642538496
    Perl Cache::Memcached, C::M::Fast, C::M::AnyEvent
    Go github.com/bradfitz/gomemcache
    Ruby memcached(gem)

    View full-size slide

  30. katsubushi ؔ࿈ϥΠϒϥϦ

    View full-size slide

  31. Perl - ΫϥΠΞϯτϥΠϒϥϦ
    metacpan.org/pod/Katsubushi::Client
    use Katubushi::Client;
    my $client = Katsubushi::Client->new({
    servers => ["127.0.0.1:11212", "10.8.0.1:11212"],
    });
    my $id = $client->fetch;
    my @ids = $client->fetch_multi(3);
    serversʹࢦఆͨ͠ϗετͷઌ಄͔Βऔಘ
    ࣦഊͨ͠Β࣍ͷϗετ΁

    View full-size slide

  32. Perl - ID ͱ UNIX time ͷ૬ޓม׵
    use Katsubushi::Converter
    # ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ
    my $now = time();
    my $id = Katsubushi::Converter::epoch_to_id($now);
    # ID͕ൃߦ͞Εͨ࣌ࠁΛunix timeͰऔಘ
    my $t = Katsubushi::Converter::id_to_epoch($id);
    katsubushi2epoch, epoch2katsubushi ίϚϯυ΋ೖΔ

    View full-size slide

  33. Go - ΫϥΠΞϯτϥΠϒϥϦ
    godoc.org/github.com/kayac/go-katsubushi#Client
    import "github.com/kayac/go-katsubushi"
    func main() {
    client := katsubushi.NewClient("127.0.0.1:11212", "10.8.0.1:11212")
    id, _ := client.Fetch() // uint64, error
    ids, _ := client.FetchMulti(3) // []uint64, error
    }
    ෳ਺ϗετͷࢼߦॱ͸ Perl ൛ͱಉ͡

    View full-size slide

  34. Go - ID ͱ time.Time ͷ૬ޓม׵
    godoc.org/github.com/kayac/go-katsubushi#ToID
    // ͦͷ࣌ࠁʹൃߦ͞ΕΔID(WorkerID=0, Sequence=0)Λऔಘ
    t1 := time.Now()
    id := katsubushi.ToID(t1)
    // ID͕ൃߦ͞Εͨ࣌ࠁΛ time.Time Ͱฦ͢
    t2 := katsubushi.ToTime(id)

    View full-size slide

  35. fluent-plugin-katsubushi
    rubygems.org/gems/fluent-plugin-katsubushi
    Fluentd ͷϑΟϧλʔϓϥάΠϯ
    ϑΟϧλʔΛ௨ա͢Δϩάʹ katsubushi Ͱൃߦͨ͠ ID Λ஫ೖ

    @type katsubushi
    id_key id # ஫ೖ͢Δkey໊
    host localhost # katsubushi ͷϗετ໊
    port 11212 # katsubushi ͷϙʔτ൪߸

    View full-size slide

  36. fluent-plugin-katsubushi
    fluent-plugin-katsubushi Λ௨ա͢Δͱ ID ͕஫ೖ͞ΕΔ
    {"message":"foo"}
    !
    {"message":"foo","id":329518490450919424}
    ϩάʹ unique ͳ ID ΛৼΓ͍ͨʂ
    ͚ͲൃߦՕॴͰ͸ৼΕͳ͍ʂͱ͍͏৔߹ʹ

    View full-size slide

  37. katsubushi ͷӡ༻

    View full-size slide

  38. worker-id ΛҰҙʹ͢Δ
    ӡ༻͍ͯ͠Δશͯͷ katsubushi process ͷ
    worker-id ΛҰҙʹ͢Δͷ͕ඞਢ
    worker-id ॏෳ ➡ ಉҰ࣌ࠁʹൃߦ͞ΕͨID͕ॏෳ͢Δ ➡ ഁ໓
    +------------------------------------------------------+
    | 64 bit |
    +--+-----------------+----------------+----------------+
    | 0| Timestamp 41bit | WorkerID 10bit | Sequence 12bit |
    +--+-----------------+----------------+----------------+

    View full-size slide

  39. worker-id ΛҰҙʹ͢Δ
    worker-id ͸ 1ʙ1023 ͷ੔਺஋
    ݻఆαʔόͰಈ͔͢ͳΒద੾ʹखͰ͚ͭΔ
    ಈతʹαʔό͕ىಈ͢Δ؀ڥͰ͸…ʁ
    ͨͱ͑͹ IP ΞυϨεͷୈ 3,4 octet Λ࢖͏
    192.168.0.10 ➡ 0 * 256 + 10 = 10
    192.168.1.20 ➡ 1 * 256 + 20 = 266
    192.168.2.30 ➡ 2 * 256 + 30 = 542

    View full-size slide

  40. worker-id ΛҰҙʹ͢Δ
    ίϯςφ؀ڥͰ͸…ʁ
    Redis Λ࢖ͬͨ worker-id ࣗಈׂΓ౰ͯػೳ (ޙड़)

    View full-size slide

  41. stats
    memcached ͷ stats ίϚϯυΛ࣮૷
    $ echo "STATS" | nc localhost 11212
    STAT pid 1
    STAT uptime 51014
    STAT time 1498716107
    STAT version 1.3.0
    STAT curr_connections 1
    STAT total_connections 8
    STAT cmd_get 18
    STAT get_hits 20
    STAT get_misses 0
    END

    View full-size slide

  42. key ஋ͷྫ ҙຯ
    pid 1 Process ID
    uptime 51014 ىಈ͔ͯ͠Βͷܦաඵ਺
    time 1498716107 ݱࡏ࣌ࠁ UNIX time
    version 1.3.0 όʔδϣϯ
    curr_connections 1 ݱࡏ઀ଓ͍ͯ͠ΔTCPίωΫγϣϯ

    total_connections 8 ىಈ͔ͯ͠Β઀ଓͨ͠ίωΫγϣϯ
    ਺ (ྦྷܭ)
    cmd_get 18 GET ίϚϯυΛॲཧͨ͠ճ਺ (ྦྷܭ)
    get_hits 20 ൃߦͨ͠ ID ਺ (ྦྷܭ)
    get_misses 0 ൃߦͰ͖ͳ͔ͬͨ ID ਺ (ྦྷܭ)

    View full-size slide

  43. ؂ࢹ͢΂͖஋
    key
    curr_connections ଟ͗͢ͳ͍͔
    uptime ר͖໭͍ͬͯΔ = ࠶ىಈͯ͠
    ͍Δ
    get_misses ൃੜ = ࣌ࠁ͕ר͖໭͍ͬͯΔ

    View full-size slide

  44. ࣌ࠁר͖໭Γ࣌ͷରԠ
    katsubushi ID ͷҰҙੑ͸࣌ࠁʹґଘ
    ࣌ࠁ͕ਐΉͷ͸໰୊ͳ͍͕ɺר͖໭Δͱ…(ͨͱ͑͹Ӟඵ)
    աڈʹൃߦͨ͠ ID ͱಉ͡ ID Λൃߦͯ͠͠·͏Մೳੑ
    ➡ ίʔυ಺Ͱ͸ר͖໭Βͳ͍Α͏ʹอޢ

    View full-size slide

  45. ҎલͷλΠϜελϯϓ͕དྷͨΒΤϥʔ
    func (g *Generator) NextID() (uint64, error) {
    g.lock.Lock()
    defer g.lock.Unlock()
    ts := g.timestamp()
    // for rewind of server clock
    if ts < g.lastTimestamp {
    return 0, errors.New("system clock was rollbacked")
    }

    View full-size slide

  46. Go 1.9 ͷ monotonic time Λ࢖͏(༧ఆ)
    Go 1.9 Ͱ monotonic time ͕ಋೖ͞ΕΔ 3
    ىಈ࣌ʹ time.Now() ͨ࣌͠ࠁ͔Βͷࠩ෼Λར༻͢Ε͹
    OS࣌ࠁ͕ר͖໭ͬͯ΋ӨڹΛड͚ͳ͍
    3 https://tip.golang.org/doc/go1.9#monotonic-time

    View full-size slide

  47. ఀࢭ͍ͯ͠Δؒʹ࣌ࠁ͕໭ͬͨΒ…
    katsubushi ϓϩηε಺Ͱ͸ݕ஌͠Α͏͕ͳ͍
    ରԠࡦɿࣗಈͰ࠶ىಈ͠ͳ͍
    Perl / Go ͷΫϥΠΞϯτ͸ଞͷϗετʹ fallback Ͱ͖Δ
    ઃఆͯ͋͠Δશͯͷϗετ͕ಉ࣌ʹམͪͳ͚Ε͹ ID औಘͰ͖Δ

    View full-size slide

  48. ҆ఆੑ
    2015೥ʹϦϦʔε͔ͯ͠Β2೥ؒ
    Lobiͷຊ൪؀ڥͷ΄΅શͯͷϗετͰՔಇ͍ͯ͠Δ͕
    ௨ৗར༻Ͱϓϩηε͕མͪͨ͜ͱ͸ͳ͍ 2
    ϝϞϦϦʔΫ΋ܦݧͳ͠
    2 ͨͩ͠ memcached protocolͰਖ਼͘͠ΞΫηεͨ͠৔߹ɻෆਖ਼ͳίϚϯυͰΫϥογϡͨ͜͠ͱ͸͋ͬͨ github.com/kayac/
    go-katsubushi/pull/8

    View full-size slide

  49. WebΞϓϦέʔγϣϯ΁ͷԠ༻

    View full-size slide

  50. Request ID ʹΑΔ
    ΞϓϦέʔγϣϯ௥੻

    View full-size slide

  51. Request ID Λ katsubushi Ͱൃߦ
    nginx͕ϦΫΤετΛड͚෇͚ͨ࣌఺Ͱ katsubushi ID Λൃߦ
    ޙॲཧʹҾ͖౉͢͜ͱͰ֤छॲཧͷ௥੻Λ༰қʹ
    nginx ͱ katsubushi ͕ memcached protocol Ͱ௨৴

    View full-size slide

  52. IDΛҾ͖ճͯ͠
    ϩάʹه࿥

    View full-size slide

  53. katsubushi ➡ nginx
    ΞΫηεຖʹ nginx lua ͕ ID Λऔಘͯ͠ม਺ $req_id ʹઃఆ
    upstream katsubushi {
    server 127.0.0.1:11212;
    keepalive 10;
    }
    location /katsubushi {
    internal;
    set $memcached_key "id"; # key͸ͳΜͰ΋OK
    memcached_pass katsubushi;
    }
    set $req_id 0;
    access_by_lua_block {
    local res = ngx.location.capture("/katsubushi")
    if res then
    ngx.var.req_id = res.body
    end
    }

    View full-size slide

  54. nginx Ͱϩάʹग़ྗ
    log_format ltsv
    'req_id:$req_id\t'
    'host:$remote_addr\t'
    'user:$remote_user\t'
    'time:$time_iso8601\t'
    'method:$request_method\t'
    ...

    View full-size slide

  55. nginx ➡ App(Plack)
    ϦΫΤετϔομ X-Request-ID Λઃఆ
    upstream app {
    server unix:/var/tmp/app.socket;
    }
    location / {
    proxy_set_header X-Request-ID $req_id;
    proxy_pass http://app;
    }

    View full-size slide

  56. App(Plack) ➡ local %ENV
    Plack::Middleware::SetLocalEnv
    Plack ͷ $env ͔Β local %ENV ʹ஋Λઃఆ͢Δ
    ΞϓϦέʔγϣϯͷͲ͔͜ΒͰ΋ $ENV{REQUEST_ID} ͰࢀরՄ
    # app.psgi
    use Plack::Builder;
    builder {
    enable "SetLocalEnv",
    REQUEST_ID => "HTTP_X_REQUEST_ID",
    ;
    # ...
    $app;
    }

    View full-size slide

  57. Plack::Middleware::SetLocalEnv
    ࣮૷͸͜Ε͚ͩ
    local ͳͷͰείʔϓΛൈ͚Δͱ(ϦΫΤετॲཧ͕ऴΘΔͱ)
    ஋͕ݩʹ໭Δ
    sub call {
    my ($self, $env) = @_;
    local %ENV = %ENV;
    for my $key (keys %$self) {
    $ENV{$key} = $env->{ $self->{$key} }
    if exists $env->{$self->{$key}};
    }
    $self->app->($env);
    }

    View full-size slide

  58. App(Plack) ➡ app.log (Log::Minimal)
    $Log::Minimal::(PRINT|DIE) Ͱग़ྗΛΧελϚΠζ
    use Log::Minimal;
    BEGIN {
    $Log::Minimal::PRINT = sub {
    my ($time, $type, $message, $trace) = @_;
    my $req_id = $ENV{REQUEST_ID} // 0;
    warn "[$$][$req_id][$type] $message at $trace\n";
    };
    }
    warnf("something bad!");
    [64302][329733861698727936][WARN] something bad! at .../App.pm line XX

    View full-size slide

  59. App(Plack) ➡ Fluentd
    ϑΝΠϧΛհͣ͞௚઀ Fluent::Logger Ͱૹ৴͢Δ৔߹
    ϝοηʔδʹ req_id Λ஫ೖ
    package Logger;
    use Fluent::Logger;
    sub log {
    my ($self, $tag, $msg) = @_;
    $msg->{req_id} = $ENV{REQUEST_ID} // 0;
    $logger->post($tag, $msg); # $logger is Fluent::Logger
    }

    View full-size slide

  60. App(Plack) ➡ MySQL
    ΫΤϦͷઌ಄ʹSQLίϝϯτͰ /* req_id=... */ ΛຒΊࠐΉ
    use parent 'Teng::Sharding';
    around ["execute", "do"] => sub {
    my ($orig, $self, $sql, @rest) = @_;
    my $req_id = $ENV{REQUEST_ID} // 0;
    return $orig->($self, "/* req_id=${req_id} */\n${sql}", @rest);
    };
    # Time: 170629 3:27:47
    # User@Host: xxx[xxx] @ [192.0.2.1]
    # Query_time: 1.080420 Lock_time: 0.000040 Rows_sent: 5 Rows_examined: 909850
    SET timestamp=1498674467;
    /* req_id=329689349781536768 */
    SELECT ..... FROM ...

    View full-size slide

  61. ͍ΖΜͳى఺Ͱௐ͕ࠪͰ͖Δ
    ΞΫηεϩάͰ status:500
    ➡ ΞϓϦͷΤϥʔϩάͱর߹
    slow.log ʹࠔͬͨΫΤϦ͕ه࿥͞Ε͍ͯΔ
    ඇಉظ job ͕ࣦഊ
    ➡ ൃੜݩʹͳͬͨHTTPϦΫΤετΛΞΫηεϩάͰಛఆ

    View full-size slide

  62. ଞܗࣜͷ Request ID ͱͷൺֱ
    $request_id (nginx >= 1.11.0)
    unique request identifier generated from 16 random bytes, in
    hexadecimal 4
    ྫ: 9f53d64cf47dd966a7462f5a737b5ade
    ௕͍
    ࣌ࠁ৘ใؚ͕·Εͳ͍
    4 http://nginx.org/en/docs/http/ngxhttpcoremodule.html#varrequest_id

    View full-size slide

  63. ೔࣌ͰύʔςΟγϣχϯάΛߦ͏
    ςʔϒϧͷϓϥΠϚϦΩʔͱͯ͠࢖͏

    View full-size slide

  64. ҰఆظؒͰݹ͍σʔλΛফ͍ͨ͠ςʔϒϧ(௨஌)
    Redis ʹ id ͷΈอ࣋ͯ͠ index ʹ͍ͯ͠Δ
    id auto_increment, created_date ͷෳ߹Primary Key
    CREATE TABLE `notification` (
    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
    `created_date` datetime NOT NULL,
    /* other columns */
    PRIMARY KEY (`id`,`created_date`)
    ) ENGINE=InnoDB AUTO_INCREMENT=xxxxx DEFAULT CHARSET=utf8
    PARTITION BY RANGE COLUMNS(created_date)
    (PARTITION p20170602 VALUES LESS THAN ('2017-06-02') ENGINE = InnoDB,
    PARTITION p20170603 VALUES LESS THAN ('2017-06-03') ENGINE = InnoDB,
    PARTITION p20170604 VALUES LESS THAN ('2017-06-04') ENGINE = InnoDB,
    .....

    View full-size slide

  65. id + datetime ύʔςΟγϣχϯάͷ໰୊
    idͷΈͰݕࡧ͢ΔΫΤϦͰ͸מΓࠐΊͳ͍ͨΊ
    શύʔςΟγϣϯΛݕࡧ͢Δ
    (30೔෼࢒ͨ͢Ίʹ30 partition੾Δͱ30ഒͷݕࡧίετ)
    ➡ Α͘΍Δ WHERE id IN (?,?,....) ͕஗͍
    ➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍͸ͣͷύʔςΟγϣϯʹ
    සൟʹࢀর͕૸Δ (InnoDB Buffer Poolͷແବ)

    View full-size slide

  66. ! katsubushi ID Ͱ RANGE
    ύʔςΟγϣϯΛ੾Ε͹…

    View full-size slide

  67. katsubushi ID ͷΈͰ1೔୯ҐͷύʔςΟγϣχϯά
    CREATE TABLE `notification` (
    `id` bigint(20) unsigned NOT NULL, /* AUTO_INCREMENT ͠ͳ͍ */
    `created_date` datetime NOT NULL, /* ෆཁ͔΋͠Εͳ͍ */
    /* other columns */
    PRIMARY KEY (`id`) /* pkey ͸ id ͷΈ */
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8
    PARTITION BY RANGE COLUMNS(id)
    (PARTITION p20170602 VALUES LESS THAN (319852589875200000) ENGINE = InnoDB,
    /* ^ 2017-06-02 00:00:00 JST */
    PARTITION p20170603 VALUES LESS THAN (320214977740800000) ENGINE = InnoDB,
    /* ^ 2017-06-03 00:00:00 JST */
    PARTITION p20170604 VALUES LESS THAN (320577365606400000) ENGINE = InnoDB,
    /* ^ 2017-06-04 00:00:00 JST */
    .....

    View full-size slide

  68. katsubushi ID only ύʔςΟγϣχϯάͷར఺
    id ͷΈͷݕࡧΫΤϦ͕ࣗಈతʹמΓࠐ·ΕΔ
    ➡ Α͘΍Δ WHERE id IN (?,?,....) ͕࠷దԽ͞ΕΔ
    ➡ Row Reads ܹݮ
    ➡ ݹͯ͘໓ଟʹΞΫηε͞Εͳ͍ύʔςΟγϣϯʹ͸
    ࢀর͕૸Βͳ͍ (InnoDB Buffer Poolͷޮ཰Up)

    View full-size slide

  69. katsubushi ID only ύʔςΟγϣχϯάͷܽ఺?
    Ͳͷ஋Λڥʹ੾Ε͹͍͍ͷ͔͕௚ײతͰ͸ͳ͍
    ! Katsubushi::Converter Ͱܭࢉ
    ಛఆͷ೔͚ͩࢀর͍ͨ͠
    ! ೔ͷൣғΛ ID ʹม׵ͯ͠ WHERE ۟ʹೖΕΕ͹OK

    View full-size slide

  70. ίϯςφ؀ڥ΁ͷదԠ

    View full-size slide

  71. ίϯςφ؀ڥͰ͸ worker-id ΛҰҙʹ
    ͢Δͷ͕೉͍͠!

    View full-size slide

  72. ίϯςφ؀ڥͰ worker-id ΛҰҙʹ͢Δ
    ❌ ਓྗͰͷݻఆׂΓ౰ͯ͸Ͱ͖ͳ͍
    ੜ·Εͯ͸ফ͑Δͷ͕ίϯςφ
    ❌ IPΞυϨεΛ࢖͏͜ͱ΋Ͱ͖ͳ͍
    ϗετͱNAT͍ͯ͠ΔͱɺผϗετͰಉҰͷ IP ΞυϨεΛ
    ࢖͍ͬͯΔίϯςφ͕ଘࡏ͠͏Δ
    Ͳ͏͢Ε͹…

    View full-size slide

  73. Redis Ͱഉଞతʹ஋Λऔಘͤ͞Δ
    ͙͢ࢥ͍ͭ͘ͷ͸ू߹ܕͰ SPOP
    > SADD id_pool 1 2 3 4 # 4ཁૉͷू߹Λ࡞Δ
    > SPOP id_pool # ू߹͔ΒϥϯμϜʹҰͭऔΓग़͢
    "3"
    > SMEMBERS id_pool # ࢒Γ
    "1"
    "2"
    "4"
    Simple is best?

    View full-size slide

  74. SPOP ํࣜͷ໰୊
    1. ϓϩηεىಈ࣌ʹ SPOP Ͱऔಘ
    2. ϓϩηεऴྃ࣌ʹ SADD Ͱฦ٫
    औΓग़͞Εͨ஋͕ฦ٫͞ΕΔ͜ͱ͕อূͰ͖ͳ͍
    ϓϩηε͸ਖ਼ৗऴྃ͢Δͱ͸ݶΒͳ͍
    ➡ ௕ظؒӡ༻͍ͯ͠Δͱฦ٫͞Εͳ͍஋͕
    ཷ·͍ͬͯͬͯރׇ͢ΔڪΕ

    View full-size slide

  75. ϝϯςφϯεϑϦʔͳํ๏Λߟ͍͑ͨ

    View full-size slide

  76. raus
    github.com/fujiwara/raus
    "RAnged Unique id Supplier"
    RedisΛ࢖ͬͯҰఆൣғͷ੔਺IDΛ
    ॏෳͳׂ͘Γ౰ͯΔͨΊͷϥΠϒϥϦ

    View full-size slide

  77. PubSub + SET NX + GETSET + EXPIRE
    1. ੜ͖͍ͯΔϓϩηε͕ PubSub channel ʹอ͍࣋ͯ͠ΔIDΛ
    ྲྀ͢
    2. อ͍࣋ͯ͠ΔIDͷ Key Λ GETSET + EXPIRE Ͱߋ৽͠ଓ͚Δ
    3. ৽نϓϩηε͸ PubSub channel Λ؍࡯ͯ͠ɺ࢖ΘΕ͍ͯͳ
    ͦ͏ͳIDΛ SET NX Ͱऔಘ
    4. औಘͰ͖ͨΒ PubSub + GETSET + EXPIRE Λଓ͚Δ

    View full-size slide

  78. 1. IDΛอ͍࣋ͯ͠Δطଘ raus ͸ PUBLISH + GETSET Λఆظతʹߦ͏

    View full-size slide

  79. 2. ৽ن raus ͸ SUBSCRIBE ͯ͠ଞͷ raus ͕࢖͍ͬͯΔIDΛ೺Ѳ

    View full-size slide

  80. 3. ৽ن raus ͸ଞͷ raus ͕࢖ͬͯͳ͍(͸ͣͷ) key Λ SET NX Ͱഉଞऔಘ

    View full-size slide

  81. 4. ID ΛऔಘͰ͖ͨΒ PUBLISH + GETSET Λఆظ࣮ߦ

    View full-size slide

  82. raus ͕ϝϯςφϯεϑϦʔͳཧ༝

    View full-size slide

  83. raus ͕ϝϯςφϯεϑϦʔͳཧ༝
    • ࢮΜͩϓϩηε͸ GETSET ͠ͳ͍ͷͰ key ͕࣌ݶ։์͞ΕΔ
    • ૄ௨Ͱ͖ͳ͘ͳͬͯ΋Ұఆ࣌ؒ͸ key ͕อ࣋͞Ε͍ͯΔͷͰ
    ଞ͕औಘͰ͖ͳ͍
    • ૄ௨͕ճ෮ͨ͠ޙʹGETSETʹࣦഊͨ͠
    (=ଞͷϓϩηε͕։์͞Εͨ key Λऔಘͯ͠͠·ͬͨ)৔߹
    ଈ࠲ʹࢮ͵

    View full-size slide

  84. katsubushi + raus
    $ katsubushi -redis redis://192.0.2.2:6379
    Waiting for worker-id automated assignment (between 1 and 1023) with redis://192.0.2.2:6379
    xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 <= ଞͷkatsubushi͕woker_id=2Λอ͍࣋ͯ͠Δ
    xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2 ͷΛsubscribeͰ؍࡯͍ͯ͠Δ
    xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2
    xuuid:f81057b3-aa5d-4e3c-857a-230bfdb2135f xid:2
    candidate ids: [1 3 4 5 6 7 8 9 10 11] <= 2 Ҏ֎ͷΛީิʹͯͦ͠ͷத͔ΒϥϯμϜʹ
    trying to get lock key raus:id:8 <= SET NX
    got lock for 8 <= ੒ޭͨ͠ͷͰ 8 Λऔಘ
    Assigned worker-id: 8
    Listening at [::]:11212
    Worker ID = 8
    <= PUBLISH + GETSET ΛཪͰଓ͚͍ͯΔ

    View full-size slide

  85. katsubushi + raus
    worker_id=8 Λऔಘͨ͠ katsubushi ͕ಈ͍͍ͯΔঢ়ଶͰ
    key Λ্ॻ͖ͯ͠ΈΔͱ…
    $ redis-cli set raus:id:8 xxx
    unexpected uuid got: xxx
    panic: unexpected uuid got: xxx
    goroutine 42 [running]:
    main.assignWorkerID.func1(0xc4201462c0, 0xc42006f020)
    src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:230 +0xb0
    created by main.assignWorkerID
    src/github.com/kayac/go-katsubushi/cmd/katsubushi/main.go:235 +0x4fe

    View full-size slide

  86. ·ͱΊ
    • Snowflake ͱಉ༷ͷΞϧΰϦζϜͰ෼ࢄ؀ڥͰϢχʔΫͳID
    ΛൃߦͰ͖Δ katsubushi
    • ܰྔͳͷͰ DB ͷߦ ID Ҏ֎ʹ΋ར༻Մೳ
    ➡ Request ID Ͱॲཧ௥੻
    • ࣌ࠁؚ͕·ΕΔಛੑΛੜ͔ͯ͠ύʔςΟγϣχϯάʹ΋
    • ίϯςφ؀ڥʹରԠ͢Δ Redis Λ࢖ͬͨࣗಈΫϥελϦϯά

    View full-size slide