Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Redis Lua Script With Red Envelope & Message Queue

Redis Lua Script With Red Envelope & Message Queue

Golang Taiepi Meetup #41

Yun Chen

April 30, 2019
Tweet

More Decks by Yun Chen

Other Decks in Programming

Transcript

  1. REDIS LUA SCRIPT
    WITH RED ENVELOPE & MESSAGE
    QUEUE
    GOLANG TAIPEI MEETUP #41
    https://github.com/chenyunchen/K8S-Meetup

    View Slide

  2. WHO AM I
    Chen Yun Chen (Alex)
    > [email protected]
    > blog.yunchen.tw
    Experience
    > Backend Engineer at
    JKOPay
    > Software Engineer at
    Linker Networks

    View Slide

  3. Ջ讕ฎLUA?
    Lua斕ᰁ承᥺牧ݝ۱珀羊墋ጱ໐ஞ޾च๜Library
    > 匍磪礍䯤ӥ൉׀狕硬皤֐ۑ胼Ӟਧጱ覄ၚ௔
    > Redis, Nginx舙襑ਮ蕣ݢ્獈Lua狶೐疻(籆ๅ碝)
    諑劭Ӯኴ㮆ՈUIՕᶎጱਮ蕣玕(Lua)
    https://wowwiki.fandom.com/wiki/Lua
    3

    View Slide

  4. ፓ獮傶Ջ讕襑ᥝֵአ REDIS+LUA ?
    > ૪磪य़ᰁֵአRedis狶傶Cache牏Queue䁰ว
    > ၚ㵕襑缏ܨℂ匍磪礍䯤ӥ䋿匍夺۱牏懱௳ۑ胼
    > ᤍᰁ姘螀牏碍硁ᰁ牏ݢ覎௔ฎ玽ᥝ疩獈ই:Kafka
    > 疩獈碝๐率ᛗ绐ਧጱ碻樌现觓檺
    (Lua姘虁500ᤈૢݦ牧薹究蟂獤襑穩)
    4

    View Slide

  5. REDIS
    > 㻌䁆ᤈ姼(Single-Thread)
    > ܻৼ௔(Atomic Operation)
    IF CLUSTER & MULTIPLE KEYS COMMAND
    > node:{nodeID}:userID (Hash Tag)
    5

    View Slide

  6. REDIS-MULTI
    > 蝚螂Ӟ稞䁆ᤈग़㮆瞲犤
    > 狒挨䁆ᤈ殼ଧ
    1. ग़執砺֢樄衴
    瞲犤樌ጱ虻碘
    舙ஂ種襑ᥝ狶墋㻌蒂ቘ
    Ֆ襑碍稞䌘Redis礚扇膏捝䌃
    2. 虻碘ਠ碉௔
    舙磪ग़稞ጱ礚扇膏捝䌃
    ׎櫞犥嘦狒ಅ磪砺֢緡౮ۑ
    6

    View Slide

  7. REDIS-LUA
    > 瞲犤䁆ᤈݢ覎ଶ斃ṛ
    > ࣁग़執Key樌蝱ᤈValue蒂ቘ
    BUT...
    1. 篷ဩ狒挨Redis吚秚盅ጱਠ碉௔
    2. Script๜蛪䌃ဩ磪藮篷ဩࢧ笔
    7

    View Slide

  8. GOLANG
    > 斉獈脻๜
    疥碉犩෈䲆斉獈Redis叨ኞSHA1 Checksum
    > 癲獈KEYS, ARGV
    KEYS: ݝ硯ᥝ砺֢ጱKEY
    ARGV: ᒫӞ㮆硯ਧ嬝ጱ砺֢ᤈ傶牧ٌ訏ஃ盅

    8

    View Slide

  9. LUA: GOLANG
    options := redis.Options{
    Network: "tcp4",
    Addr: "127.0.0.1:6379",
    DB: 0,
    PoolSize: 10,
    }
    client := redis.NewClient(&options)
    fileBytes, err := ioutil.ReadFile("path")
    if err != nil {
    return
    }
    sha, err := client.ScriptLoad(string(fileBytes)).Result()
    if err != nil {
    return
    }
    keys := []string{"redenvelopeID"}
    args := []string{"get_redenvelope", "userID"}
    res, err := client.EvalSha(sha, keys, args).Result()
    9

    View Slide

  10. LUA: DEBUG
    ./redis-cli --ldb --eval script.lua
    mykey somekey , arg1 arg2
    More Detail:
    https://redis.io/topics/ldb
    10

    View Slide

  11. 犥䋿匍夺۱傶ֺ
    咳夺۱ᤈ傶
    > አ䜛戔ਧᰂ氃牏碍ᰁ
    > 獤蟴褰秚ᰂ氃ጱग़㮆夺۱
    硩夺۱ᤈ傶
    > 嘦藨አ䜛ฎ玽磪砋螂夺۱
    > 覿ڊӞ㮆褰秚夺۱妔አ䜛
    > 疥扗硯۱硯獈૪窞揲ጱ褧ڜ
    11

    View Slide

  12. 毆ض叨ኞ夺۱
    > 究ਧᰂ氃ጱ獤蟴秚ګ
    (ex: 1000ز 獤蟴 10۱)
    50% 60% 70% 80% 90% 99%
    [30 41.11 52.22 63.33 74.44
    125.56 136.67 147.78 158.89 170]
    > 矎ଧ(褰秚Ի矦) or 藶穩瞲窔
    [136.67 158.89 41.11 170 74.44
    147.78 125.56 30 52.22 63.33]
    12

    View Slide

  13. ח扖藯஑অ:
    ᓒ桽አၶ讨牧螛෱ᤩՈ瞇
    var money1 float32 = 9.9
    fmt.Println(money1 * 100) // 989.99994
    var money2 float64 = 1.1
    fmt.Println(money2 * 100) // 110.00000000000001
    13

    View Slide

  14. 羊伛ଶ蔭纈
    > 2蝱ګ(IEEE-754)
    (膏ֵአ聜䙼犋ݶ)
    > 10蝱ګ
    (Ֆ䨝०羊伛ଶ)
    疰ᓒ纷ୗ胼ᛔ㵕犚盏狕ྋ
    褰茐蕦褾ጱࢥ㳷螀ᓒ懯ᓒ牧磧奰奾ຎՖ䨝ڊ㺔氂
    14

    View Slide

  15. 砋夺۱
    1. ฎ玽磪砋螂夺۱HashMap
    2. Ֆ磪夺۱׎ᑏ獈૪窞揲
    3. አ䜛膏夺۱ጱ橕狌ਂࢧ
    HashMap
    > Existے蝧玲஑አ䜛膏夺۱橕狌
    > Consume஑Ꭳ糷玲殼ଧ牏ڜ蔭
    > Lua䋿֢1~3 狒挨ܻৼ௔
    15

    View Slide

  16. LUA: 砋夺۱
    local function getRedenvelope(rID, userID)
    if redis.call('hexists', rID, userID) ~= 0 then
    return nil;
    else
    local redenvelope = redis.call('rpop', 'redenvelope');
    if redenvelope then
    local r = cjson.decode(redenvelope);
    r['userID'] = tonumber(userID);
    local re = cjson.encode(r);
    redis.call('hset', rID, userID, userID);
    redis.call('lpush', 'consume_' .. rID, re);
    return re;
    end
    end
    return nil;
    end
    16

    View Slide

  17. DEMO
    SEND & RECEIVE RED ENVELOPE

    View Slide

  18. BUT... RETRY & ACK
    1. Worker
    篷ဩ狒挨玲አRedis虻碘ጱWorker 100%౮ۑ
    ݢ胼襑ᥝᛔᤈ蒂ቘӞ犚奞℄:
    ๚蒂ቘ牏蒂ቘӾ牏૪蒂ቘ
    2. አ䜛ᒒ
    苭Ӟአ䜛ᒒ䷱硩ک夺۱蝢Ꭳ䨝ெ讕䰬?
    3. ䷱硩ک夺۱蝢Ꭳ吚簁疰100%砋犋ک
    4. 虋๚䋿匍ᰂ氃蜴ࢧ咳硯ጱአ䜛癱蒈
    18

    View Slide

  19. ℂ夺۱獤蟴ک蝢Ꭳ
    夺۱牏懱௳๐率ጱ瞥獤
    > 虻რ藲ଶӤጱᘍᰁ
    > ०硻Retryጱ㺔氂
    19

    View Slide

  20. 懱௳褧ڜ
    螨ع矾疨&ے蝧捝玲
    User᮷磪ᛔ૩ጱQueueᥝ姘虁
    褰茐አ䜛碍/虻碘ᰁɁ
    仂੝懿ࣁग़執虻碘樌ጱਂ玲
    20

    View Slide

  21. 犥䋿匍懱௳褧ڜ傶ֺ
    List
    > 膏夺۱砺֢Ӟ膌牧墋㻌ᓕቘ
    > 㰍ᔱ୚㬵瞲ਧ牧篷messageID
    > Value: String Data
    Bit Manipulation
    > 㻌ӞKEY:Value珀ಅ襑橕狌獉਻
    > ᛔᤈ姘虁虻碘bytes裾ଶ/֖ᑏ
    > Value: Binary Data
    21

    View Slide

  22. LUA: MESSAGE HEADER
    local function getHeader(name)
    return redis.call('getrange', name, 0, 7);
    end
    local function parseHeader(header)
    return struct.unpack('I4HH', header);
    end
    local function setHeader(name, lastID, head, tail)
    local header = struct.pack('I4HH', lastID, head, tail);
    redis.call('setrange', name, 0, header);
    end
    22

    View Slide

  23. DEMO
    PUSH, GET & FINISH MESSAGE

    View Slide

  24. GROUP MESSAGE
    ग़㮆Userਂ玲ݶӞ犩MessageList
    > Hashݚ夵袅ݱUserጱ蒂ቘ蝱ଶ
    > 獋㮆Unsigned Short಑۱Head, Tail
    24

    View Slide

  25. BUT...
    > 瞱ԋ玕&盋ܻ
    稲ݥ懱௳ጱ礚扇౲Redis䥁娄蝨౮ጱ०硻
    > 瞙稞蒂ቘ
    ࢩၚ㵕ᘒ叨ኞ碍苭㳷ጱ夺۱懱௳
    瞙稞蒂ቘӞ䰬䨝磪Retry缛㺔氂
    綍ᛗ瞙稞Ӿጱ蟂獤०硻ᥝݚᤈ蒂ቘ
    獊蟂᯿蝑䨝虏य़蟂獤౮ۑጱ懱௳ڊ匍獋稞
    25

    View Slide

  26. ٚ℄፜Ӟ犚懿䛂誢: MSGP
    26

    View Slide

  27. MORE CHALLENGE NEXT
    Redis奰绗犋ฎ䌕槹㬵蒂ቘQueue
    褰茐禂率౮裾ᘒᛗӥӞ褩ྦྷԏ獮
    ᴻԧ薪疗匍褩ྦྷ狶ဩಅ胼ک螈ጱ禅褖
    稭蜰䨝盅Ӟ蚏獤Ձݱ圵犋ݶጱ狶ဩ
    27

    View Slide

  28. Q & A

    View Slide