Redis Lua Script With Red Envelope & Message Queue

Golang Taiepi Meetup #41

Transcript

1. REDIS LUA SCRIPT
   WITH RED ENVELOPE & MESSAGE
   QUEUE
   GOLANG TAIPEI MEETUP #41
   https://github.com/chenyunchen/K8S-Meetup
   

   View Slide

2. WHO AM I
   Chen Yun Chen (Alex)
   > me@yunchen.tw
   > blog.yunchen.tw
   Experience
   > Backend Engineer at
   JKOPay
   > Software Engineer at
   Linker Networks
   

   View Slide

3. Ջ讕ฎLUA?
   Lua斕ᰁ承᥺牧ݝ۱珀羊墋ጱ໐ஞ޾च๜Library
   > 匍磪礍䯤ӥ൉׀狕硬皤֐ۑ胼Ӟਧጱ覄ၚ௔
   > Redis, Nginx舙襑ਮ蕣ݢ્獈Lua狶೐疻(籆ๅ碝)
   諑劭Ӯኴ㮆ՈUIՕᶎጱਮ蕣玕(Lua)
   https://wowwiki.fandom.com/wiki/Lua
   3
   

   View Slide

4. ፓ獮傶Ջ讕襑ᥝֵአ REDIS+LUA ?
   > ૪磪य़ᰁֵአRedis狶傶Cache牏Queue䁰ว
   > ၚ㵕襑缏ܨℂ匍磪礍䯤ӥ䋿匍夺۱牏懱௳ۑ胼
   > ᤍᰁ姘螀牏碍硁ᰁ牏ݢ覎௔ฎ玽ᥝ疩獈ই:Kafka
   > 疩獈碝๐率ᛗ绐ਧጱ碻樌现觓檺
   (Lua姘虁500ᤈૢݦ牧薹究蟂獤襑穩)
   4
   

   View Slide

5. REDIS
   > 㻌䁆ᤈ姼(Single-Thread)
   > ܻৼ௔(Atomic Operation)
   IF CLUSTER & MULTIPLE KEYS COMMAND
   > node:{nodeID}:userID (Hash Tag)
   5
   

   View Slide

6. REDIS-MULTI
   > 蝚螂Ӟ稞䁆ᤈग़㮆瞲犤
   > 狒挨䁆ᤈ殼ଧ
   1. ग़執砺֢樄衴
   瞲犤樌ጱ虻碘
   舙ஂ種襑ᥝ狶墋㻌蒂ቘ
   Ֆ襑碍稞䌘Redis礚扇膏捝䌃
   2. 虻碘ਠ碉௔
   舙磪ग़稞ጱ礚扇膏捝䌃
   ׎櫞犥嘦狒ಅ磪砺֢緡౮ۑ
   6
   

   View Slide

7. REDIS-LUA
   > 瞲犤䁆ᤈݢ覎ଶ斃ṛ
   > ࣁग़執Key樌蝱ᤈValue蒂ቘ
   BUT...
   1. 篷ဩ狒挨Redis吚秚盅ጱਠ碉௔
   2. Script๜蛪䌃ဩ磪藮篷ဩࢧ笔
   7
   

   View Slide

8. GOLANG
   > 斉獈脻๜
   疥碉犩෈䲆斉獈Redis叨ኞSHA1 Checksum
   > 癲獈KEYS, ARGV
   KEYS: ݝ硯ᥝ砺֢ጱKEY
   ARGV: ᒫӞ㮆硯ਧ嬝ጱ砺֢ᤈ傶牧ٌ訏ஃ盅
   硎
   8
   

   View Slide

9. LUA: GOLANG
   options := redis.Options{
   Network: "tcp4",
   Addr: "127.0.0.1:6379",
   DB: 0,
   PoolSize: 10,
   }
   client := redis.NewClient(&options)
   fileBytes, err := ioutil.ReadFile("path")
   if err != nil {
   return
   }
   sha, err := client.ScriptLoad(string(fileBytes)).Result()
   if err != nil {
   return
   }
   keys := []string{"redenvelopeID"}
   args := []string{"get_redenvelope", "userID"}
   res, err := client.EvalSha(sha, keys, args).Result()
   9
   

   View Slide

10. LUA: DEBUG
    ./redis-cli --ldb --eval script.lua
    mykey somekey , arg1 arg2
    More Detail:
    https://redis.io/topics/ldb
    10
    

    View Slide

11. 犥䋿匍夺۱傶ֺ
    咳夺۱ᤈ傶
    > አ䜛戔ਧᰂ氃牏碍ᰁ
    > 獤蟴褰秚ᰂ氃ጱग़㮆夺۱
    硩夺۱ᤈ傶
    > 嘦藨አ䜛ฎ玽磪砋螂夺۱
    > 覿ڊӞ㮆褰秚夺۱妔አ䜛
    > 疥扗硯۱硯獈૪窞揲ጱ褧ڜ
    11
    

    View Slide

12. 毆ض叨ኞ夺۱
    > 究ਧᰂ氃ጱ獤蟴秚ګ
    (ex: 1000ز 獤蟴 10۱)
    50% 60% 70% 80% 90% 99%
    [30 41.11 52.22 63.33 74.44
    125.56 136.67 147.78 158.89 170]
    > 矎ଧ(褰秚Ի矦) or 藶穩瞲窔
    [136.67 158.89 41.11 170 74.44
    147.78 125.56 30 52.22 63.33]
    12
    

    View Slide

13. ח扖藯஑অ:
    ᓒ桽አၶ讨牧螛෱ᤩՈ瞇
    var money1 float32 = 9.9
    fmt.Println(money1 * 100) // 989.99994
    var money2 float64 = 1.1
    fmt.Println(money2 * 100) // 110.00000000000001
    13
    

    View Slide

14. 羊伛ଶ蔭纈
    > 2蝱ګ(IEEE-754)
    (膏ֵአ聜䙼犋ݶ)
    > 10蝱ګ
    (Ֆ䨝०羊伛ଶ)
    疰ᓒ纷ୗ胼ᛔ㵕犚盏狕ྋ
    褰茐蕦褾ጱࢥ㳷螀ᓒ懯ᓒ牧磧奰奾ຎՖ䨝ڊ㺔氂
    14
    

    View Slide

15. 砋夺۱
    1. ฎ玽磪砋螂夺۱HashMap
    2. Ֆ磪夺۱׎ᑏ獈૪窞揲
    3. አ䜛膏夺۱ጱ橕狌ਂࢧ
    HashMap
    > Existے蝧玲஑አ䜛膏夺۱橕狌
    > Consume஑Ꭳ糷玲殼ଧ牏ڜ蔭
    > Lua䋿֢1~3 狒挨ܻৼ௔
    15
    

    View Slide

16. LUA: 砋夺۱
    local function getRedenvelope(rID, userID)
    if redis.call('hexists', rID, userID) ~= 0 then
    return nil;
    else
    local redenvelope = redis.call('rpop', 'redenvelope');
    if redenvelope then
    local r = cjson.decode(redenvelope);
    r['userID'] = tonumber(userID);
    local re = cjson.encode(r);
    redis.call('hset', rID, userID, userID);
    redis.call('lpush', 'consume_' .. rID, re);
    return re;
    end
    end
    return nil;
    end
    16
    

    View Slide

17. DEMO
    SEND & RECEIVE RED ENVELOPE
    

    View Slide

18. BUT... RETRY & ACK
    1. Worker
    篷ဩ狒挨玲አRedis虻碘ጱWorker 100%౮ۑ
    ݢ胼襑ᥝᛔᤈ蒂ቘӞ犚奞℄:
    ๚蒂ቘ牏蒂ቘӾ牏૪蒂ቘ
    2. አ䜛ᒒ
    苭Ӟአ䜛ᒒ䷱硩ک夺۱蝢Ꭳ䨝ெ讕䰬?
    3. ䷱硩ک夺۱蝢Ꭳ吚簁疰100%砋犋ک
    4. 虋๚䋿匍ᰂ氃蜴ࢧ咳硯ጱአ䜛癱蒈
    18
    

    View Slide

19. ℂ夺۱獤蟴ک蝢Ꭳ
    夺۱牏懱௳๐率ጱ瞥獤
    > 虻რ藲ଶӤጱᘍᰁ
    > ०硻Retryጱ㺔氂
    19
    

    View Slide

20. 懱௳褧ڜ
    螨ع矾疨&ے蝧捝玲
    User᮷磪ᛔ૩ጱQueueᥝ姘虁
    褰茐አ䜛碍/虻碘ᰁɁ
    仂੝懿ࣁग़執虻碘樌ጱਂ玲
    20
    

    View Slide

21. 犥䋿匍懱௳褧ڜ傶ֺ
    List
    > 膏夺۱砺֢Ӟ膌牧墋㻌ᓕቘ
    > 㰍ᔱ୚㬵瞲ਧ牧篷messageID
    > Value: String Data
    Bit Manipulation
    > 㻌ӞKEY:Value珀ಅ襑橕狌獉਻
    > ᛔᤈ姘虁虻碘bytes裾ଶ/֖ᑏ
    > Value: Binary Data
    21
    

    View Slide

22. LUA: MESSAGE HEADER
    local function getHeader(name)
    return redis.call('getrange', name, 0, 7);
    end
    local function parseHeader(header)
    return struct.unpack('I4HH', header);
    end
    local function setHeader(name, lastID, head, tail)
    local header = struct.pack('I4HH', lastID, head, tail);
    redis.call('setrange', name, 0, header);
    end
    22
    

    View Slide

23. DEMO
    PUSH, GET & FINISH MESSAGE
    

    View Slide

24. GROUP MESSAGE
    ग़㮆Userਂ玲ݶӞ犩MessageList
    > Hashݚ夵袅ݱUserጱ蒂ቘ蝱ଶ
    > 獋㮆Unsigned Short಑۱Head, Tail
    24
    

    View Slide

25. BUT...
    > 瞱ԋ玕&盋ܻ
    稲ݥ懱௳ጱ礚扇౲Redis䥁娄蝨౮ጱ०硻
    > 瞙稞蒂ቘ
    ࢩၚ㵕ᘒ叨ኞ碍苭㳷ጱ夺۱懱௳
    瞙稞蒂ቘӞ䰬䨝磪Retry缛㺔氂
    綍ᛗ瞙稞Ӿጱ蟂獤०硻ᥝݚᤈ蒂ቘ
    獊蟂᯿蝑䨝虏य़蟂獤౮ۑጱ懱௳ڊ匍獋稞
    25
    

    View Slide

26. ٚ℄፜Ӟ犚懿䛂誢: MSGP
    26
    

    View Slide

27. MORE CHALLENGE NEXT
    Redis奰绗犋ฎ䌕槹㬵蒂ቘQueue
    褰茐禂率౮裾ᘒᛗӥӞ褩ྦྷԏ獮
    ᴻԧ薪疗匍褩ྦྷ狶ဩಅ胼ک螈ጱ禅褖
    稭蜰䨝盅Ӟ蚏獤Ձݱ圵犋ݶጱ狶ဩ
    27
    

    View Slide

28. Q & A
    

    View Slide