Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Redis Lua Script With Red Envelope & Message Queue

Redis Lua Script With Red Envelope & Message Queue

Golang Taiepi Meetup #41

Cf6d8219ae9edc7620396d0f8ae7bc38?s=128

Yun Chen

April 30, 2019
Tweet

Transcript

  1. REDIS LUA SCRIPT WITH RED ENVELOPE & MESSAGE QUEUE GOLANG

    TAIPEI MEETUP #41 https://github.com/chenyunchen/K8S-Meetup
  2. WHO AM I Chen Yun Chen (Alex) > me@yunchen.tw >

    blog.yunchen.tw Experience > Backend Engineer at JKOPay > Software Engineer at Linker Networks
  3. Ջ讕ฎLUA? Lua斕ᰁ承᥺牧ݝ۱珀羊墋ጱ໐ஞ޾च๜Library > 匍磪礍䯤ӥ൉׀狕硬皤֐ۑ胼Ӟਧጱ覄ၚ௔ > Redis, Nginx舙襑ਮ蕣ݢ્獈Lua狶೐疻(籆ๅ碝) 諑劭Ӯኴ㮆ՈUIՕᶎጱਮ蕣玕(Lua) https://wowwiki.fandom.com/wiki/Lua 3

  4. ፓ獮傶Ջ讕襑ᥝֵአ REDIS+LUA ? > ૪磪य़ᰁֵአRedis狶傶Cache牏Queue䁰ว > ၚ㵕襑缏ܨℂ匍磪礍䯤ӥ䋿匍夺۱牏懱௳ۑ胼 > ᤍᰁ姘螀牏碍硁ᰁ牏ݢ覎௔ฎ玽ᥝ疩獈ই:Kafka >

    疩獈碝๐率ᛗ绐ਧጱ碻樌现觓檺 (Lua姘虁500ᤈૢݦ牧薹究蟂獤襑穩) 4
  5. REDIS > 㻌䁆ᤈ姼(Single-Thread) > ܻৼ௔(Atomic Operation) IF CLUSTER & MULTIPLE

    KEYS COMMAND > node:{nodeID}:userID (Hash Tag) 5
  6. REDIS-MULTI > 蝚螂Ӟ稞䁆ᤈग़㮆瞲犤 > 狒挨䁆ᤈ殼ଧ 1. ग़執砺֢樄衴 瞲犤樌ጱ虻碘 舙ஂ種襑ᥝ狶墋㻌蒂ቘ Ֆ襑碍稞䌘Redis礚扇膏捝䌃

    2. 虻碘ਠ碉௔ 舙磪ग़稞ጱ礚扇膏捝䌃 ׎櫞犥嘦狒ಅ磪砺֢緡౮ۑ 6
  7. REDIS-LUA > 瞲犤䁆ᤈݢ覎ଶ斃ṛ > ࣁग़執Key樌蝱ᤈValue蒂ቘ BUT... 1. 篷ဩ狒挨Redis吚秚盅ጱਠ碉௔ 2. Script๜蛪䌃ဩ磪藮篷ဩࢧ笔

    7
  8. GOLANG > 斉獈脻๜ 疥碉犩෈䲆斉獈Redis叨ኞSHA1 Checksum > 癲獈KEYS, ARGV KEYS: ݝ硯ᥝ砺֢ጱKEY

    ARGV: ᒫӞ㮆硯ਧ嬝ጱ砺֢ᤈ傶牧ٌ訏ஃ盅 硎 8
  9. LUA: GOLANG options := redis.Options{ Network: "tcp4", Addr: "127.0.0.1:6379", DB:

    0, PoolSize: 10, } client := redis.NewClient(&options) fileBytes, err := ioutil.ReadFile("path") if err != nil { return } sha, err := client.ScriptLoad(string(fileBytes)).Result() if err != nil { return } keys := []string{"redenvelopeID"} args := []string{"get_redenvelope", "userID"} res, err := client.EvalSha(sha, keys, args).Result() 9
  10. LUA: DEBUG ./redis-cli --ldb --eval script.lua mykey somekey , arg1

    arg2 More Detail: https://redis.io/topics/ldb 10
  11. 犥䋿匍夺۱傶ֺ 咳夺۱ᤈ傶 > አ䜛戔ਧᰂ氃牏碍ᰁ > 獤蟴褰秚ᰂ氃ጱग़㮆夺۱ 硩夺۱ᤈ傶 > 嘦藨አ䜛ฎ玽磪砋螂夺۱ >

    覿ڊӞ㮆褰秚夺۱妔አ䜛 > 疥扗硯۱硯獈૪窞揲ጱ褧ڜ 11
  12. 毆ض叨ኞ夺۱ > 究ਧᰂ氃ጱ獤蟴秚ګ (ex: 1000ز 獤蟴 10۱) 50% 60% 70%

    80% 90% 99% [30 41.11 52.22 63.33 74.44 125.56 136.67 147.78 158.89 170] > 矎ଧ(褰秚Ի矦) or 藶穩瞲窔 [136.67 158.89 41.11 170 74.44 147.78 125.56 30 52.22 63.33] 12
  13. ח扖藯஑অ: ᓒ桽አၶ讨牧螛෱ᤩՈ瞇 var money1 float32 = 9.9 fmt.Println(money1 * 100)

    // 989.99994 var money2 float64 = 1.1 fmt.Println(money2 * 100) // 110.00000000000001 13
  14. 羊伛ଶ蔭纈 > 2蝱ګ(IEEE-754) (膏ֵአ聜䙼犋ݶ) > 10蝱ګ (Ֆ䨝०羊伛ଶ) 疰ᓒ纷ୗ胼ᛔ㵕犚盏狕ྋ 褰茐蕦褾ጱࢥ㳷螀ᓒ懯ᓒ牧磧奰奾ຎՖ䨝ڊ㺔氂 14

  15. 砋夺۱ 1. ฎ玽磪砋螂夺۱HashMap 2. Ֆ磪夺۱׎ᑏ獈૪窞揲 3. አ䜛膏夺۱ጱ橕狌ਂࢧ HashMap > Existے蝧玲஑አ䜛膏夺۱橕狌

    > Consume஑Ꭳ糷玲殼ଧ牏ڜ蔭 > Lua䋿֢1~3 狒挨ܻৼ௔ 15
  16. LUA: 砋夺۱ local function getRedenvelope(rID, userID) if redis.call('hexists', rID, userID)

    ~= 0 then return nil; else local redenvelope = redis.call('rpop', 'redenvelope'); if redenvelope then local r = cjson.decode(redenvelope); r['userID'] = tonumber(userID); local re = cjson.encode(r); redis.call('hset', rID, userID, userID); redis.call('lpush', 'consume_' .. rID, re); return re; end end return nil; end 16
  17. DEMO SEND & RECEIVE RED ENVELOPE

  18. BUT... RETRY & ACK 1. Worker 篷ဩ狒挨玲አRedis虻碘ጱWorker 100%౮ۑ ݢ胼襑ᥝᛔᤈ蒂ቘӞ犚奞℄: ๚蒂ቘ牏蒂ቘӾ牏૪蒂ቘ

    2. አ䜛ᒒ 苭Ӟአ䜛ᒒ䷱硩ک夺۱蝢Ꭳ䨝ெ讕䰬? 3. ䷱硩ک夺۱蝢Ꭳ吚簁疰100%砋犋ک 4. 虋๚䋿匍ᰂ氃蜴ࢧ咳硯ጱአ䜛癱蒈 18
  19. ℂ夺۱獤蟴ک蝢Ꭳ 夺۱牏懱௳๐率ጱ瞥獤 > 虻რ藲ଶӤጱᘍᰁ > ०硻Retryጱ㺔氂 19

  20. 懱௳褧ڜ 螨ع矾疨&ے蝧捝玲 User᮷磪ᛔ૩ጱQueueᥝ姘虁 褰茐አ䜛碍/虻碘ᰁɁ 仂੝懿ࣁग़執虻碘樌ጱਂ玲 20

  21. 犥䋿匍懱௳褧ڜ傶ֺ List > 膏夺۱砺֢Ӟ膌牧墋㻌ᓕቘ > 㰍ᔱ୚㬵瞲ਧ牧篷messageID > Value: String Data

    Bit Manipulation > 㻌ӞKEY:Value珀ಅ襑橕狌獉਻ > ᛔᤈ姘虁虻碘bytes裾ଶ/֖ᑏ > Value: Binary Data 21
  22. LUA: MESSAGE HEADER local function getHeader(name) return redis.call('getrange', name, 0,

    7); end local function parseHeader(header) return struct.unpack('I4HH', header); end local function setHeader(name, lastID, head, tail) local header = struct.pack('I4HH', lastID, head, tail); redis.call('setrange', name, 0, header); end 22
  23. DEMO PUSH, GET & FINISH MESSAGE

  24. GROUP MESSAGE ग़㮆Userਂ玲ݶӞ犩MessageList > Hashݚ夵袅ݱUserጱ蒂ቘ蝱ଶ > 獋㮆Unsigned Short಑۱Head, Tail 24

  25. BUT... > 瞱ԋ玕&盋ܻ 稲ݥ懱௳ጱ礚扇౲Redis䥁娄蝨౮ጱ०硻 > 瞙稞蒂ቘ ࢩၚ㵕ᘒ叨ኞ碍苭㳷ጱ夺۱懱௳ 瞙稞蒂ቘӞ䰬䨝磪Retry缛㺔氂 綍ᛗ瞙稞Ӿጱ蟂獤०硻ᥝݚᤈ蒂ቘ 獊蟂᯿蝑䨝虏य़蟂獤౮ۑጱ懱௳ڊ匍獋稞

    25
  26. ٚ℄፜Ӟ犚懿䛂誢: MSGP 26

  27. MORE CHALLENGE NEXT Redis奰绗犋ฎ䌕槹㬵蒂ቘQueue 褰茐禂率౮裾ᘒᛗӥӞ褩ྦྷԏ獮 ᴻԧ薪疗匍褩ྦྷ狶ဩಅ胼ک螈ጱ禅褖 稭蜰䨝盅Ӟ蚏獤Ձݱ圵犋ݶጱ狶ဩ 27

  28. Q & A