Redis Lua Script With Red Envelope & Message Queue

Golang Taiepi Meetup #41

Transcript

1. REDIS LUA SCRIPT WITH RED ENVELOPE & MESSAGE QUEUE GOLANG TAIPEI MEETUP #41 https://github.com/chenyunchen/K8S-Meetup
2. WHO AM I Chen Yun Chen (Alex) > me@yunchen.tw > blog.yunchen.tw Experience > Backend Engineer at JKOPay > Software Engineer at Linker Networks
3. Ջ讕ฎLUA? Lua斕ᰁ承᥺牧ݝ۱珀羊墋ጱ໐ஞ޾च๜Library > 匍磪礍䯤ӥ൉׀狕硬皤֐ۑ胼Ӟਧጱ覄ၚ௔ > Redis, Nginx舙襑ਮ蕣ݢ્獈Lua狶೐疻(籆ๅ碝) 諑劭Ӯኴ㮆ՈUIՕᶎጱਮ蕣玕(Lua) https://wowwiki.fandom.com/wiki/Lua 3
4. ፓ獮傶Ջ讕襑ᥝֵአ REDIS+LUA ? > ૪磪य़ᰁֵአRedis狶傶Cache牏Queue䁰ว > ၚ㵕襑缏ܨℂ匍磪礍䯤ӥ䋿匍夺۱牏懱௳ۑ胼 > ᤍᰁ姘螀牏碍硁ᰁ牏ݢ覎௔ฎ玽ᥝ疩獈ই:Kafka > 疩獈碝๐率ᛗ绐ਧጱ碻樌现觓檺 (Lua姘虁500ᤈૢݦ牧薹究蟂獤襑穩) 4
5. REDIS > 㻌䁆ᤈ姼(Single-Thread) > ܻৼ௔(Atomic Operation) IF CLUSTER & MULTIPLE KEYS COMMAND > node:{nodeID}:userID (Hash Tag) 5
6. REDIS-MULTI > 蝚螂Ӟ稞䁆ᤈग़㮆瞲犤 > 狒挨䁆ᤈ殼ଧ 1. ग़執砺֢樄衴 瞲犤樌ጱ虻碘 舙ஂ種襑ᥝ狶墋㻌蒂ቘ Ֆ襑碍稞䌘Redis礚扇膏捝䌃 2. 虻碘ਠ碉௔ 舙磪ग़稞ጱ礚扇膏捝䌃 ׎櫞犥嘦狒ಅ磪砺֢緡౮ۑ 6
7. REDIS-LUA > 瞲犤䁆ᤈݢ覎ଶ斃ṛ > ࣁग़執Key樌蝱ᤈValue蒂ቘ BUT... 1. 篷ဩ狒挨Redis吚秚盅ጱਠ碉௔ 2. Script๜蛪䌃ဩ磪藮篷ဩࢧ笔 7
8. GOLANG > 斉獈脻๜ 疥碉犩෈䲆斉獈Redis叨ኞSHA1 Checksum > 癲獈KEYS, ARGV KEYS: ݝ硯ᥝ砺֢ጱKEY ARGV: ᒫӞ㮆硯ਧ嬝ጱ砺֢ᤈ傶牧ٌ訏ஃ盅 硎 8
9. LUA: GOLANG options := redis.Options{ Network: "tcp4", Addr: "127.0.0.1:6379", DB: 0, PoolSize: 10, } client := redis.NewClient(&options) fileBytes, err := ioutil.ReadFile("path") if err != nil { return } sha, err := client.ScriptLoad(string(fileBytes)).Result() if err != nil { return } keys := []string{"redenvelopeID"} args := []string{"get_redenvelope", "userID"} res, err := client.EvalSha(sha, keys, args).Result() 9
10. LUA: DEBUG ./redis-cli --ldb --eval script.lua mykey somekey , arg1 arg2 More Detail: https://redis.io/topics/ldb 10
11. 犥䋿匍夺۱傶ֺ 咳夺۱ᤈ傶 > አ䜛戔ਧᰂ氃牏碍ᰁ > 獤蟴褰秚ᰂ氃ጱग़㮆夺۱ 硩夺۱ᤈ傶 > 嘦藨አ䜛ฎ玽磪砋螂夺۱ > 覿ڊӞ㮆褰秚夺۱妔አ䜛 > 疥扗硯۱硯獈૪窞揲ጱ褧ڜ 11
12. 毆ض叨ኞ夺۱ > 究ਧᰂ氃ጱ獤蟴秚ګ (ex: 1000ز 獤蟴 10۱) 50% 60% 70% 80% 90% 99% [30 41.11 52.22 63.33 74.44 125.56 136.67 147.78 158.89 170] > 矎ଧ(褰秚Ի矦) or 藶穩瞲窔 [136.67 158.89 41.11 170 74.44 147.78 125.56 30 52.22 63.33] 12
13. ח扖藯஑অ: ᓒ桽አၶ讨牧螛෱ᤩՈ瞇 var money1 float32 = 9.9 fmt.Println(money1 * 100) // 989.99994 var money2 float64 = 1.1 fmt.Println(money2 * 100) // 110.00000000000001 13
14. 羊伛ଶ蔭纈 > 2蝱ګ(IEEE-754) (膏ֵአ聜䙼犋ݶ) > 10蝱ګ (Ֆ䨝०羊伛ଶ) 疰ᓒ纷ୗ胼ᛔ㵕犚盏狕ྋ 褰茐蕦褾ጱࢥ㳷螀ᓒ懯ᓒ牧磧奰奾ຎՖ䨝ڊ㺔氂 14
15. 砋夺۱ 1. ฎ玽磪砋螂夺۱HashMap 2. Ֆ磪夺۱׎ᑏ獈૪窞揲 3. አ䜛膏夺۱ጱ橕狌ਂࢧ HashMap > Existے蝧玲஑አ䜛膏夺۱橕狌 > Consume஑Ꭳ糷玲殼ଧ牏ڜ蔭 > Lua䋿֢1~3 狒挨ܻৼ௔ 15
16. LUA: 砋夺۱ local function getRedenvelope(rID, userID) if redis.call('hexists', rID, userID) ~= 0 then return nil; else local redenvelope = redis.call('rpop', 'redenvelope'); if redenvelope then local r = cjson.decode(redenvelope); r['userID'] = tonumber(userID); local re = cjson.encode(r); redis.call('hset', rID, userID, userID); redis.call('lpush', 'consume_' .. rID, re); return re; end end return nil; end 16
17. DEMO SEND & RECEIVE RED ENVELOPE
18. BUT... RETRY & ACK 1. Worker 篷ဩ狒挨玲አRedis虻碘ጱWorker 100%౮ۑ ݢ胼襑ᥝᛔᤈ蒂ቘӞ犚奞℄: ๚蒂ቘ牏蒂ቘӾ牏૪蒂ቘ 2. አ䜛ᒒ 苭Ӟአ䜛ᒒ䷱硩ک夺۱蝢Ꭳ䨝ெ讕䰬? 3. ䷱硩ک夺۱蝢Ꭳ吚簁疰100%砋犋ک 4. 虋๚䋿匍ᰂ氃蜴ࢧ咳硯ጱአ䜛癱蒈 18
19. ℂ夺۱獤蟴ک蝢Ꭳ 夺۱牏懱௳๐率ጱ瞥獤 > 虻რ藲ଶӤጱᘍᰁ > ०硻Retryጱ㺔氂 19
20. 懱௳褧ڜ 螨ع矾疨&ے蝧捝玲 User᮷磪ᛔ૩ጱQueueᥝ姘虁 褰茐አ䜛碍/虻碘ᰁɁ 仂੝懿ࣁग़執虻碘樌ጱਂ玲 20
21. 犥䋿匍懱௳褧ڜ傶ֺ List > 膏夺۱砺֢Ӟ膌牧墋㻌ᓕቘ > 㰍ᔱ୚㬵瞲ਧ牧篷messageID > Value: String Data Bit Manipulation > 㻌ӞKEY:Value珀ಅ襑橕狌獉਻ > ᛔᤈ姘虁虻碘bytes裾ଶ/֖ᑏ > Value: Binary Data 21
22. LUA: MESSAGE HEADER local function getHeader(name) return redis.call('getrange', name, 0, 7); end local function parseHeader(header) return struct.unpack('I4HH', header); end local function setHeader(name, lastID, head, tail) local header = struct.pack('I4HH', lastID, head, tail); redis.call('setrange', name, 0, header); end 22
23. DEMO PUSH, GET & FINISH MESSAGE
24. GROUP MESSAGE ग़㮆Userਂ玲ݶӞ犩MessageList > Hashݚ夵袅ݱUserጱ蒂ቘ蝱ଶ > 獋㮆Unsigned Short಑۱Head, Tail 24
25. BUT... > 瞱ԋ玕&盋ܻ 稲ݥ懱௳ጱ礚扇౲Redis䥁娄蝨౮ጱ०硻 > 瞙稞蒂ቘ ࢩၚ㵕ᘒ叨ኞ碍苭㳷ጱ夺۱懱௳ 瞙稞蒂ቘӞ䰬䨝磪Retry缛㺔氂 綍ᛗ瞙稞Ӿጱ蟂獤०硻ᥝݚᤈ蒂ቘ 獊蟂᯿蝑䨝虏य़蟂獤౮ۑጱ懱௳ڊ匍獋稞 25
26. ٚ℄፜Ӟ犚懿䛂誢: MSGP 26
27. MORE CHALLENGE NEXT Redis奰绗犋ฎ䌕槹㬵蒂ቘQueue 褰茐禂率౮裾ᘒᛗӥӞ褩ྦྷԏ獮 ᴻԧ薪疗匍褩ྦྷ狶ဩಅ胼ک螈ጱ禅褖 稭蜰䨝盅Ӟ蚏獤Ձݱ圵犋ݶጱ狶ဩ 27
28. Q & A