Introduction to Redis Modules

Transcript

1. REDIS: Need for speed @elena_kolevska Elena Kolevska 1 About Our Company Redis Modules Elena Kolevska
2. Who am I? A nomad earthling | Self employed developer BLOG.ELENAKOLEVSKA.COM @ELENA_KOLEVSKA
3. (Really) SHORT INTRO No, seriously!
4. REDIS IS AN OPEN SOURCE (BSD licensed), IN-MEMORY DATA STRUCTURE STORE, USED AS DATABASE, CACHE AND MESSAGE BROKER The 'Definition' on redis.io
5. • Different data structures • Keys with a limited time-to-live • Lua scripting • Transactions • Pipelining • Pub/Sub • Built-in replication • Different levels of on-disk persistence BASIC FEATURES:
6. SPEED
7. AVAILABLE CLIENTS IN: ActionScript Common Erlang Haxe Matlab Pascal Racket Smalltalk bash Crystal Fancy Io mruby Perl Rebol Swift C D gawk Java Nim PHP Ruby Tcl C# Dart GNUProlog Javascript Nodejs PureData Rust VB C++ Elixir Go Julia ObjectiveC Python Scala VCL Clojure Emacs lisp Haskell Lua OCaml R Scheme
8. AVAILABLE DATA STRUCTURES: •STRINGS (Binary safe, can be anything from "hello world" to a jpeg file) •LISTS (Collections of string elements sorted according to the order of insertion) •SETS (Collections of unique, unsorted string elements) •SORTED SETS (It's like Sets with a score) •HASHES (Objects. Maps of fields associated to values. Think non-nested json objects) •BITMAPS (Manipulate Strings on a bit level) •HYPERLOGLOGS (Probabilistic data structure used to estimate the cardinality of a set)
10. Imagine ...
11. OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751 Main stream 120 sub-stream foo 107 sub-stream bar       ... Foo Bar Baz 0 30 60 90 120 86 107 120
12. OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751 Main stream 120 sub-stream foo 107 sub-stream bar       ... Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets
13. OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751 Main stream 120 sub-stream foo 107 sub-stream bar       ... Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets Main Stream
14. OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751 Main stream 120 sub-stream foo 107 sub-stream bar       ... Substream:keyword 1 Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets Main Stream
15. OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751 Main stream 120 sub-stream foo 107 sub-stream bar       ... Substream:keyword 1 Substream:keyword 2 Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets Main Stream
16. OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751 Main stream 120 sub-stream foo 107 sub-stream bar       ... Substream:keyword 1 Substream:keyword 2 Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets Scores Main Stream
17. [1] CONNECTING TO REDIS
18. [1] CONNECTING TO REDIS Install the PRedis package using Composer > composer require predis/predis Initialise the client $parameters = [ 'scheme' => 'tcp', 'host' => '127.0.0.1', 'port' => 6379 ]; $client = new Predis\Client($parameters, ['prefix' => 'stats:']);
19. [2] SET TRACKED DATA
20. [2] SET TRACKED DATA Use sets to store all the hashtags we'll be looking at and all the keywords as well $client->sadd('hashtags', 'thestatscompany','statscompany', 'statscampaign'); // hashtags | 'thestatscompany' // | 'statscompany' // | 'statscampaign' $client->sadd('keywords', 'loved', 'amazed', 'frustrated', 'problem', 'terrible', 'amazing', 'great');
21. [3] GET THE DATA
22. [3] GET THE DATA Use Twitter Stream API to receive notifications for tweets containing any of the hashtags we're following $hashtags = $client->smembers('hashtags'); // array (size=3) // 0 => string 'thestatscompany' (length=15) // 1 => string 'statscompany' (length=12) // 2 => string 'statscampaign' (length=13)
23. [3] GET THE DATA Use Twitter Stream API to receive notifications for tweets containing any of the hashtags we're following $hashtags = $client->smembers('hashtags'); // array (size=3) // 0 => string 'thestatscompany' (length=15) // 1 => string 'statscompany' (length=12) // 2 => string 'statscampaign' (length=13)
24. And then push to a queue to be processed asynchronously Save every new tweet from the stream as a separate string: $keyname = 'tweet_id:' . $tweet_id; $tweet_contents = "Amazed by the customer service of #statscompany"; $client->set($keyname, $tweet_contents); $client->lpush('message_queue', $keyname); // 'message_queue' | 'tweet_id:45645656' // | 'tweet_id:44645234' // | 'tweet_id:43645232'
25. And then push to a queue to be processed asynchronously Save every new tweet from the stream as a separate string: $keyname = 'tweet_id:' . $tweet_id; $tweet_contents = "Amazed by the customer service of #statscompany"; $client->set($keyname, $tweet_contents); $keyname = 'tweet_id:' . $tweet_id; $tweet_contents = "Amazed by the customer service of #statscompany"; $client->set($keyname, $tweet_contents); $client->lpush('message_queue', $keyname); // 'message_queue' | 'tweet_id:45645656' // | 'tweet_id:44645234' // | 'tweet_id:43645232'
26. And then push to a queue to be processed asynchronously Save every new tweet from the stream as a separate string: $keyname = 'tweet_id:' . $tweet_id; $tweet_contents = "Amazed by the customer service of #statscompany"; $client->set($keyname, $tweet_contents); $client->lpush('message_queue', $keyname); // 'message_queue' | 'tweet_id:45645656' // | 'tweet_id:44645234' // | 'tweet_id:43645232'
27. And then push to a queue to be processed asynchronously Save every new tweet from the stream as a separate string: $keyname = 'tweet_id:' . $tweet_id; $tweet_contents = "Amazed by the customer service of #statscompany"; $client->set($keyname, $tweet_contents); $client->lpush('message_queue', $keyname); // 'message_queue' | 'tweet_id:45645656' // | 'tweet_id:44645234' // | 'tweet_id:43645232' $client->lpush('message_queue', $keyname); // 'message_queue' | 'tweet_id:45645656' // | 'tweet_id:44645234' // | 'tweet_id:43645232'
28. [4] WORKER TO PROCESS THE QUEUED JOBS
29. [4] WORKER TO PROCESS THE QUEUED JOBS A separate worker will be grabbing jobs off the top of the queue and processing them: $message_queue = $client->rpop('message_queue'); Reliable queue: RPOPLPUSH, BRPOPLPUSH Blocking queue: BLPOP, BRPOP
30. [4] WORKER TO PROCESS THE QUEUED JOBS A separate worker will be grabbing jobs off the top of the queue and processing them: $message_queue = $client->rpop('message_queue'); $message_queue = $client->rpop('message_queue'); Reliable queue: RPOPLPUSH, BRPOPLPUSH Blocking queue: BLPOP, BRPOP
31. [4] WORKER TO PROCESS THE QUEUED JOBS A separate worker will be grabbing jobs off the top of the queue and processing them: $message_queue = $client->rpop('message_queue'); Reliable queue: RPOPLPUSH, BRPOPLPUSH Blocking queue: BLPOP, BRPOP
32. [5] PROCESS THE TWEET CONTENT
33. [5] PROCESS THE TWEET CONTENT $tweet_contents = $client->get($keyname); $keywords = $client->smembers('keywords'); foreach ($keywords as $keyword) { $tweet_contents = strtolower($tweet_contents); $keyword = strtolower($keyword); if (strpos($tweet_contents,$keyword) !== false){ $client->zincrby('mention_counter', 1, $keyword); $keyword_feed_keyname = 'keyword_feeds:'. $keyword; $client->lpush($keyword_feed_keyname, $tweet_contents); $client->ltrim($keyword_feed_keyname, 0, 50); } } $client->incr('total_count'); // Increase the general tweet count $client->lpush('main_feed', $tweet_contents); $client->ltrim('main_feed', 0, 100);
34. [6] SHOW THE STATS
35. [6] SHOW THE STATS $total_count = $client->get('total_count'); // 'total_count' | 259 $scores = $client->zrevrangebyscore('mention_counter', '+inf', '-inf', ['withscores'=>1]); // mention_counter | 'amazed' => 9.00 // | 'frustrated' => 5.00 // | 'love' => 4.00 // Feed by keyword foreach ($scores as $keyname => $score) { $keyword_feeds[$keyname] = $client->lrange('keyword_feeds:' . $keyname, 0, -1); } // Feed of all tweets containing one of the specified hashtags $main_feed = $client->lrange('main_feed', 0, -1);
36. [6] SHOW THE STATS $total_count = $client->get('total_count'); // 'total_count' | 259 $scores = $client->zrevrangebyscore('mention_counter', '+inf', '-inf', ['withscores'=>1]); // mention_counter | 'amazed' => 9.00 // | 'frustrated' => 5.00 // | 'love' => 4.00 // Feed by keyword foreach ($scores as $keyname => $score) { $keyword_feeds[$keyname] = $client->lrange('keyword_feeds:' . $keyname, 0, -1); } // Feed of all tweets containing one of the specified hashtags $main_feed = $client->lrange('main_feed', 0, -1);
37. [7] CAN WE MAKE IT FASTER?
38. REDIS: Need for speed @elena_kolevska Elena Kolevska 25 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: •The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response •The server processes the command and sends the response back to the client Client Server
39. REDIS: Need for speed @elena_kolevska Elena Kolevska 25 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: •The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response •The server processes the command and sends the response back to the client Client Server Networking link
40. REDIS: Need for speed @elena_kolevska Elena Kolevska 25 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: •The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response •The server processes the command and sends the response back to the client Client Server
41. REDIS: Need for speed @elena_kolevska Elena Kolevska 25 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: •The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response •The server processes the command and sends the response back to the client Client Server Request from client to server
42. REDIS: Need for speed @elena_kolevska Elena Kolevska 25 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: •The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response •The server processes the command and sends the response back to the client Client Server Request from client to server Response from server to client
43. Redis is a TCP server using the client/server model. A request to the server is accomplished in two steps: •The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response •The server processes the command and sends the response back to the client Client Server Request from client to server Response from server to client RTT: Round Trip Time
44. THE STEPS: •Save the complete tweet •Get all monitored keywords •Check tweet for any keywords •Increase counter for total tweets •Add tweet to main feed •Increase counter for found keywords •Add tweet to sub-feed
45. THE STEPS: SET request to the server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score
46. THE STEPS: SET request to the server 1 Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score
47. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 1
48. THE STEPS: LPUSH Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 1
49. THE STEPS: LPUSH Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 2
50. THE STEPS: LPUSH Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score + LTRIM 2
51. THE STEPS: LPUSH Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score + LTRIM 3
52. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 3
53. THE STEPS: INCR Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 3
54. THE STEPS: INCR Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 4
55. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 4
56. THE STEPS: SMEMBERS Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 4
57. THE STEPS: SMEMBERS Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 5
58. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 5
59. THE STEPS: Logic on client side Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 5
60. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 5
61. THE STEPS: INCR Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 5
62. THE STEPS: INCR Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 6
63. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 6
64. THE STEPS: LPUSH Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 6
65. THE STEPS: LPUSH Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 7
66. THE STEPS: LPUSH Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 7 + LTRIM
67. THE STEPS: LPUSH Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score + LTRIM 8
68. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 8
69. THE STEPS: ZINCRBY Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 8
70. THE STEPS: ZINCRBY Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 9
71. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 9
72. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 9
73. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 9 } + 4 requests for every found keyword
74. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score 13? } + 4 requests for every found keyword
75. THE STEPS: Number of requests to server •Save the tweet •Add tweet to main feed •Increase counter for total tweets •Get all monitored keywords •Check tweet for any keywords •Increase counter for found keywords •Add tweet to sub-feed •Update score } + 4 requests for every found keyword 17?
76. [WARP 8] PIPELINING
77. Using the PRedis library for PHP </php $keywords = $client->smembers("keywords"); $responses = $client->pipeline(function ($pipe) use ($keywords) { $pipe->set(...); // Save the tweet $pipe->lpush(...); // Save the tweet to main feed $pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever foreach ($keywords as $keyword) { $pipe->incr(...); // Increase the counter for the keyword $pipe->lpush(...); // Add to the subbed $pipe->ltrim(...); // Trim the stream to X tweets $pipe->zincrby(...); // Update score } });
78. Using the PRedis library for PHP </php $keywords = $client->smembers("keywords"); $responses = $client->pipeline(function ($pipe) use ($keywords) { $pipe->set(...); // Save the tweet $pipe->lpush(...); // Save the tweet to main feed $pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever foreach ($keywords as $keyword) { $pipe->incr(...); // Increase the counter for the keyword $pipe->lpush(...); // Add to the subbed $pipe->ltrim(...); // Trim the stream to X tweets $pipe->zincrby(...); // Update score } });
79. Using the PRedis library for PHP 1 Number of requests to server </php $keywords = $client->smembers("keywords"); $responses = $client->pipeline(function ($pipe) use ($keywords) { $pipe->set(...); // Save the tweet $pipe->lpush(...); // Save the tweet to main feed $pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever foreach ($keywords as $keyword) { $pipe->incr(...); // Increase the counter for the keyword $pipe->lpush(...); // Add to the subbed $pipe->ltrim(...); // Trim the stream to X tweets $pipe->zincrby(...); // Update score } });
80. Using the PRedis library for PHP 2 Number of requests to server </php $keywords = $client->smembers("keywords"); $responses = $client->pipeline(function ($pipe) use ($keywords) { $pipe->set(...); // Save the tweet $pipe->lpush(...); // Save the tweet to main feed $pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever foreach ($keywords as $keyword) { $pipe->incr(...); // Increase the counter for the keyword $pipe->lpush(...); // Add to the subfeed $pipe->ltrim(...); // Trim the stream to X tweets $pipe->zincrby(...); // Update score } });
81. [WARP 9] LUA SCRIPTING
82. Create a hello.lua file: return "Hello" > redis-cli --eval /path/to/script/hello.lua "Hello"
83. Create a hello.lua file: return "Hello" > redis-cli --eval /path/to/script/hello.lua "Hello"
84. hello.lua: local tweet = ARGV[1] local key_tweet = KEYS[1] local key_keywords = KEYS[2] local key_total_tweets_count = KEYS[3] local key_scores = KEYS[4] local key_main_feed = KEYS[5] redis.call("SET", key_tweet, tweet) -- Save the tweet redis.call("INCR", key_total_tweets_count) -- Increase the total tweet count redis.call("LPUSH", key_main_feed, tweet) -- Push the tweet to the main feed redis.call("LTRIM", key_main_feed, 0, 100) -- Trim the main feed local keywords = redis.call("SMEMBERS", key_keywords) -- Get the keywords for i, name in ipairs(keywords) do if string.find(tweet, name) then local substream_name = "sub_feed:" .. name redis.call("LPUSH", substream_name, tweet) -- Push the tweet to the sub feed redis.call("LTRIM", substream_name, 0, 100) -- Trim the sub feed redis.call("ZINCRBY", key_scores, 1, name) -- Increment the score for the keyword in the leaderboard end end return "OK"
85. > redis-cli --eval hello.lua tweet_123 keywords ... , "A tweet about the words foo and "OK"
86. > redis-cli --eval hello.lua tweet_123 keywords ... , "A tweet about the words foo and "OK" 1! Number of requests to server
87. SCRIPT •Evaluates a script cached on the server side by its SHA1 digest. The command is otherwise identical to EVAL. > redis-cli SCRIPT LOAD "$(cat hello.lua)" "6d52847f03028ab1d4620b60dd6ef4a14c8727d7" > redis-cli evalsha 6d52847f03028ab1d4620b60dd6ef4a14c8727d7 5 \ > tweet_123 keywords ... "A tweet about the words foo and bar" "OK" EVALSHA •Load a script into the scripts cache, without executing it. Returns a SHA-1 digest of the script.
88. [WARP 10] MODULES
89. What are Redis modules? •Dynamically loaded libraries •Written in C •Almost as fast as the Redis core •Let you extend Redis commands, create new data structures, access data almost as fast as native Redis commands •Add-ons to Redis •Coming in version 4.0, currently in Beta (RC)
90. Layers of the Modules API: •Low-level: Close to native access to core data structures •High-level: Client-like access to core and modules' commands
94. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
95. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); The context object c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
96. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Command c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
97. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Arguments c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
98. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Format specifier c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
99. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString objec l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
100. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Format specifier c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString objec l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
101. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
102. High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Format specifier c -- Null terminated C string pointer. b -- C buffer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
103. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }
104. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Definitions of the API
105. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Definitions of the API
106. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Definitions of the API Called whenever the module is loaded Must be present in each Redis module
107. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Definitions of the API Register the module or error out
108. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Definitions of the API Register the command or error out
109. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Definitions of the API Command name
110. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Definitions of the API Function name
111. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }
112. Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } The logic of the module
113. How much faster does it get? Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // We need EXACTLY 5 arguments if (argc != 5) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *reply; reply = RedisModule_Call(ctx, "INCR", "s", argv[3]); RedisModule_ReplyWithCallReply(ctx, reply); return REDISMODULE_OK; }
114. How much faster does it get? Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // We need EXACTLY 5 arguments if (argc != 5) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *reply; reply = RedisModule_Call(ctx, "INCR", "s", argv[3]); RedisModule_ReplyWithCallReply(ctx, reply); return REDISMODULE_OK; } Enable automatic memory management
115. How much faster does it get? Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // We need EXACTLY 5 arguments if (argc != 5) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *reply; reply = RedisModule_Call(ctx, "INCR", "s", argv[3]); RedisModule_ReplyWithCallReply(ctx, reply); return REDISMODULE_OK; } Call Redis commands
116. How much faster does it get? Anatomy of a module #include "redismodule.h" #include <stdlib.h> int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // We need EXACTLY 5 arguments if (argc != 5) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *reply; reply = RedisModule_Call(ctx, "INCR", "s", argv[3]); RedisModule_ReplyWithCallReply(ctx, reply); return REDISMODULE_OK; } Reply with a call object
117. Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key);
118. Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key); Open a key. Return a key pointer
119. Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key); Checks the type of the key
120. Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key); Set a string value for a key
121. Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key); Close the key
122. Compiling $ gcc -fPIC -std=gnu99 -c -o process_tweet process_tweet.c $ ld -o process_tweet.so process_tweet.o -shared -Bsymbolic -lc On Linux: $ gcc -dynamic -fno-common -std=gnu99 -c -o process_tweet.o process_tweet.c $ ld -o process_tweet.so process_tweet.o -bundle -undefined dynamic_lookup -lc On OSX: Loading ./redis-unstable/src/redis-server --loadmodule ./process_tweet.so
123. Calling the command 127.0.01:6379> tweet.process tweet_id:42 "This is my tweet about the words foo and bar" ... Our command
124. Thank you ! @elena_kolevska