Lua scripting for Redis
This article is based on A Speed Guide to Redis Lua Scripting from the IBM Compose blog. In this post, we’ll introduce Lua scripting for Redis, but unlike the original article, we’ve made it so all the commands work with IBM Cloud Databases for Redis.
Lua is a language which has been around since 1993. Its origins in engineering made for a compact language which could be embedded in other applications. It’s been embedded in applications as diverse as World of Warcraft and the Nginx web server. And Redis, which is why we are here. If you’ve already got your Databases for Redis database set up, along with the IBM Cloud CLI and tools, skip over this next section.
Connecting to Redis
We’re running the IBM Cloud CLI with the IBM Cloud Databases plugin. You can find out how to install them in Command Line Tools for Redis. The
cxn subcommand looks up the database name and the
-s flag says “Start an interactive shell.” We also pass the password with the
-p flag. The plugin works out what command and configuration are needed and runs the command. For Databases for Redis, that’s the
redli command, which you can download here. So, once you’ve installed the IBM Cloud CLI, IBM Cloud Databases plugin, and redli, you can get an interactive shell quickly. If you create a database called “MyRedis,” all you need to run is:
ibmcloud cdb cxn MyRedis -s
It’ll prompt you for a password. Then you’ll meet the interactive shell:
Connected to 4.0.10 >
You can get to exploring Redis now—or in this case, Redis Lua scripts.
What does Redis let you do with Lua?
It lets you create your own scripted extensions to the Redis database. That means that with Redis, you can execute Lua scripts like this:
> EVAL 'local val="Hello IBM Cloud" return val' 0 "Hello Compose"
The string after the EVAL is the Lua script:
local val="Hello IBM Cloud" return val
So at its simplest, you can run Lua scripts. But more importantly, you can run Lua scripts that act like an intelligent transaction. You can handle errors smartly, so instead of just rolling back, you can carry on processing. Of course, the intelligence of the transaction will be up to you.
But to start tapping into that power, you’ll probably want to pass the script some keys and arguments.
Keys and arguments?
The 0 at the end of the EVAL is the number of keys being passed to the Lua code; in that example, there were none. But if instead of 0 it was
2 foo bar fizz buzz, then the first two items in the list,
bar, would be passed as keys and
buzz would be arguments.
If you do pass keys, they are available to the Lua script in the KEYS table (a table is Lua’s associative array, which also is used as an 1-based array). If you have arguments, they appear in the ARGV table. For example:
return ARGV..' '..KEYS
.. is Lua’s string concatenation operator—this returns whatever the argument is concatenated with a space and then the key name given in the arguments.
> EVAL "return ARGV..' '..KEYS" 1 name:first "Hello" "Hello name:first"
No magic is applied to the KEYS, they are just strings so we still have to look up their value.
And how do you call Redis from inside Lua?
We can get at Redis’s functions through the
redis.call() command. If we use this script:
return ARGV.." "..redis.call("get",KEYS)
And then we EVAL it, assuming we’ve
SET the key
name:first to something, we’ll see this:
> EVAL 'return ARGV.." "..redis.call("get",KEYS)' 1 name:first "Hello" "Hello Brian"
(Note: If you get
attempt to concatenate a boolean value as a response, then it’s likely you haven’t set the
name:first key to a value).
So now, in a script, we’ve taken a parameter, looked up a key’s value, and created a string and returned that as a result.
This EVAL command is going to get pretty messy isn’t it?
Yes. That’s why there are other ways to get Lua scripts up to the server. The one we like is by using the command line arguments of the
redis-cli) command. Just write your “more complex” Lua script:
local name=redis.call("get", KEYS) local greet=ARGV local result=greet.." "..name return result
We’ll save that as
longhello.lua and then run this at the command line:
$ ibmcloud cdb cxn RedFive -s -p password -- --eval longhello.lua name:first , Hello "Hello Brian"
This looks a little like the command we used at the beginning to connect to the interactive shell. Apart from also passing the password with the
-p flag, this command shows another way to use the plugin. Behind the scenes, the plugin works out what command and configuration are needed to run the command. Everything it finds after the
-- then becomes arguments and flags for that command. Here, we’re add a
This the new bit,
--eval, lets you name a file to be sent up to the server and EVAL’d. So we follow that with
longhello.lua. Now our script needs keys and arguments. The keys come first and redli (and redis-cli) counts them for us; each command line parameter is a key, right up to the comma. What comes after the comma are arguments.
We can now write Lua code for Redis locally and quickly test it on the server, even a remote one.
I’m pretty good for greetings functions, got anything meatier?
Why yes, here’s a little problem for you that Lua solves nicely. Consider a situation where various divisions of a company increment different counters in the big scheme of things. So say “region:one” bumps the counter keys “count:emea,” “count:usa,” and “count:atlantic,” while “region:two” just bumps “count:usa.” These counter lists may be added to in the future, but you really want to make sure it happens all in one fell swoop. Remember what we said about this being an “intelligent transaction”? Well, here we can do all that in one script.
Let’s set up our regions as lists:
> rpush region:one count:emea count:usa count:atlantic (integer) 3 > rpush region:two "count:usa" (integer) 1
Now we’ll create a Lua script locally:
We start with a count variable—we will count all the increments we do and return that value:
local broadcast=redis.call("lrange", KEYS, 0,-1)
Here we ask Redis to give us all the values in the list that should be referred to in the first key:
for _,key in ipairs(broadcast) do
This is the opener of a Lua for loop. The ipairs function iterates through the Lua table we just got in order and we take the key from each:
And for each key, we ask Redis to increment it. Oh, and then we bump up our counter. And that’s nearly it. All that’s left is . . .
end return count
. . . to end the for loop and return the count. Save that to a file and then run it with an argument:
$ ibmcloud cdb cxn RedFive -s -p password -- --eval broadcast.lua region:one (integer) 3
And, if we go look, we find:
> mget count:usa count:atlantic count:emea 1) "1" 2) "1" 3) "1"
If we use the script on region 2:
$ ibmcloud cdb cxn RedFive -s -p password -- --eval broadcast.lua region:two (integer) 1 ... > mget count:usa count:atlantic count:emea 1) "2" 2) "1" 3) "1"
So now we’ve got ourselves a useful little function. What if there was an error? Ah, well as it stands, this script would error out too. That’s because we’re using
redis.call(), which explicitly does that. If we used
redis.pcall(), if there was an error, the error details would be returned instead and we could decide what to do. But this is a speed guide and . . .
Wait a minute, I have to upload the script every time?
No, Redis has a script cache and a command SCRIPT LOAD to just load scripts into the cache. We’ll use it from the command line here, but you can incorporate it into your applications like any other Redis command:
$ ibmcloud cdb cxn RedFive -s -p password -- SCRIPT LOAD "$(cat broadcast.lua)" "84ffc8b6e4b45af697cfc5cd83894417b7946cc1"
"$(cat broadcast.lua)" just turns our script into a quoted argument. The important bit is the number that comes back (it’s in hex). It’s the SHA1 signature of the script. We can use this to invoke the script using the
EVALSHA command like this:
> EVALSHA 84ffc8b6e4b45af697cfc5cd83894417b7946cc1 1 region:one (integer) 3
There are commands to check for scripts (
SCRIPT EXISTS) and to flush them out (
SCRIPT FLUSH) so you can manage your script loading too.
So what’s the catch?
After a particular time limit (five seconds by default), Lua scripts will start getting errors in response to queries and an error will be written to the log—the only commands available will either kill the script (
KILL SCRIPT) or shut down the server (
SHUTDOWN NOSAVE) in that situation. The five-second time limit is insanely generous as you should be writing your Lua scripts to run very quickly, in milliseconds. Why? Because while your script is running, everything else is on hold.
Hey, Lua has a lot of libraries, can I use them?
The EVAL commands documentation is where you start as it goes into detail on how Redis types are converted to Lua types. The Lua Manual covers the entire language; and remember there are versions to download that you can run outside Redis for practice. There’s also an alternative introduction from 2013 which touches on using the libraries and common gotchas too.