Object Queries with Redis

Published
//

Traditionally, many developers have opted to use relational databases when dealing with complex relationships between objects. For those of you who are used to working with relational databases like MySQL, a NoSQL database like Redis may seems like a difficult transition.

While Redis is best suited for use cases like cache, queues, and leaderboards - it is possible to use Redis in front of a MySQL database. Doing this can improve overall query performance, while still allowing you to execute complex queries.

Let's go ahead and dive deeper to see what can be achieved using Redis based queries.

Storing Object Data

Unlike traditional (RDBMS) databases, Redis is a key-value store. But it's an advanced one. There are many ways to store the values beyond a simple key/value pair, like lists or hashes. The way you choose to initially store the value will affect what you can do with it later.

The methods we've listed below is not intended to be comprehensive. It's intended to just be a starting point.

Set (SADD)

Within Redis, a set is a list of values. It may be used when a value is shared by multiple objects.

To store a member in a set, use SADD:

> SADD key member

Multiple members can be stored in a single call. We can also store product categories using a set, since several products will be under the same category.

SADD can be called this way:

> SADD product:category:1 5 10

With this command, products with an ID 5 and 10 will be stored under the key of category with an ID 1. It may seem odd to use the value (category ID) as the key. However, as you'll see in a bit, this is the best approach.

Sorted Set (ZADD)

If you plan to query a value using ranges or comparison conditions such as less than or greater than, your pick is the sorted set.

The command used to store sorted set members is ZADD:

> ZADD key score member

You may pass multiple score/member pairs, if you wish to.

Sorted set are great to store information like prices. For this situation, the command will look something like:

> ZADD product:price 49.99 1 54.90 2

We are storing the price (score) 49.90 for the product with ID 1, and 54.90 for the product with ID 2. Both are set under the product:price key.

Hash (HSET and HMSET)

When each entity has a different value for an attribute, but you still want to query it, hash is a good choice.

Call HSET to set a field using a value:

> HSET key field value

Note: Unlike the ZADD and SADD commands, HSET doesn't allow you to store multiple members (field and value) in a single call. Fortunately you can use a seperate but related command achieve that: HMSET.

Say you would like to search for products by their SKU. Since each product has a unique SKU, you can use a hash to store them by doing this:

> HSET product:sku '1ZN820' 1

Doing that, you're storing the SKU 1ZN820 for the product with an ID 1 within product:name key.

Querying Object Data

For each storage type (set, sorted set and hash) there are specific commands to fetch the keys and values. That is why its important to choose your storage method carefully, as you will be limited by the query on the other end.

Querying Sets (SMEMBERS)

Starting with the set type and the example given above, if you want to retrieve all products in a specific category - let's use the category with ID 2 - you simply call:

> SMEMBERS product:category:2

That will return a list of product ID's. With those ID's you can retrieve the product's objects.

In our specific use case, the full objects were stored in MySQL and we used Redis to speed up querying. We were doing this by performing a SELECT query on MySQL:

WHERE products.id IN ( ... ).

Ultimately, it's up to you to decide where to store the objects: in Redis or elsewhere. Each use case is different.

Querying Sorted Sets (ZRANGEBYSCORE)

Previously, we saw that sorted set is a good choice to store values we want to query as numeric values – by range or using comparisons. This is not the only use for sorted sets, but its the only one we'll cover.

If you want to retrieve all product with price greater or equal to 50, you may call:

> ZRANGEBYSCORE product:price 50 +INF

The command expects 3 arguments: the key, lower limit and upper limit. As you can see we're passing the positive infinite (+INF) as the third argument. The negative infinite (-INF) is also available if you need it.

A specific range, like 40-50, can be also be used:

> ZRANGEBYSCORE product:price 40 50

Querying Hashes (HMGET and HSCAN)

Finally, to search within the strings stored under a hset, let's perform the following call:

> HMGET product:name 'Cool product'

This would return the product having that exact name. It's also possible to do a partial match, similar to SQL's LIKE operator:

> HSCAN product:name 0 MATCH *ol*

The * is a wildcard for any amount of any characters. The command above would match cool, toll and pole.

You probably noticed the 0 as second argument. That is the current cursor. All Redis' commands from the SCAN family are iterators based on a cursor. After each call, the server will return the cursor for the next iteration. You should use it in the next call. Keep calling the server until you get 0 in the cursor.

Using Lua for SCAN iteration

There is no command in Redis to loop through an iterator. Fortunately, Redis supports scripting with Lua and we can use a small script to achieve that:

    local cursor = 0
    local fields = {}
    local ids = {}
    local key = 'product:name'
    local value = '*' .. ARGV[1] .. '*'

    repeat
        local result = redis.call('HSCAN', key, cursor, 'MATCH', value)
        cursor = tonumber(result[1])
        fields = result[2]
        for i, id in ipairs(fields) do
            if i % 2 == 0 then
                ids[#ids + 1] = id
                redis.call('SADD', KEYS[1], id)
            end
        end
    until cursor == 0
    return ids

The code above returns all products that match a given value. It also stores that same result in a set, so we can use it again later.

To parse and run the script, call:

> EVAL 'local cursor; [...]' 1 tmp:name cool

EVAL expects a script as the first argument, then the number of keys we're going to pass to the script, followed by the list of keys, and finally the list of arguments.

In our case here we passed one key (tmp:name) and one argument (cool). Inside the script, the key is accessed with KEYS[1] and the argument is accessed using ARGV[1].

Putting all together

By now, you have seen how to store and query each type of data. It's really useful, but in some cases the querying rules are more complex.

Let's say you have to find all products whose name contains "cool" and are priced in 50.00 or more. One way to do this is to fetch ID's for each condition and intersect the results client-side. However, in our opinion there is a better way:

> ZUNIONSTORE tmp:price 1 product:price WEIGHTS 1
> ZREMRANGEBYSCORE tmp:price -INF (50
> EVAL 'local cursor; [...]' 1 tmp:name cool
> ZINTERSTORE tmp:result tmp:price tmp:name
> ZRANGE tmp:result 0 -1

Let's go through each call:

  1. We start by creating a temporary copy of the product:price key using ZUNIONSTORE.
  2. We remove all values lower than 50 from the temporary copy. The ( before the number indicates that 50 must be kept in the set.
  3. Run the iterator script storing all matched ID's in a temporary key.
  4. Intersect all values in tmp:price and tmp:name, storing the result as another key.
  5. Finally, we get all the values from the previous intersection.

The last call will give you a list of ID's that, as explained in the begin, can be used to retrieve full objects.

Getting all products whose name contains "Cool" or are priced in 50.00 or more, is as simple as replacing ZINTERSTORE with ZUNIONSTORE in the fourth call.

Conclusion

It is possible to combine more unions (OR) and intersections (AND) to fulfill even more complex logic. However, keep in mind that when mixing those group operations, the order does matter.

To dive into further detail on each command, the Redis list of commands is a great place to start.