A reminder about queue depth with Redis Stream

Published October 8, 2024

Hello!

Do not use Redis as queue system they say. Don’t tell them you use Redis Stream for queues!
I kind of agree, I think Redis Stream does not provide the right API, and it is not ergonomics like other tools but if you are prototyping or if you ended up growing to quickly and moving to something else is not an option because of different priorities or because like sucks I hear you!

Queue depth is an important metric when it comes to the healthiness of a queuing system because it can drive resource management, if you are falling behind or if you are wasting unused resources.

var infos []redis.XInfoGroup
infos, err = q.redisClient.XInfoGroups(ctx, streamKey.String()).Result()
if err != nil {
    return nil, err
}
var consumer *redis.XInfoGroup
for _, info := range infos {
    if info.Name == "theconsumeryouarelookingfor" {
        consumer = &info
    }
}
if pikeGroup == nil {
    continue
}
queue := QueueDepth{}
queue.Completed = int(consumer.EntriesRead - consumer.Pending)
queue.InProcess = int(consumer.Pending)
queue.Waiting = int(consumer.Lag)

I want to remember how I implemented this logic for future use. In practice since my code uses consumer groups I need to pick the one I need to calculate queue depth for (there are a million ways to do it this is just boilerplate) and then I do the math go calculate how many jobs are Completed, how many are InProcess so currently in execution and how many are Waiting.

This is useful and currently in use for observability and autoscaling.

Are you having trouble figuring out your way to building automation, release and troubleshoot your software? Let's get actionables lessons learned straight to you via email.