Implementing a Redis Job Queue Without Laravel

We all love job queues and workers. It makes responses faster, more responsive. We can use worker servers or processes to do the heavy work. Laravel makes it really easy to run a Redis or database job queue.

But I meet a lot of junior and medior developers who think that job queues in Laravel is some kind of magic. They think that some big architectural stuff going on in the background. But in fact, it's not true. The basics of a Redis job queue (or any other kind of queue) is really simple, so in this blog post we're gonna implement a really simple one.

What Is a Queue?

A queue is a really old data structure. It has nothing to do with jobs or complex events, it's just a data structure. You can push an element to a queue and pop an element out of it. Queues also have some rules about the ordering of the elements. For example a FIFO queue means: First In First Out. So if you put 1, 2, 3 to a FIFO queue, when you request an element from it you'll get 1. If you want to get one more element you'll get 2. Then 3.

You can implement it with an array:

class QueueWithArray
{
    /**
     * @var int[]
     */
    private array $items = [];

    public function enqueue(int $item): void
    {
        $this->items[] = $item;
    }

    public function dequeue(): int
    {
        return array_shift($this->items);
    }
}

The array_shift() guarantees the FIFO order.

Usage:

$queue = new QueueWithArray();
$queue->enqueue(1);
$queue->enqueue(2);
$queue->enqueue(3);

var_dump($queue->dequeue());
var_dump($queue->dequeue());
var_dump($queue->dequeue());

Output:

int(1)
int(2)
int(3)

That's it! So a queue is basically a data structure that stores stuff. It also enforces some rules (FIFO in this case) about in which order you can get elements out of the queue.

This is the data structure that backs every advanced job queue system. The difference is in a Laravel job queue you put serialized PHP objects (the Job classes) to the queue instead of integers. The other difference is the naming:

• enqueue = MyJob::dispatch($data)
• dequeue = php artisan queue:work

When you dispatch a job, you enqueue a new item. When you run queue:work you dequeue an item from the queue.

And you have one more important difference: instead of an array you store the items in Redis.

Implementing a Queue With Redis

If a queue can be implemented using an array than we need to find an array-like data structure in Redis. And you know this is easy. Redis has Lists.

We also need to find a way to implement FIFO. Of course Redis gives us the solution:

• RPUSH (Right push): append an element to a list (or in other words in inserts an element to the end of a list)
• LPOP (Left Pop): remove and get the first element from a list

We can visualize it:

(You also have an option to use LPUSH and RPOP. It achieves the same result but in different order)

Now let's see the code:

use Predis\Client;

class Queue
{
    private Client $client;

    public function __construct(private string $key)
    {
        $this->key = $key;
        $this->client = new Client([
            'scheme' => 'tcp',
            'host'   => '127.0.0.1',
            'port'   => 6379,
        ]);
    }

    public function enqueue(int $item): void
    {
        $this->client->rpush($this->key, $item);
    }

    public function dequeue(): ?int
    {
        return $this->client->lpop($this->key);
    }
}

The main difference compared to arrays:

• We use Redis instead of an array
• We use Redis commands instead of array operations

The key parameter is the key in Redis for the list.

The usage is very straightforward:

$queue = new Queue('queue');
$queue->enqueue(1);
$queue->enqueue(2);
$queue->enqueue(3);

And we can easily implement a simple queue worker:

$queue = new Queue('queue');

while (true) {
    sleep(1);
    $item = $queue->dequeue();
    
    if ($item !== null) {
        var_dump($item);
    }
}

This is the bare minimum implementation of a job queue. If you replace the integer with json or deserialized PHP objects you have a very basic job queue. As you can see there's no magic involved.